Auxiliary labeling method, electronic device, storage medium and program product
By acquiring high-quality manually labeled image frames from 3D point cloud data and performing multiple coordinate transformations and auxiliary annotations, the annotation difficulties caused by the sparsity of 3D point cloud data are solved, and efficient and accurate annotation of stationary targets is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING MAICHI ZHIXING TECHNOLOGY CO LTD
- Filing Date
- 2022-12-16
- Publication Date
- 2026-07-03
AI Technical Summary
Due to the sparsity, discontinuity, and irregularity of 3D point cloud data, existing technologies struggle to efficiently and accurately label stationary targets such as vehicles and pedestrians, resulting in low vehicle labeling efficiency.
By acquiring high-quality manually labeled target image frames, performing multiple coordinate transformations to generate auxiliary labeling data, and combining coordinate system transformation and vehicle labeling tools, overlapping or occluded label boxes are automatically deleted, improving labeling efficiency and accuracy.
It improves the efficiency and accuracy of labeling stationary targets, simplifies the manual labeling process, reduces repetitive labeling work for each frame of image, and improves overall labeling efficiency.
Smart Images
Figure CN116152700B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and more specifically, to an auxiliary annotation method, electronic device, computer-readable storage medium, and computer program product. Background Technology
[0002] With the rise of computer vision technology, autonomous driving has experienced rapid development. Autonomous driving systems perceive the environment around the vehicle through a perception system and make driving decisions based on this environment to control the vehicle's autonomous driving. During the process of perceiving the vehicle's surroundings, the perception system annotates the sensor data (such as obstacle annotations), and the environmental conditions around the vehicle are determined based on the entity annotation results.
[0003] Currently, vehicle annotation tools based on 3D point cloud data have become an indispensable requirement in the development of autonomous driving. However, due to the sparsity, discontinuity, and irregularity of 3D point cloud maps, it is difficult to distinguish the collected vehicle, pedestrian, and other annotation objects, increasing the difficulty of manually determining the presence and type of vehicles, thereby reducing the efficiency of vehicle annotation.
[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The purpose of this disclosure is to provide an auxiliary annotation method, electronic device, computer-readable storage medium, and computer program product, thereby overcoming, at least to some extent, the problem of low efficiency in vehicle annotation due to related technologies.
[0006] According to a first aspect of this disclosure, an auxiliary annotation method is provided, comprising:
[0007] Obtain manually labeled data of stationary targets in a target image frame, wherein the image quality of the target image frame is higher than that of other image frames;
[0008] Based on the manually labeled data, multiple coordinate transformations are performed to obtain auxiliary labeling data for the stationary target in each of the image frames;
[0009] The target label identifier of the stationary target in each of the image frames is determined based on the auxiliary annotation data.
[0010] In one exemplary embodiment of this disclosure, the manually labeled data is labeled data in a first coordinate system;
[0011] The process of performing multiple coordinate transformations based on the manually labeled data to obtain auxiliary labeling data for the stationary target in each image frame includes:
[0012] The manually labeled data is transformed into a second coordinate system to obtain the first transformed data;
[0013] Based on the first transformation data, determine the second transformation data of the stationary target in each of the image frames in the second coordinate system;
[0014] The second transformed data in the second coordinate system is transformed into the first coordinate system to obtain the auxiliary annotation data.
[0015] In an exemplary embodiment of this disclosure, determining the second transformation data of the stationary target in the second coordinate system in each of the image frames based on the first transformation data includes:
[0016] The second transformation data of the stationary target in each image frame in the second coordinate system is composed of the labeled coordinates in the first transformation data and the frame identifier of each image frame.
[0017] In one exemplary embodiment of this disclosure, determining the target annotation identifier of the stationary target in each of the image frames based on the auxiliary annotation data includes:
[0018] Based on the auxiliary annotation data, generate auxiliary annotation labels for the stationary targets in each of the image frames;
[0019] Determine whether the auxiliary annotation of the stationary target in each of the image frames overlaps with the pre-annotation of the stationary target;
[0020] When the auxiliary label of the stationary target in each of the image frames overlaps with the pre-label of the stationary target, the target label of the stationary target is determined according to the auxiliary label.
[0021] In an exemplary embodiment of this disclosure, determining the target label of the stationary target based on the auxiliary label when the auxiliary label of the stationary target overlaps with the pre-label of the stationary target in each of the image frames includes:
[0022] When the auxiliary label of the stationary target in each of the image frames overlaps with the pre-label of the stationary target, the pre-label of the stationary target in the corresponding image frame is automatically deleted, and the auxiliary label is used as the target label of the stationary target in the corresponding image frame.
[0023] In one exemplary embodiment of this disclosure, after obtaining the target annotation identifier of the stationary target in each of the image frames, the method further includes:
[0024] Traverse the target annotations of the stationary target in each of the image frames and obtain the number of point clouds corresponding to the target annotations;
[0025] When the number of point clouds corresponding to the target annotation is less than the point cloud number threshold, the target annotation of the stationary target in the corresponding image frame is automatically deleted.
[0026] In one exemplary embodiment of this disclosure, the first coordinate system is a sensor coordinate system, and the second coordinate system is a world coordinate system.
[0027] According to a second aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any one of the preceding methods by executing the executable instructions.
[0028] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.
[0029] According to a fourth aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the method described in any of the preceding claims.
[0030] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:
[0031] In the auxiliary annotation method provided in the exemplary embodiments of this disclosure, manually annotated data of stationary targets in target image frames are obtained, wherein the image quality of the target image frame is higher than that of other image frames; multiple coordinate transformations are performed based on the manually annotated data to obtain auxiliary annotation data of the stationary targets in each of the image frames; and target annotation identifiers of the stationary targets in each of the image frames are determined according to the auxiliary annotation data. On the one hand, by performing multiple coordinate transformations on the manually annotated data, target annotation identifiers of multiple image frames can be generated efficiently, improving the annotation efficiency of stationary targets; on the other hand, using the generated manually annotated data to assist in the annotation of static targets can improve the accuracy of the annotation of stationary targets.
[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0034] Figure 1 A schematic diagram of a system architecture for which the auxiliary annotation method of the present disclosure embodiments can be applied is shown;
[0035] Figure 2 A flowchart illustrating an auxiliary annotation method according to an embodiment of this disclosure is shown schematically;
[0036] Figure 3 A flowchart illustrating a method for rapidly generating labeled data is shown in an embodiment of this disclosure.
[0037] Figure 4 A flowchart illustrating another auxiliary annotation method in an embodiment of this disclosure is shown schematically;
[0038] Figure 5 A block diagram of an auxiliary annotation device according to an embodiment of the present disclosure is shown schematically;
[0039] Figure 6 The schematic diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation
[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0041] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0042] Figure 1 A schematic diagram of a system architecture for an auxiliary annotation method that can be applied to embodiments of this disclosure is shown.
[0043] like Figure 1 As shown, the system architecture 100 may include one or more of sensors 101 and 102, a network 103, and an auxiliary annotation device 104. Sensors 101 and 102 may be image acquisition sensors such as cameras, radar, and lidar mounted on autonomous vehicles. The network 103 serves as the medium for providing communication links between sensors 101 and 102 and the auxiliary annotation device 104. The network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables. The auxiliary annotation device 104 may be a terminal device or a server. The terminal device may be various electronic devices, including but not limited to desktop computers, laptops, smartphones, and tablets. The server may be a single server, a server cluster consisting of multiple servers, a cloud computing platform, or a virtualization center. The auxiliary annotation device 104 can annotate stationary targets in the image frames acquired by sensors 101 and 102. It should be understood that... Figure 1 The number of devices is merely exemplary; depending on the implementation requirements, there may be any number of sensors 101, 102, network 103, and auxiliary labeling device 104.
[0044] The auxiliary annotation method provided in this embodiment is generally executed by the auxiliary annotation device 104 (terminal device). Correspondingly, the auxiliary annotation apparatus is generally disposed in the terminal device. For example, after the terminal device executes the auxiliary annotation method, the generated auxiliary annotation identifier can be directly displayed on the display screen of the terminal device. This exemplary embodiment does not impose any special limitations on this. However, it is readily understood by those skilled in the art that the auxiliary annotation method provided in this embodiment can also be executed by the auxiliary annotation device 104 (server). Correspondingly, the auxiliary annotation apparatus can also be disposed in the terminal device. The server can send the generated auxiliary annotation identifier to the terminal device, and the terminal device can display it to the user. This exemplary embodiment does not impose any special limitations on this.
[0045] The technical solutions of the embodiments of this disclosure are described in detail below:
[0046] Currently, vehicle labeling tools based on 3D point cloud data have become an indispensable requirement in the development of autonomous driving. However, due to the sparsity, discontinuity, and irregularity of 3D point cloud maps, it is difficult to distinguish the collected vehicle, pedestrian, and other labeled objects, increasing the difficulty of manually determining the presence and type of vehicles, thereby reducing the efficiency of vehicle labeling.
[0047] To address the aforementioned issues, this example implementation provides an auxiliary annotation method. (Reference) Figure 2 As shown, the auxiliary annotation method may include steps S210 to S230:
[0048] Step S210. Obtain manually labeled data of stationary targets in the target image frame, wherein the image quality of the target image frame is higher than that of other image frames;
[0049] Step S220. Perform multiple coordinate transformations based on the manually labeled data to obtain auxiliary labeling data for the stationary target in each of the image frames;
[0050] Step S230. Determine the target label of the stationary target in each of the image frames based on the auxiliary annotation data.
[0051] In the auxiliary annotation method provided in the exemplary embodiments of this disclosure, manually annotated data of stationary targets in target image frames are obtained, wherein the image quality of the target image frame is higher than that of other image frames; multiple coordinate transformations are performed based on the manually annotated data to obtain auxiliary annotation data of the stationary targets in each of the image frames; and target annotation identifiers of the stationary targets in each of the image frames are determined according to the auxiliary annotation data. On the one hand, by performing multiple coordinate transformations on the manually annotated data, target annotation identifiers of multiple image frames can be generated efficiently, improving the annotation efficiency of stationary targets; on the other hand, using the generated manually annotated data to assist in the annotation of static targets can improve the accuracy of the annotation of stationary targets.
[0052] The steps described above in this example implementation will now be explained in more detail.
[0053] In step S210, manually labeled data of stationary targets in the target image frame are obtained, wherein the image quality of the target image frame is higher than that of other image frames.
[0054] In related technologies, when using algorithms to identify and generate pre-labeled vehicle markers from 3D point cloud data, inaccurate labeling is inevitable. Therefore, manual verification of each pre-labeled marker is required. In the example implementation of this disclosure, vehicle labeling starts with stationary targets, combining coordinate system transformation and vehicle labeling tools to achieve rapid labeling of stationary targets, thereby improving the overall labeling efficiency (stationary target labeling + moving target labeling).
[0055] For example, after acquiring multiple frames of images using a LiDAR sensor mounted on an autonomous vehicle, the stationary targets in each frame can be labeled as a group. This could be done by grouping images into sets of 60 or 300 frames, or by selecting the number of frames based on actual needs; this disclosure does not impose specific limitations on this. Taking the labeling of 60 frames as an example, each frame is likely to contain stationary targets such as stationary vehicles. The frame with the highest image quality among the 60 frames can be selected as the target image frame. For instance, the t-th frame with the highest pixel count among the 60 frames can be selected as the target image frame. The stationary targets in this image frame are then manually identified and labeled, resulting in manually labeled data for the stationary targets.
[0056] The manually labeled data refers to the labeled data in a first coordinate system. This first coordinate system can be a sensor coordinate system, such as a lidar (light detection and ranging) coordinate system or a camera coordinate system; this disclosure does not limit this. Taking the lidar coordinate system as the first coordinate system as an example, for a stationary target in the t-th frame image, the manually labeled data for that stationary target includes the labeled coordinates of the stationary target and a frame identifier, denoted as B(t, lidar), where t is the frame identifier representing the t-th frame image, and lidar represents the labeled coordinates of the stationary target in the t-th frame image. Correspondingly, the labeled coordinates in the manually labeled data are the lidar coordinates of the center point of the stationary target. It is understood that the lidar coordinates in the corresponding manually labeled data will be different depending on the stationary target in the image frame.
[0057] In step S220, multiple coordinate transformations are performed based on the manually labeled data to obtain auxiliary labeling data for the stationary target in each of the image frames.
[0058] Still target annotation refers to the manual annotation data of the current frame image being in the LiDAR coordinate system. When a still target is stationary in the world coordinate system, the manual annotation data of the still target in the current frame image can be applied to other frame images. However, since a still target may be in motion in the LiDAR coordinate system, the manual annotation data of the still target in the current frame image cannot be directly copied to other frame images. Instead, it is necessary to first transform the manual annotation data of the still target in the current frame image to the world coordinate system, then copy the transformed annotation data of the still target in the current frame image in the world coordinate system to other frame images, and finally transform the annotation data of the still target in each frame image in the world coordinate system to the LiDAR coordinate system to obtain the target annotation label of the still target in each frame image.
[0059] For example, refer to Figure 3 As shown, auxiliary annotation data for each image frame can be quickly generated according to steps S310 to S330.
[0060] Step S310. Transform the manually labeled data into the second coordinate system to obtain the first transformed data.
[0061] In the exemplary embodiments of this disclosure, the second coordinate system can be the world coordinate system. Correspondingly, after obtaining the manually labeled data of a stationary target in the t-th frame image, the manually labeled data can be transformed into the world coordinate system to obtain the first transformed data, that is, B(t, lidar) transformed to obtain B(t, world), where B(t, world) represents the transformed data of the stationary target in the t-th frame image in the world coordinate system, and world represents the labeled coordinates of the stationary target in the t-th frame image. At this time, the labeled coordinates are the world coordinates of the center point of the stationary target. Specifically, the first coordinate system transformation matrix can be obtained from the sensor calibration attributes of the pre-labeled JSON (JavaScript Object Notation, data exchange format) file, and the first calibration parameters, such as the vehicle position where the sensor is located, can be obtained from the data file of the corresponding frame of the pre-labeled JSON file. The manually labeled data B(t, lidar) is then calculated based on the vehicle position and the first coordinate system transformation matrix to obtain the first transformed data B(t, world).
[0062] Step S320. Determine the second transformation data of the stationary target in the second coordinate system in each of the image frames based on the first transformation data.
[0063] After obtaining the first transformation data of the stationary coordinates in the target image frame, the first transformation data can be applied to other image frames to obtain the second transformation data of the stationary target in the other image frames in the second coordinate system. For example, the second transformation data of the stationary target in each image frame in the second coordinate system can be composed of the labeled coordinates in the first transformation data and the frame identifiers of other image frames. For instance, based on the first transformation data B(t, world) of the stationary target in the t-th frame image in the world coordinate system, the second transformation data of the stationary target in each image frame in the world coordinate system can be obtained, denoted as B(T, world), where T represents other image frames besides the t-th frame image. For example, B(T, world) can be {…, B(t-1, world), B(t, world), B(t+1, world),…}.
[0064] Step S330. Transform the second transformed data in the second coordinate system to the first coordinate system to obtain the auxiliary annotation data.
[0065] After obtaining the second transformation data B(T, world) of the stationary target in each image frame in the world coordinate system, the second transformation data can be transformed into the lidar coordinate system to obtain the auxiliary annotation data of the stationary target in each image frame, denoted as B(T, lidar). For example, B(T, lidar) can be {…, B(t-1, lidar), B(t, lidar), B(t+1, lidar),…}. Specifically, the second coordinate system transformation matrix can be obtained from the sensor calibration properties of the pre-annotated JSON (JavaScript Object Notation, data exchange format) file, and the second calibration parameters, such as the vehicle position where the sensor is located, can be obtained from the data file of the corresponding frame in the pre-annotated JSON file. The second transformation data B(T, world) is then calculated based on the vehicle position and the second coordinate system transformation matrix to obtain the auxiliary annotation data B(T, lidar).
[0066] In this example, by transforming the labeled data back and forth between lidar coordinates and world coordinates, auxiliary labeled data can be quickly generated. This auxiliary labeled data can be used to assist the labeler in labeling static targets, thereby improving the efficiency of static target labeling.
[0067] In step S230, the target label identifier of the stationary target in each of the image frames is determined based on the auxiliary labeling data.
[0068] After obtaining the auxiliary annotation data B(T, lidar) of stationary targets in each image frame in the lidar coordinate system, auxiliary annotation labels for stationary targets in each image frame are generated based on the auxiliary annotation data. For example, for a certain frame of image, the lidar coordinates of the center point of the stationary target can be used to diverge, such as by creating a 3D bounding box around the center point of the stationary target, to obtain the auxiliary annotation labels for the stationary targets in that frame of image. For example, the auxiliary annotation box of the stationary target can be obtained, or other forms of auxiliary annotation labels can be obtained. This disclosure does not limit this.
[0069] After generating auxiliary annotations for stationary targets in each image frame, the presence of a pre-annotated label at the location of the auxiliary annotation is determined by checking whether the auxiliary annotation overlaps with the pre-annotated label of the stationary target in each image frame. For example, the IOU (Intersection Over Union) algorithm can be used to determine if the auxiliary annotation overlaps with the pre-annotated label of the stationary target in each image frame. When it is determined that the auxiliary annotation overlaps with the pre-annotated label, the target annotation for the stationary target is determined based on the auxiliary annotation. For instance, when the auxiliary bounding box of a stationary target overlaps with the pre-annotated bounding box of a stationary target in each image frame, the pre-annotated bounding box of the corresponding image frame is automatically deleted, only the auxiliary bounding box is retained, and the auxiliary bounding box is used as the target bounding box of the stationary target in the corresponding image frame.
[0070] Furthermore, after obtaining the target labels of stationary targets in each image frame, the target labels of stationary targets in each image frame can be traversed, and the number of point clouds corresponding to the target labels can be obtained. When the number of point clouds corresponding to the target labels is less than the point cloud count threshold, the target labels of the stationary targets in the corresponding image frame are automatically deleted. For example, if the total number of point clouds in the auxiliary label box of the stationary target is 10, and the preset point cloud count threshold is 5, the number of point clouds in the auxiliary label box of the stationary target in a certain frame image can be obtained. When the number of point clouds in the auxiliary label box is less than 5, it indicates that the auxiliary label box is occluded by other obstacles. At this time, the auxiliary label box is automatically deleted, which means that there is no need to label the stationary target in the current frame image.
[0071] In the exemplary implementation of this disclosure, the annotator only needs to annotate a stationary target in one frame of image and confirm whether the target is stationary. If it is confirmed to be stationary, the manually annotated bounding box of the stationary target can be directly applied to all frames of image. After coordinate system transformation, manually annotated bounding boxes can be generated at the corresponding positions in other image frames. Then, based on the number of point clouds within the manually annotated bounding boxes, individual manually annotated bounding boxes that are occluded are automatically deleted. Then, it is determined whether there is a pre-annotated bounding box above the predicted position. If there is a pre-annotated bounding box, the predicted box is deleted. The entire annotation process only needs to confirm whether the manually annotated bounding box moves midway and whether the manually annotated bounding box is accurately annotated, without having to re-annotate the stationary coordinates of each frame of image.
[0072] In one example implementation, reference Figure 4 As shown, steps S401 to S409 can be used to assist in the labeling of stationary targets.
[0073] Step S401. Manually determine the stationary target, such as determining that target X is a stationary target;
[0074] Step S402: Label the stationary target X in the t-th frame image to obtain the Box information in the lidar coordinate system. The Box information is B(t, lidar).
[0075] Step S403. Transform Box information from lidar coordinate system to world coordinate system: Transform B(t, lidar) from lidar coordinate system to world coordinate system to obtain B(t, world);
[0076] Step S404. Apply Box information to all other frames: Copy the Box information B(t, world) of the stationary target in the image of frame t to all frames to obtain the Box information of the stationary target in all frames, such as {…, B(t-1, world), B(t, world), B(t+1, world), …};
[0077] Step S405. Transform the Box information from the world coordinate system to the lidar coordinate system: Transform the Box information {…, B(t-1, world), B(t, world), B(t+1, world), …} of all stationary targets in all frames from the world coordinate system to the lidar coordinate system to obtain {…, B(t-1, lidar), B(t, lidar), B(t+1, lidar), …};
[0078] Step S406. Determine whether there is a pre-labeled box at the location of the auxiliary label box. Generate an auxiliary label box according to {…, B(t-1, lidar), B(t, lidar), B(t+1, lidar), …} and determine whether the auxiliary label box of the stationary target in all frames overlaps with the pre-labeled box. If they overlap, it indicates that there is a pre-labeled box. Then proceed to step S407.
[0079] Step S407. Delete the pre-annotation boxes, keeping only the auxiliary annotation boxes;
[0080] Step S408. Traverse all auxiliary annotation boxes and delete the occluded auxiliary annotation boxes according to the number of point clouds inside the box;
[0081] Step S409. Manually inspect the generated auxiliary annotation box of the stationary target X.
[0082] It should be noted that after obtaining the Box information of stationary targets in all frames in the lidar coordinate system through step S405, step S408 can be executed directly to delete the occluded auxiliary annotation boxes, and then steps S406 and S407 can be executed to delete the pre-annotation boxes that overlap with the auxiliary annotation boxes. This disclosure does not limit the execution order of steps S406 to S407 and steps S408.
[0083] Vehicle annotation includes both stationary and moving target annotation. In this example, the annotation of stationary targets is achieved by combining coordinate system transformation and vehicle annotation tools, which improves the annotation efficiency of stationary targets and thus improves the overall annotation efficiency.
[0084] In the auxiliary annotation method provided in the exemplary embodiments of this disclosure, manually annotated data of stationary targets in target image frames are obtained, wherein the image quality of the target image frame is higher than that of other image frames; multiple coordinate transformations are performed based on the manually annotated data to obtain auxiliary annotation data of the stationary targets in each of the image frames; and target annotation identifiers of the stationary targets in each of the image frames are determined according to the auxiliary annotation data. On the one hand, by performing multiple coordinate transformations on the manually annotated data, target annotation identifiers of multiple image frames can be generated efficiently, improving the annotation efficiency of stationary targets; on the other hand, using the generated manually annotated data to assist in the annotation of static targets can improve the accuracy of the annotation of stationary targets.
[0085] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0086] Furthermore, this example embodiment also provides an auxiliary annotation device. This device can be applied to a terminal device or a server. (See reference...) Figure 5 As shown, the auxiliary annotation device 500 may include a data acquisition module 510, a data generation module 520, and an identifier determination module 530, wherein:
[0087] The data acquisition module 510 is used to acquire manually labeled data of stationary targets in the target image frame, wherein the image quality of the target image frame is higher than that of other image frames;
[0088] The data generation module 520 performs multiple coordinate transformations based on the manually labeled data to obtain auxiliary labeling data for the stationary target in each of the image frames;
[0089] The identifier determination module 530 is used to determine the target label identifier of the stationary target in each of the image frames based on the auxiliary annotation data.
[0090] In one optional implementation, the manually labeled data is labeled data in a first coordinate system; the data generation module 520 includes:
[0091] The first transformation data generation module is used to transform the manually labeled data to the second coordinate system to obtain the first transformation data;
[0092] The second transformation data generation module is used to determine the second transformation data of the stationary target in each of the image frames in the second coordinate system based on the first transformation data.
[0093] The auxiliary annotation data generation module is used to transform the second transformation data in the second coordinate system to the first coordinate system to obtain the auxiliary annotation data.
[0094] In one alternative implementation, the second transformation data generation module is configured to compose the second transformation data of the stationary target in each of the image frames in the second coordinate system, which consists of the labeled coordinates in the first transformation data and the frame identifiers of each of the image frames.
[0095] In one optional implementation, the identifier determination module 530 includes:
[0096] The identifier generation submodule is used to generate auxiliary annotation identifiers for the stationary targets in each of the image frames based on the auxiliary annotation data;
[0097] The identification judgment submodule is used to determine whether the auxiliary annotation of the stationary target in each of the image frames overlaps with the pre-annotation of the stationary target;
[0098] The identifier determination submodule is used to determine the target identifier of the stationary target based on the auxiliary identifier when the auxiliary identifier of the stationary target in each of the image frames overlaps with the pre-label identifier of the stationary target.
[0099] In one optional implementation, the identifier determination submodule is configured to automatically delete the pre-label of the stationary target in the corresponding image frame when the auxiliary label identifier of the stationary target in each of the image frames overlaps with the pre-label identifier of the stationary target, and use the auxiliary label identifier as the target label identifier of the stationary target in the corresponding image frame.
[0100] In an optional embodiment, the auxiliary annotation device 500 further includes:
[0101] The point cloud count determination module is used to traverse the target annotation identifiers of the stationary target in each of the image frames and obtain the point cloud count corresponding to the target annotation identifier;
[0102] The label deletion module is used to automatically delete the target label of the stationary target in the corresponding image frame when the number of point clouds corresponding to the target label is less than the number of point clouds threshold.
[0103] The specific details of each module in the above-mentioned auxiliary annotation device have been described in detail in the corresponding auxiliary annotation methods, so they will not be repeated here.
[0104] The modules in the above-described device can be general-purpose processors, including central processing units (CPUs), network processors, etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Each module can also be implemented using software, firmware, etc. The processors in the above-described device can be independent processors or integrated together.
[0105] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) including program code and may run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0107] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.
[0108] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0109] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0110] Exemplary embodiments of this disclosure also provide an electronic device capable of implementing the above-described method. Referring below... Figure 6 To describe an electronic device 600 according to such an exemplary embodiment of the present disclosure. Figure 6The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0111] like Figure 6 As shown, the electronic device 600 can be represented as a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), and a display unit 640.
[0112] Storage unit 620 stores program code that can be executed by processing unit 610, causing processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, processing unit 610 can execute... Figures 2 to 4 Any one or more of the method steps.
[0113] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.
[0114] Storage unit 620 may also include a program / utility 624 having a set (at least one) of program modules 625, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0115] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0116] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with the electronic device 600, and / or with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Figure 6 As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although... Figure 6 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0117] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.
[0118] Furthermore, the above figures are merely illustrative representations of the processes included in the methods according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0119] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0120] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. An auxiliary annotation method, characterized in that, include: Obtain manually labeled data of stationary targets in a target image frame, wherein the image quality of the target image frame is higher than that of other image frames; Based on the manually labeled data, multiple coordinate transformations are performed to obtain auxiliary labeling data for the stationary target in each of the image frames; The target annotation identifier of the stationary target in each of the image frames is determined based on the auxiliary annotation data; The manually labeled data refers to the labeled data in the first coordinate system; The process of performing multiple coordinate transformations based on the manually labeled data to obtain auxiliary labeling data for the stationary target in each image frame includes: The manually labeled data is transformed into a second coordinate system to obtain the first transformed data; Based on the first transformation data, determine the second transformation data of the stationary target in each of the image frames in the second coordinate system; The second transformed data in the second coordinate system is transformed into the first coordinate system to obtain the auxiliary annotation data.
2. The auxiliary annotation method according to claim 1, characterized in that, The step of determining the second transformation data of the stationary target in the second coordinate system in each of the image frames based on the first transformation data includes: The second transformation data of the stationary target in each image frame in the second coordinate system is composed of the labeled coordinates in the first transformation data and the frame identifier of each image frame.
3. The auxiliary annotation method according to claim 1, characterized in that, The step of determining the target label of the stationary target in each of the image frames based on the auxiliary annotation data includes: Based on the auxiliary annotation data, generate auxiliary annotation labels for the stationary targets in each of the image frames; Determine whether the auxiliary annotation of the stationary target in each of the image frames overlaps with the pre-annotation of the stationary target; When the auxiliary label of the stationary target in each of the image frames overlaps with the pre-label of the stationary target, the target label of the stationary target is determined according to the auxiliary label.
4. The auxiliary annotation method according to claim 3, characterized in that, When the auxiliary annotation of the stationary target in each of the image frames overlaps with the pre-annotation of the stationary target, determining the target annotation of the stationary target based on the auxiliary annotation includes: When the auxiliary label of the stationary target in each of the image frames overlaps with the pre-label of the stationary target, the pre-label of the stationary target in the corresponding image frame is automatically deleted, and the auxiliary label is used as the target label of the stationary target in the corresponding image frame.
5. The auxiliary annotation method according to claim 1, characterized in that, After obtaining the target annotation identifier of the stationary target in each of the image frames, the method further includes: Traverse the target annotations of the stationary target in each of the image frames and obtain the number of point clouds corresponding to the target annotations; When the number of point clouds corresponding to the target annotation is less than the point cloud number threshold, the target annotation of the stationary target in the corresponding image frame is automatically deleted.
6. The auxiliary annotation method according to claim 1, characterized in that, The first coordinate system is the sensor coordinate system, and the second coordinate system is the world coordinate system.
7. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-6 by executing the executable instructions.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.
9. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.