Target marking method, electronic device and storage medium

Abstract

The present application relates to the field of computer vision, and specifically provides a method for labeling a target, an electronic device, and a storage medium, aiming to solve the problems of large errors, inability to modify, and low data conversion rate in existing target labeling methods. To this end, the method of the present application includes obtaining multiple groups of initial multi-channel image frames corresponding to multiple moments; obtaining corresponding depth maps, three-dimensional prediction frames and prediction lines of the target to be labeled; based on the depth map and the three-dimensional prediction frame of the target to be labeled, performing dynamic and static separation of the target to be labeled to obtain a point cloud of a dynamic target and a point cloud of a static target; performing multi-frame splicing of the point cloud of the static target in the multiple groups of initial multi-channel image frames to obtain a complete static point cloud; fusing the point cloud of the dynamic target with the complete static point cloud to obtain a global point cloud; labeling the global point cloud based on the three-dimensional prediction frame and prediction line of the target to be labeled to obtain a three-dimensional point cloud with the labeled target. Through the above implementation, the accuracy of data labeling can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target labeling method, electronic device and storage medium. Background Art

[0002] At present, in the field of computer vision, when annotating objects in a 3D point cloud, the 3D prediction box is usually directly output through the 3D detection model, and then the 3D point cloud is projected onto a 2D image to obtain a 2D image with a 3D prediction box. On this basis, the correctness of the annotated prediction box is manually judged to filter out valid data, providing basic data for subsequent model training and data analysis.

[0003] However, this annotation method has some problems. First, projecting the 3D point cloud onto the 2D image will affect the judgment of the edge position of the target, resulting in large errors in manual judgment of whether the annotation is correct. Second, since the 2D image cannot estimate the depth information of the target, the annotation on the 2D image cannot be modified. Therefore, most of the data can only be discarded if there is one error, resulting in a low conversion rate of the annotated data.

[0004] Accordingly, the art needs a new technical solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to provide a target labeling method, electronic device and storage medium that solves or at least partially solves the technical problems of existing target labeling methods such as large errors, inability to modify and low data conversion rate.

[0006] In a first aspect, a method for marking a target is provided, wherein the method comprises:

[0007] Acquire multiple groups of initial multi-channel image frames corresponding to multiple moments; wherein each group of the initial multi-channel image frames includes multiple frames of images at different viewing angles at the same moment, each frame of the image includes a target to be annotated, and the target to be annotated includes a dynamic target and a static target;

[0008] Acquire depth maps corresponding to the multiple groups of initial multi-channel image frames, and three-dimensional prediction frames and prediction lines of the target to be labeled;

[0009] Based on the depth map and the three-dimensional prediction frame of the target to be labeled, the target to be labeled is separated into a dynamic and a static point cloud to obtain a point cloud of the dynamic target and a point cloud of the static target;

[0010] Performing multi-frame stitching on the point clouds of the static target in the multiple groups of initial multi-channel image frames to obtain a complete static point cloud;

[0011] Fusing the point cloud of the dynamic target with the complete static point cloud to obtain a global point cloud;

[0012] The global point cloud is labeled based on the three-dimensional prediction box and the prediction line of the target to be labeled, so as to obtain a three-dimensional point cloud with the target labeled.

[0013] In a technical solution of the above-mentioned method for marking targets, the static targets include static obstacles and / or static markers, the three-dimensional prediction frames are used to mark the dynamic targets and / or the static obstacles, and the prediction lines are used to mark the static markers; the step of obtaining the depth maps corresponding to the multiple groups of initial multi-channel image frames, the three-dimensional prediction frames and prediction lines of the targets to be marked includes:

[0014] Inputting each group of the initial multi-channel image frames into a depth estimation model to obtain a depth map corresponding to each frame of the initial multi-channel image frames;

[0015] Inputting each group of the initial multi-channel image frames into a three-dimensional prediction model to obtain three-dimensional prediction frames and prediction lines of all the objects to be labeled corresponding to the initial multi-channel image frames;

[0016] The three-dimensional prediction box includes geometric information and posture information, and the prediction line includes endpoint position information.

[0017] In a technical solution of the above-mentioned target labeling method, the depth map includes depth information of each pixel; the dynamic and static separation of the target to be labeled is performed based on the depth map and the three-dimensional prediction frame of the target to be labeled, and the point cloud of the dynamic target and the point cloud of the static target are obtained, including:

[0018] Acquire position information of the target to be labeled at different times based on the depth information and the three-dimensional prediction frame of the target to be labeled;

[0019] Based on the position information, the dynamic and static separation of all the objects to be marked is performed to obtain the dynamic objects and the static objects;

[0020] Based on the depth information and the position information corresponding to the dynamic target and the static target, a point cloud of the dynamic target and a point cloud of the static target are obtained.

[0021] In a technical solution of the above-mentioned target labeling method, the step of separating the dynamic and static objects of all the targets to be labeled based on the position information to obtain the dynamic objects and the static objects includes:

[0022] Based on the position information, determining whether the position of each of the to-be-annotated objects changes;

[0023] If the position changes, the target to be marked is determined to be a dynamic target; otherwise, the target to be marked is determined to be a static target;

[0024] The dynamic target includes at least one of a moving life form, a driving device and a flying device.

[0025] In a technical solution of the above target labeling method, the step of performing multi-frame stitching on the point clouds of the static target in the multiple groups of initial multi-channel image frames to obtain a complete static point cloud comprises:

[0026] splicing the point clouds of the static targets in each group of the initial multi-channel image frames to obtain the static target point clouds corresponding to each group of the initial multi-channel image frames;

[0027] The static target point clouds corresponding to the multiple groups of initial multi-channel image frames are superimposed to obtain the complete static point cloud.

[0028] In a technical solution of the above-mentioned target labeling method, fusing the point cloud of the dynamic target with the complete static point cloud to obtain a global point cloud includes:

[0029] Acquire the coordinate information of the point cloud of the dynamic target and the coordinate information of the complete static point cloud;

[0030] Based on the coordinate information, the point cloud of the dynamic target at any moment and the complete static point cloud are fused to obtain the global point cloud corresponding to any moment.

[0031] In a technical solution of the above-mentioned method for marking targets, the point cloud of the dynamic target includes a color point cloud of the dynamic target, the point cloud of the static target includes a color point cloud of the static target, and the three-dimensional point cloud with the marked target includes a three-dimensional color point cloud; the method further includes:

[0032] Determining the quality of the labeled objects in the three-dimensional color point cloud;

[0033] When the annotation quality does not meet the preset requirement, a modification annotation operation or a re-annotation operation is performed on at least the already-annotated target in the three-dimensional color point cloud whose annotation quality does not meet the preset requirement.

[0034] In a technical solution of the above target labeling method, the depth map includes depth information of each pixel; the method further includes:

[0035] Acquire color information of each pixel in the depth map based on the multiple groups of initial multi-channel image frames;

[0036] Based on the color information of each pixel and the depth information, a color point cloud of the dynamic target and a color point cloud of the static target are obtained.

[0037] In a second aspect, an electronic device is provided, which includes a processor and a memory, wherein the memory is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the target marking method described in any one of the technical solutions of the above-mentioned target marking method.

[0038] In a third aspect, a computer-readable storage medium is provided, in which a plurality of program codes are stored, wherein the program codes are suitable for being loaded and run by a processor to execute the target labeling method described in any one of the technical solutions of the target labeling method described above.

[0039] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0040] In the technical solution for implementing the present application, multiple groups of initial multi-channel image frames corresponding to multiple moments can be obtained; wherein each group of initial multi-channel image frames includes multiple frames of images at different perspectives at the same moment, and each frame of image includes a target to be labeled, and the target to be labeled includes a dynamic target and a static target; then, depth maps, three-dimensional prediction boxes and prediction lines of the targets to be labeled corresponding to the multiple groups of initial multi-channel image frames are obtained, and based on the depth maps and the three-dimensional prediction boxes of the targets to be labeled, dynamic and static separation of the targets to be labeled is performed to obtain a point cloud of the dynamic target and a point cloud of the static target, and then the point clouds of the static targets in the multiple groups of initial multi-channel image frames are spliced ​​across multiple frames to obtain a complete static point cloud, and the point cloud of the dynamic target and the complete static point cloud are fused to obtain a global point cloud, and finally, the global point cloud is labeled based on the three-dimensional prediction boxes and prediction lines of the targets to be labeled to obtain a three-dimensional point cloud with labeled targets. Through the above implementation, the dynamic and static separation of the target to be labeled can avoid the loss or error of image information caused by the mutual interference of dynamic and static targets when acquiring multiple groups of initial multi-channel image frames corresponding to multiple moments, thereby improving the accuracy of labeling; the static target point cloud in the separated multiple groups of initial multi-channel image frames is spliced ​​into multiple frames to obtain a complete static point cloud, thereby improving the density and quality of the point cloud; the point cloud of the dynamic target and the complete static point cloud are merged to obtain a global point cloud, which comprehensively and accurately presents the information of the entire scene; finally, the three-dimensional global point cloud can be directly labeled through the three-dimensional prediction box and prediction line of the target, thereby improving the accuracy of data labeling.

[0041] Furthermore, the point cloud of the dynamic target obtained includes the color point cloud of the dynamic target, the point cloud of the static target includes the color point cloud of the static target, and the three-dimensional point cloud with the labeled target includes the three-dimensional color point cloud. The three-dimensional color point cloud can make the semantic reference of the target more accurate, which helps to improve the accuracy of judging the target annotation quality. When the target annotation quality does not meet the preset requirements, the annotation can also be modified according to the color point cloud, such as modifying the three-dimensional prediction box and prediction line of the annotation, or re-annotating, so as to avoid discarding all the annotation data due to one or part of the annotation errors, thereby improving the data conversion rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0043] Figure 1 It is a schematic diagram of the main process of the existing target labeling method;

[0044] Figure 2 It is a schematic flow chart of the main steps of a method for marking a target according to an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of the main process of a method for marking a target according to an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of the main structure of an electronic device according to an embodiment of the present application.

[0047] List of reference numerals:

[0048] 401: processor; 402: memory. DETAILED DESCRIPTION

[0049] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0050] In the description of the present application, "processor" may include hardware, software or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as just A, just B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include just A, just B or A and B. The singular terms "one" and "the" may also include plural forms.

[0051] As described in the background technology, in the field of computer vision, the method of marking objects in a three-dimensional point cloud is described in the attached Figure 1 , Figure 1 This is a schematic diagram of the main process of the existing target labeling method. Figure 1 As shown in the figure, when annotating objects in a 3D point cloud, the 3D prediction box of the original image is usually directly output by the 3D detection model, and then the 3D point cloud is projected onto the 2D image to obtain a 2D image annotated with the 3D prediction box. On this basis, the correctness of the annotated prediction box is manually judged to screen out valid data, providing basic data for subsequent model training and data analysis.

[0052] However, this annotation method has some problems. First, projecting the 3D point cloud onto the 2D image will affect the judgment of the edge position of the target, resulting in large errors in manual judgment of whether the annotation is correct. Second, since the 2D image cannot estimate the depth information of the target, the annotation on the 2D image cannot be modified. Therefore, most of the data can only be discarded if there is one error, resulting in a low conversion rate of the annotated data.

[0053] In order to solve the above problems, the present application provides a target labeling method, an electronic device and a storage medium.

[0054] See attached Figure 2 , Figure 2 FIG. 1 is a flow chart of the main steps of a method for marking a target according to an embodiment of the present application. Figure 2 As shown, the target marking method in the embodiment of the present application mainly includes the following steps S201 to S206.

[0055] Step S201: Acquire multiple groups of initial multi-channel image frames corresponding to multiple moments;

[0056] Each group of initial multi-channel image frames includes multiple frames of images at different viewing angles at the same time, and each frame of image includes a target to be labeled, and the target to be labeled includes a dynamic target and a static target.

[0057] Step S202: obtaining depth maps corresponding to multiple groups of initial multi-channel image frames, three-dimensional prediction boxes and prediction lines of the objects to be marked;

[0058] Step S203: Separating the target to be annotated from the dynamic and static parts based on the depth map and the three-dimensional predicted frame of the target to be annotated, and obtaining a point cloud of the dynamic target and a point cloud of the static target;

[0059] Step S204: performing multi-frame stitching on the point clouds of the static targets in the multiple groups of initial multi-channel image frames to obtain a complete static point cloud;

[0060] Step S205: Fusing the point cloud of the dynamic target with the complete static point cloud to obtain a global point cloud;

[0061] Step S206: annotate the global point cloud based on the three-dimensional prediction box and prediction line of the target to be annotated, and obtain a three-dimensional point cloud with the target annotated.

[0062] Based on the methods described in steps S201 to S206 above, separating the dynamic and static objects to be labeled can avoid missing or erroneous image information caused by mutual interference between dynamic and static objects when acquiring multiple groups of initial multi-channel image frames corresponding to multiple moments, thereby improving the accuracy of labeling; performing multi-frame splicing of the static object point clouds in the separated multiple groups of initial multi-channel image frames can obtain a complete static point cloud, thereby improving the density and quality of the point cloud; fusing the point cloud of the dynamic object with the complete static point cloud can obtain a global point cloud, thereby comprehensively and accurately presenting the information of the entire scene; finally, the three-dimensional global point cloud can be directly labeled through the three-dimensional prediction box and prediction line of the target, thereby improving the accuracy of data labeling.

[0063] The above steps S201 to S206 are further explained below.

[0064] In some implementations of the above step S201, the target scene may be imaged at different times by a multi-channel camera or a scanner or other device to obtain multiple groups of initial multi-channel image frames corresponding to multiple times, wherein each group of initial multi-channel image frames includes multiple frames of images at different viewing angles at the same time, each frame of the image includes a target to be annotated, and the target to be annotated includes a dynamic target and a static target.

[0065] Specifically, dynamic targets include at least one of moving life forms, driving equipment, and flying equipment. Among them, life forms include pedestrians and animals, driving equipment includes motor vehicles and non-motor vehicles, and flying equipment includes drones, etc. Static targets include static obstacles and / or static signs. Among them, static obstacles include cones, water barriers, traffic lights, warning signs, etc., and static signs include zebra crossings, lane lines, road markings, arrows, parking spaces, etc.

[0066] For example, in a traffic scene monitoring application scenario, six cameras can be used to collect images at preset time intervals (such as every 1 second) over a continuous period of time. During each acquisition, the six cameras simultaneously capture the target scene from different perspectives. In this way, each acquisition results in a set of initial multi-channel image frames, each of which contains targets such as pedestrians, driving equipment, and traffic signs.

[0067] By acquiring multiple groups of initial multi-channel image frames corresponding to multiple moments, the scene can be fully captured from multiple time points and multiple perspectives, providing a rich and comprehensive data foundation and avoiding the problem of information loss caused by a single perspective or limited time point.

[0068] The above is a further description of step S201. The following is a further description of step S202.

[0069] In some implementations of the above step S202, depth maps corresponding to multiple groups of initial multi-channel image frames, three-dimensional prediction boxes and prediction lines of the objects to be labeled can be obtained through the following steps S2021 to S2022.

[0070] Step S2021: input each group of initial multi-channel image frames into a depth estimation model to obtain a depth map corresponding to each frame of the initial multi-channel image frames;

[0071] The depth estimation model is a trained model that can infer the depth information of each pixel in the image based on the input image information, such as the DepthAnything model, the ResNet model, etc. After the initial multi-channel image frames are input into the depth estimation model, the model will analyze and process each image frame to obtain the depth map corresponding to each frame in the initial multi-channel image frames.

[0072] A depth map is a special type of image data that includes the depth information of each pixel, which can be expressed in the form of grayscale values ​​to indicate the distance of each pixel in the image relative to the image acquisition device. For example, brighter pixels may indicate points that are closer to the image acquisition device, while darker pixels may indicate points that are farther away from the image acquisition device.

[0073] The above is the description of step S2021.

[0074] Step S2022: input each group of initial multi-channel image frames into the three-dimensional prediction model to obtain the three-dimensional prediction boxes and prediction lines of all the objects to be labeled corresponding to the initial multi-channel image frames.

[0075] Among them, the three-dimensional prediction box is used to mark dynamic targets and / or static obstacles, and the prediction line is used to mark static signs.

[0076] Specifically, multiple frames of images in each group of initial multi-channel image frames can be input into the depth estimation model of the three-dimensional prediction model. The three-dimensional prediction model is a model that has been trained extensively and can predict the position, size, shape and other information of all the targets to be labeled in the image in three-dimensional space based on the input image frame information, such as the BEVDepth model, the SMOKE model, etc. After each group of initial multi-channel image frames is input into the three-dimensional prediction model, the model will fuse the features in the image frames of different perspectives to obtain complete three-dimensional information, and then perform target recognition and positioning based on the fused features. After determining the three-dimensional position of the target, a three-dimensional prediction box and prediction line are generated. The three-dimensional prediction box is a three-dimensional geometric shape that surrounds the target, such as a cuboid, and its size and position are obtained based on the actual three-dimensional state of the target. For static signs, such as zebra crossings, road signs, etc., prediction lines corresponding to these signs will be generated, and finally the three-dimensional prediction boxes and prediction lines of all the targets to be labeled corresponding to the initial multi-channel image frames will be output.

[0077] The 3D prediction frame of the target to be labeled includes geometric information and pose information. The geometric information includes the coordinates of the center point (x, y, z) of the 3D prediction frame, as well as the corresponding length (w), width (h), and height (l). The pose information includes the rotation angles of the 3D prediction frame in the three directions of xyz, which are yaw, pitch, and roll. The prediction line of the target to be labeled includes endpoint position information, such as the coordinates of the two endpoints, which are not limited here.

[0078] Through the depth estimation model and the 3D prediction model, the depth map corresponding to each frame in the initial multi-channel image frame and the 3D prediction boxes and prediction lines of all targets to be labeled can be obtained, providing more realistic spatial dimension information for subsequent target labeling.

[0079] The above is a further description of step S202 , and the following is a further description of step S203 .

[0080] In some implementations of the above step S203, the target to be labeled can be separated into dynamic and static states based on the depth information of each pixel in the depth map and the geometric information and posture information of the three-dimensional prediction box corresponding to the target to be labeled, so as to obtain the point cloud of the dynamic target and the static target.

[0081] It should be pointed out that the target to be labeled corresponding to the prediction line is the static mark in the static target, so there is no need to separate the dynamic and static state of the target to be labeled through the prediction line of the target to be labeled.

[0082] Specifically, step S203 may include the following steps S2031 to S2033.

[0083] Step S2031: acquiring position information of the target to be labeled at different times based on the depth information and the three-dimensional prediction frame of the target to be labeled;

[0084] Among them, the depth map provides the depth information of each pixel, and the three-dimensional prediction frame provides information such as the position, size, and shape of the target to be labeled in three-dimensional space. By combining this depth information with the three-dimensional prediction frame of the target to be labeled, the position information of the target to be labeled at different times can be obtained.

[0085] Step S2032: Separate all the objects to be labeled from dynamic objects and static objects based on the position information to obtain dynamic objects and static objects;

[0086] In some implementations, it may be determined based on the location information whether the location of each target to be marked has changed.

[0087] Specifically, for each target to be labeled, its position information at different times can be compared. For example, the position coordinates (x1, y1, z1) of each target to be labeled at time t1 and the position coordinates (x2, y2, z2) at time t2 are compared, and the degree of position change is quantified by calculating the coordinate difference between time t1 and time t2.

[0088] If the calculated coordinate difference is within a certain error tolerance range, it means that the position of the target remains basically unchanged at different times. On the contrary, if the coordinate difference exceeds the error tolerance range, it means that the position of the target has changed significantly.

[0089] Furthermore, if a position change occurs, the target to be marked is determined to be a dynamic target; otherwise, the target to be marked is determined to be a static target.

[0090] For example, in a traffic scene, if the position coordinates of a moving car change significantly at different times, it will be judged as a dynamic target. If the position coordinates of a trash can do not change at different times, it will be judged as a static target.

[0091] The above is the description of step S2032.

[0092] Step S2033: Based on the depth information and position information corresponding to the dynamic target and the static target, a point cloud of the dynamic target and a point cloud of the static target are obtained.

[0093] After the dynamic targets and static targets have been distinguished, the three-dimensional coordinate points corresponding to each dynamic target or static target can be gathered together using the depth information and position information corresponding to the dynamic targets and static targets to obtain their respective point clouds.

[0094] The point cloud of the dynamic target may include a color point cloud of the dynamic target, and the point cloud of the static target may include a color point cloud of the static target.

[0095] In some implementations, the color information of each pixel in the depth map may be acquired based on multiple groups of initial multi-channel image frames, and based on the color information and depth information of each pixel, a color point cloud of a dynamic target and a color point cloud of a static target may be obtained.

[0096] Specifically, the color information of each frame of the image can be extracted based on multiple groups of initial multi-channel image frames, including the three color channel information of red (R), green (G), and blue (B) for each pixel. Then, the color information of each pixel in the depth map is combined with the depth information to finally obtain the color point cloud of the dynamic target and the color point cloud of the static target.

[0097] By separating the dynamic and static objects to be labeled through the above implementation, it is possible to avoid image information loss or error caused by mutual interference between dynamic and static objects when acquiring multiple groups of initial multi-channel image frames corresponding to multiple moments, thereby improving the accuracy of object labeling.

[0098] The above is a further description of step S203 , and the following is a further description of step S204 .

[0099] In some implementations of the above step S204, the point clouds of static targets in multiple groups of initial multi-channel image frames may be stitched together to obtain a complete static point cloud through the following steps S2041 to S2042.

[0100] Step S2041: stitching the point clouds of the static targets in each group of initial multi-channel image frames to obtain the static target point clouds corresponding to each group of initial multi-channel image frames;

[0101] Specifically, the color point cloud of the static target in each group of initial multi-channel image frames can be spliced ​​according to the three-dimensional coordinates (x, y, z) corresponding to each point in the color point cloud of the static target. During the splicing process, the points in the overlapping area can be directly merged, or weighted fused according to factors such as the distance and density of the points, to finally obtain the color point cloud of the static target corresponding to each group of initial multi-channel image frames.

[0102] Step S2042: superimpose the static target point clouds corresponding to the multiple groups of initial multi-channel image frames to obtain a complete static point cloud.

[0103] Specifically, the coordinates of the static target color point clouds corresponding to the multiple groups of initial multi-channel image frames can be aligned so that they are in a common world coordinate system, and then the multiple groups of static target color point clouds can be directly superimposed after the coordinates are aligned. During the superposition process, the points in the overlapping areas of different groups of initial multi-channel image frames can be processed by weighted averaging method, nearest neighbor method and fusion algorithm, etc., to finally obtain a complete static color point cloud.

[0104] Through the above implementation, the point clouds of static targets corresponding to different perspectives and different times can be integrated to improve the density and quality of the point clouds, while also improving the integrity and presentation accuracy of the static targets in three-dimensional space.

[0105] The above is a further description of step S204. The following is a further description of step S205.

[0106] In some implementations of the above step S205, the point cloud of the dynamic target and the complete static point cloud may be fused through the following steps S2051 to S2052 to obtain a global point cloud.

[0107] Step S2051: obtaining the coordinate information of the point cloud of the dynamic target and the coordinate information of the complete static point cloud;

[0108] In the previous steps, the color point clouds of dynamic targets have been obtained. These point clouds are composed of a large number of points, and each point has its specific coordinate position in three-dimensional space. Therefore, the coordinate information of the color point cloud of the dynamic target and the coordinate information of the complete static color point cloud can be obtained.

[0109] Step S2052: Based on the coordinate information, the point cloud of the dynamic target at any moment and the complete static point cloud are merged to obtain the global point cloud corresponding to any moment.

[0110] Since dynamic targets change over time, their point clouds will also be different at different times. Therefore, the color point cloud of the dynamic target at any moment and the complete static color point cloud can be fused according to the coordinate information to generate a global color point cloud corresponding to any moment, which can fully and accurately present the information of the entire scene.

[0111] The above is a further description of step S205. The following is a further description of step S206.

[0112] In some implementations of the above step S206, the three-dimensional point cloud with the target labeled includes a three-dimensional color point cloud.

[0113] Specifically, the three-dimensional prediction box and prediction line of the target to be labeled obtained in step S202 can be converted to the same coordinate system as the global color point cloud obtained in step S205. In this coordinate system, the three-dimensional prediction box of the target to be labeled is added to the global color point cloud based on the geometric information and posture information, and the prediction line of the target is added to the global color point cloud based on the endpoint position information, and finally a three-dimensional color point cloud with the three-dimensional prediction box and prediction line labeled with the target is obtained.

[0114] Furthermore, in some implementations, the labeling quality of the labeled objects in the three-dimensional color point cloud may also be determined.

[0115] Specifically, the quality of the labeled objects can be judged based on the accuracy, completeness and precision of the 3D prediction boxes and prediction lines in the 3D color point cloud. For example, it can be judged whether the position and geometry of the labeled objects are accurate, whether only part of the object is labeled and other parts are omitted, whether there are objects to be labeled that are omitted, and whether the coordinate precision and size of the labeled objects are accurate.

[0116] The three-dimensional color point cloud in this application can make the semantic reference of the target more accurate, thereby improving the accuracy of judging the target annotation quality.

[0117] Furthermore, when the annotation quality does not meet the preset requirements, a modification annotation operation or a re-annotation operation is performed on at least the already-annotated targets in the three-dimensional color point cloud whose annotation quality does not meet the preset requirements.

[0118] Specifically, when the annotation quality does not meet the preset requirements, the already annotated targets whose annotation quality does not meet the preset requirements can be modified according to the three-dimensional color point cloud, such as modifying the annotated three-dimensional prediction box and prediction line, etc. All targets can also be re-annotated to avoid discarding all annotation data due to one or part of the annotation errors, thereby improving the data conversion rate.

[0119] The above is the description of steps S201 to S206.

[0120] Further, in some embodiments, see the attached Figure 3 , Figure 3 It is a main flow chart of a method for marking a target according to an embodiment of the present application.

[0121] like Figure 3 As shown, firstly, multiple groups of initial multi-channel image frames corresponding to multiple moments are obtained, and then the multiple groups of initial multi-channel image frames are respectively input into the depth estimation model and the 3D prediction model to obtain the depth map corresponding to each frame image in each group of initial multi-channel image frames, and the 3D prediction boxes and prediction lines of all the objects to be labeled corresponding to each group of initial multi-channel image frames;

[0122] Furthermore, based on the depth map and the three-dimensional prediction frame of the target to be labeled, the target to be labeled is separated into dynamic and static parts to obtain the color point cloud of the dynamic target and the color point cloud of the static target. Then, the color point cloud of the static target in multiple groups of initial multi-channel image frames is spliced ​​to obtain a complete static color point cloud. The color point cloud of the dynamic target and the complete static color point cloud are fused to obtain a global color point cloud.

[0123] Furthermore, the global color point cloud is annotated based on the three-dimensional prediction box and prediction line of the target to be annotated to obtain a three-dimensional color point cloud with the target annotated, and the annotation quality of the already annotated targets in the three-dimensional color point cloud is judged. When the annotation quality does not meet the preset requirements, at least the annotation modification operation or the re-annotation operation is performed on the already annotated targets in the three-dimensional color point cloud whose annotation quality does not meet the preset requirements.

[0124] The above is a further explanation of the target marking method provided in this application.

[0125] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present application.

[0126] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0127] Furthermore, the present application also provides an electronic device. Figure 4 , Figure 4 Schematic diagram of the main structure of an electronic device according to an embodiment of the present application. Figure 4As shown, the electronic device in the embodiment of the present application mainly includes a processor 401 and a memory 402. The memory 402 can be configured to store a program for executing the method for marking the target of the above method embodiment, and the processor 401 can be configured to execute the program in the memory 402, which includes but is not limited to the program for executing the method for marking the target of the above method embodiment. For ease of explanation, only the part related to the embodiment of the present application is shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application.

[0128] In some possible implementations of the present application, the electronic device may include multiple processors 401 and multiple memories 402. The program for executing the method for marking the target of the above method embodiment can be divided into multiple subprograms, and each subprogram can be loaded and run by the processor 401 to execute different steps of the method for marking the target of the above method embodiment. Specifically, each subprogram can be stored in different memories 402, and each processor 401 can be configured to execute the program in one or more memories 402 to jointly implement the method for marking the target of the above method embodiment, that is, each processor 401 executes different steps of the method for marking the target of the above method embodiment, to jointly implement the method for marking the target of the above method embodiment.

[0129] The above-mentioned multiple processors 401 may be processors deployed on the same device. For example, the above-mentioned electronic device may be a high-performance device composed of multiple processors, and the above-mentioned multiple processors 401 may be processors configured on the high-performance device. In addition, the above-mentioned multiple processors 401 may also be processors deployed on different devices. For example, the above-mentioned electronic device may be a server cluster, and the above-mentioned multiple processors 401 may be processors on different servers in the server cluster, or the above-mentioned computer device may be a driving device cluster, and the above-mentioned multiple processors 401 may be processors on different driving devices in the driving device cluster.

[0130] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for a method for marking a target of the above-mentioned method embodiment, and the program can be loaded and run by a processor to implement the method for marking the above-mentioned target. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a memory device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0131] It should be noted that the relevant information that may be involved in the embodiments of this application is strictly in accordance with the requirements of laws and regulations, follow the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as personal information obtained with the user's authorization.

[0132] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0133] So far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A method for labeling a target, characterized in that: The method comprises: Acquire multiple groups of initial multi-channel image frames corresponding to multiple moments; wherein each group of the initial multi-channel image frames includes multiple frames of images at different viewing angles at the same moment, each frame of the image includes a target to be annotated, and the target to be annotated includes a dynamic target and a static target; Acquire depth maps corresponding to the multiple groups of initial multi-channel image frames, and three-dimensional prediction frames and prediction lines of the target to be labeled; Based on the depth map and the three-dimensional prediction frame of the target to be labeled, the target to be labeled is separated into a dynamic and a static point cloud to obtain a point cloud of the dynamic target and a point cloud of the static target; splicing the point clouds of the static targets in each group of the initial multi-channel image frames to obtain the static target point clouds corresponding to each group of the initial multi-channel image frames; superimposing the static target point clouds corresponding to the multiple groups of initial multi-channel image frames to obtain the complete static point clouds corresponding to the multiple moments; Fusing the point cloud of the dynamic target at any moment with the complete static point cloud to obtain a global point cloud corresponding to any moment; The global point cloud corresponding to any moment is labeled based on the three-dimensional prediction box and the prediction line of the target to be labeled, so as to obtain a three-dimensional point cloud with the target labeled.

2. The target labeling method according to claim 1, characterized in that: The static target includes a static obstacle and / or a static mark, the three-dimensional prediction frame is used to mark the dynamic target and / or the static obstacle, and the prediction line is used to mark the static mark; The step of obtaining the depth maps corresponding to the multiple groups of initial multi-channel image frames, the three-dimensional prediction frames and the prediction lines of the target to be labeled comprises: Inputting each group of the initial multi-channel image frames into a depth estimation model to obtain a depth map corresponding to each frame of the initial multi-channel image frames; Inputting each group of the initial multi-channel image frames into a three-dimensional prediction model to obtain three-dimensional prediction frames and prediction lines of all the objects to be labeled corresponding to the initial multi-channel image frames; The three-dimensional prediction box includes geometric information and posture information, and the prediction line includes endpoint position information.

3. The target labeling method according to claim 1, characterized in that: The depth map includes depth information of each pixel; the step of performing dynamic and static separation on the target to be labeled based on the depth map and the three-dimensional prediction frame of the target to be labeled to obtain a point cloud of the dynamic target and a point cloud of the static target includes: Acquire position information of the target to be labeled at different times based on the depth information and the three-dimensional prediction frame of the target to be labeled; Based on the position information, the dynamic and static separation of all the objects to be marked is performed to obtain the dynamic objects and the static objects; Based on the depth information and the position information corresponding to the dynamic target and the static target, a point cloud of the dynamic target and a point cloud of the static target are obtained.

4. The target labeling method according to claim 3, characterized in that: The performing dynamic and static separation of all the objects to be marked based on the position information to obtain the dynamic objects and the static objects comprises: Based on the position information, determining whether the position of each of the to-be-annotated objects changes; If the position changes, the target to be marked is determined to be a dynamic target; otherwise, the target to be marked is determined to be a static target; The dynamic target includes at least one of a moving life form, a driving device and a flying device.

5. The target labeling method according to claim 1, characterized in that: The step of fusing the point cloud of the dynamic target at any moment with the complete static point cloud to obtain the global point cloud corresponding to any moment comprises: Acquire the coordinate information of the point cloud of the dynamic target and the coordinate information of the complete static point cloud; Based on the coordinate information, the point cloud of the dynamic target at any moment and the complete static point cloud are fused to obtain a global point cloud corresponding to any moment.

6. The method for marking a target according to any one of claims 1 to 5, characterized in that: The point cloud of the dynamic target includes a color point cloud of the dynamic target, the point cloud of the static target includes a color point cloud of the static target, and the three-dimensional point cloud with the target marked includes a three-dimensional color point cloud; the method further includes: Determining the quality of the labeled objects in the three-dimensional color point cloud; When the annotation quality does not meet the preset requirement, a modification annotation operation or a re-annotation operation is performed on at least the already-annotated target in the three-dimensional color point cloud whose annotation quality does not meet the preset requirement.

7. The target labeling method according to claim 6, characterized in that: The depth map includes depth information of each pixel; the method further includes: Acquire color information of each pixel in the depth map based on the multiple groups of initial multi-channel image frames; Based on the color information of each pixel and the depth information, a color point cloud of the dynamic target and a color point cloud of the static target are obtained.

8. An electronic device, comprising a processor and a memory, wherein the memory is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by the processor to execute the target marking method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the target marking method according to any one of claims 1 to 7.

Images