A fusion labeling method, device, apparatus and storage medium

Abstract

The application provides a fusion labeling method, device and equipment and a storage medium. The method comprises the following steps: obtaining an initial three-dimensional point cloud and an initial two-dimensional image, synchronizing data of the initial three-dimensional point cloud and the initial two-dimensional image, obtaining a three-dimensional point cloud to be labeled and a two-dimensional image to be labeled, determining a three-dimensional target frame based on the three-dimensional point cloud to be labeled, mapping the three-dimensional target frame, obtaining a pseudo three-dimensional target frame in the two-dimensional image to be labeled, performing first classification identification on a plurality of target point clouds in the three-dimensional target frame, performing second classification identification on a plurality of target images in the pseudo three-dimensional target frame, inputting point cloud identification information and image identification information obtained through identification into a decision maker for integrated learning, and performing fusion labeling according to an output result of the decision maker. The three-dimensional point cloud and the two-dimensional image are identified by a weak classifier, and the decision maker is used to make decisions on the respective identification results, so that the requirements of the labeling process on the classification algorithm capability and data resources are effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data labeling, in particular to a fusion labeling method and device, equipment and storage medium. BACKGROUND

[0002] With the development of science and technology, automatic driving has become a development goal in the future. However, the data required by the automatic driving algorithm is very large, and it is seriously dependent on high-quality labeled data. Data labeling is highly dependent on manual work, and the labeling cost is high. Therefore, many companies have invested in research in data labeling tools, auxiliary labeling algorithms, and automatic labeling algorithms in order to improve labeling efficiency through tools and algorithms.

[0003] In the visual perception algorithm, the collected data is mainly two kinds, one is point cloud data, and the other is picture data. In order to improve the efficiency, a fusion labeling method based on three-dimensional point cloud and two-dimensional image is proposed. However, due to the large amount of data, the existing fusion labeling method has high requirements on the performance of a single algorithm large model, and requires a large amount of data and resources for training. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the present application provides a fusion labeling method, device, equipment and storage medium to solve the technical problems of high requirements on the performance of a single algorithm large model and the need for a large amount of data and resources for training.

[0005] The fusion labeling method provided by the present application comprises: obtaining an initial three-dimensional point cloud and an initial two-dimensional image; synchronizing the data of the initial three-dimensional point cloud and the initial two-dimensional image to obtain a three-dimensional point cloud to be labeled and a two-dimensional image to be labeled; determining a three-dimensional target frame based on the three-dimensional point cloud to be labeled, and mapping the three-dimensional target frame to obtain a pseudo-three-dimensional target frame in the two-dimensional image to be labeled; performing first classification identification on a plurality of target point clouds in the three-dimensional target frame based on different point cloud classes to obtain point cloud identification information, and performing second classification identification on a plurality of target images in the pseudo-three-dimensional target frame based on different image classes to obtain image identification information; inputting the point cloud identification information and the image identification information into a decision maker for integrated learning to perform fusion labeling according to the output result of the decision maker.

[0006] In an embodiment of the present application, the data synchronization of the initial three-dimensional point cloud and the initial two-dimensional image comprises: spatial synchronization of the initial three-dimensional point cloud and the initial two-dimensional image to obtain the initial three-dimensional point cloud and the initial two-dimensional image of the same target scene; time synchronization processing of the initial three-dimensional point cloud and the initial two-dimensional image in the target scene to obtain the initial three-dimensional point cloud and the initial two-dimensional image of the same moment in the target scene; determination of any group of initial three-dimensional point cloud and initial two-dimensional image of the same moment in the target scene as the to-be-labeled information, the initial three-dimensional point cloud in the to-be-labeled information as the to-be-labeled point cloud, and the initial two-dimensional image in the to-be-labeled information as the to-be-labeled two-dimensional image.

[0007] In an embodiment of the present application, the mapping of the three-dimensional target frame to obtain the pseudo three-dimensional target frame in the to-be-labeled two-dimensional image comprises: obtaining the mapping relationship between the initial three-dimensional point cloud and the initial two-dimensional image; based on the mapping relationship, mapping the three-dimensional target frame to the to-be-labeled two-dimensional image to obtain the pseudo three-dimensional recognition frame on the to-be-labeled two-dimensional image.

[0008] In an embodiment of the present application, the second classification and recognition of the multi-class target image in the pseudo three-dimensional target frame based on different picture classes comprises: determining the image region in the pseudo three-dimensional recognition frame as a region of interest image; inputting the region of interest image into a preset image recognition classifier to obtain an image recognition parameter and an image recognition confidence based on the preset image recognition classifier; traversing a plurality of preset image recognition classifiers to obtain a plurality of image recognition parameters and an image recognition confidence corresponding to each image recognition parameter; determining any target image recognition parameter and the image recognition confidence corresponding to the target image recognition parameter as a group of image recognition information, and traversing each group of image recognition parameters to obtain a plurality of groups of image recognition information.

[0009] In an embodiment of the present application, the first classification and recognition of the plurality of target point clouds in the three-dimensional target frame based on different point cloud classes comprises: determining the point cloud in the three-dimensional recognition frame as a point cloud of interest, and inputting the point cloud of interest into a preset point cloud recognition classifier to obtain a point cloud recognition parameter and a point cloud recognition confidence based on the point cloud recognition classifier; traversing a plurality of preset point cloud recognition classifiers to obtain a plurality of point cloud recognition parameters and a point cloud recognition confidence corresponding to each point cloud recognition parameter; determining any target point cloud recognition parameter and the point cloud recognition confidence corresponding to the target point cloud recognition parameter as a group of point cloud recognition information, and traversing each group of point cloud recognition parameters to obtain a plurality of groups of point cloud recognition information.

[0010] In an embodiment of the present application, the fusion labeling according to the output result of the decision maker comprises: obtaining the output result of the decision maker, the output result comprising a target object three-dimensional bounding box and target object attribute information; mapping the target object three-dimensional bounding box to obtain a target object two-dimensional bounding box of the target object on a to-be-labeled two-dimensional image; and performing fusion labeling on the target object based on the target object two-dimensional bounding box and the target object attribute information.

[0011] The present application provides a fusion labeling device, the device comprising: an information acquisition module for obtaining an initial three-dimensional point cloud and an initial two-dimensional image; a data synchronization module for synchronizing data of the initial three-dimensional point cloud and the initial two-dimensional image to obtain a to-be-labeled three-dimensional point cloud and a to-be-labeled two-dimensional image; a target determination module for determining a three-dimensional target box based on the to-be-labeled three-dimensional point cloud and mapping the three-dimensional target box to obtain a pseudo three-dimensional target box in the to-be-labeled two-dimensional image; a weak classifier module for performing first classification recognition on a plurality of target point clouds in the three-dimensional target box based on different point cloud classes to obtain point cloud recognition information, and performing second classification recognition on a plurality of target images in the pseudo three-dimensional target box based on different image classes to obtain image recognition information; and a fusion labeling module for inputting the point cloud recognition information and the image recognition information into a decision maker for integrated learning to perform fusion labeling according to an output result of the decision maker.

[0012] In an embodiment of the present application, the weak classifier module comprises: an image recognition classifier for obtaining image recognition parameters of a region of interest image and image recognition confidence of the region of interest image based on the input region of interest image; and a point cloud recognition classifier for obtaining point cloud recognition parameters of a point cloud of interest and point cloud recognition confidence of the point cloud of interest based on the input point cloud of interest.

[0013] The present application provides an electronic device comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the fusion labeling method as described above.

[0014] The present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, causes the computer to perform the fusion labeling method as described above.

[0015] The beneficial effects of the present application: the fusion labeling method, device, equipment and storage medium in the present application, through multiple weak classifiers, respectively identify three-dimensional point cloud and two-dimensional image, and input the results of multiple weak classifiers into a decision maker to obtain the final recognition result, and according to the final recognition result, the target object is fused and labeled, in the process of fusion labeling, the idea of ensemble learning is introduced, multiple algorithms are used to identify images and point clouds respectively, then multiple point cloud algorithms and multiple image algorithms are fused, and the mutual mapping relationship between point cloud and picture is used, which effectively improves the efficiency of fusion labeling.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the present application, and together with the specification, serve to explain the principles of the present application. It is obvious that the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. In the drawings:

[0018] Figure 1 is an implementation environment diagram for fusion labeling of target objects shown by an exemplary embodiment of the present application;

[0019] Figure 2 is a fusion labeling flowchart shown by an exemplary embodiment of the present application;

[0020] Figure 3 is a weak classifier learning framework shown by an exemplary embodiment of the present application;

[0021] Figure 4 is a fusion labeling flowchart based on ensemble learning shown by an exemplary embodiment of the present application;

[0022] Figure 5 is a fusion labeling flowchart based on ensemble learning shown by another exemplary embodiment of the present application;

[0023] Figure 6 is an ensemble learning decision maker algorithm network training diagram shown by an exemplary embodiment of the present application;

[0024] Figure 7 is a block diagram of a fusion labeling device shown by an exemplary embodiment of the present application;

[0025] Figure 8 shows a structural schematic diagram of a computer system of an electronic device suitable for realizing the embodiments of the present application. DETAILED DESCRIPTION

[0026] Other advantages and novel features of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein:

[0027] It should be noted that the drawings included herewith are included merely for purposes of illustration and are not intended to limit the scope of the present application. The drawings are in fact a simplification of the structures being described and are included to explain specific structures and how they interrelate to each other. The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0028] In the following description, numerous specific details are discussed in order to provide a thorough understanding of the embodiments of the present application. However, those of ordinary skill in the art will realize that the embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid obscuring the application.

[0029] First of all, it needs to be pointed out that the characteristics of 3D data are mainly spatial, which can be freely rotated for viewing, and has depth information of objects. The disadvantage is that the point cloud data is sparse and easy to be blocked, and the annotator needs to fill in the missing point cloud category, attribute and target box. Considering that the point cloud is always sparse, it is difficult to directly identify what the car belongs to, thus leading to the problem of brain supplementing the length, attribute and category of the car, and often the object category on the point cloud cannot be directly judged, and the picture at the same time needs to be used for auxiliary judgment, so the results of the picture algorithm are considered to be applied to the point cloud annotation.

[0030] Figure 1 is a schematic diagram of an implementation environment for fusion annotation of a target object according to an example embodiment of the present application. As shown in Figure 1As shown, in the process of fusion labeling of the target object, the implementation environment includes a three-dimensional point cloud acquisition device 101, a two-dimensional image acquisition device 102, and a computer device 103. The three-dimensional point cloud acquisition device 101 can be a laser radar scanning device, the two-dimensional image acquisition device 102 can be an optical camera device, and the three-dimensional point cloud acquisition device 101 and the two-dimensional image acquisition device 102 can also be any device that can realize three-dimensional information scanning or two-dimensional information acquisition integrated in the same device. The present application does not make any limitation on this. The computer device 103 can be at least one of a desktop graphic processing unit (GPU) computer, a GPU computing cluster, a neural network computer, etc. Related technical personnel can obtain three-dimensional point cloud data and two-dimensional image information of a target scene containing a target object through the three-dimensional point cloud acquisition device 101 and the two-dimensional image acquisition device 102, process the related data through the computer device 103, obtain image recognition information based on two-dimensional image information and point cloud recognition information based on three-dimensional point cloud information, and process the image recognition information and the point cloud recognition information to obtain related information of the target object, so as to fusion label the target object.

[0031] As shown in the example embodiment, Figure 2 The fusion labeling method at least includes steps S210 to S250, which are described in detail as follows:

[0032] Step S210: obtaining initial three-dimensional point cloud and initial two-dimensional image.

[0033] In an embodiment of the present application, a vehicle is taken as a target object, and a parking lot where the vehicle is located is taken as a target environment. The three-dimensional point cloud of the target scene is acquired through a laser radar scanning device, and the two-dimensional image of the target scene is acquired through an optical camera device. The acquired three-dimensional point cloud and two-dimensional image are determined as the initial three-dimensional point cloud and the initial two-dimensional image.

[0034] It should be understood that, considering the characteristics that the two-dimensional image information cannot reflect the three-dimensional structure and the three-dimensional point cloud information cannot reflect the plane information, the combination of the two-dimensional image information and the three-dimensional point cloud information can effectively improve the labeling efficiency in the fusion labeling scene of simultaneously labeling 3D point cloud and 2D picture.

[0035] Step S220: synchronizing data of the initial three-dimensional point cloud and the initial two-dimensional image to obtain a to-be-labeled three-dimensional point cloud and a to-be-labeled two-dimensional picture.

[0036] It should be noted that after the two-dimensional image and the three-dimensional point cloud are collected, the two-dimensional image and the three-dimensional point cloud need to be preprocessed, including but not limited to analysis, frame extraction, merging, etc., to obtain two-dimensional image information and three-dimensional point cloud information available for fusion labeling of the target object. In addition, the obtained three-dimensional point cloud information and two-dimensional image information are also subjected to synchronous processing, and the two-dimensional image corresponding to each time of the to-be-labeled 3D point cloud data is synchronized out, and the one-to-one correspondence is prepared, to obtain the to-be-labeled information at the same time in the same scene.

[0037] In an embodiment of the present application, the data synchronization of the initial three-dimensional point cloud and the initial two-dimensional image includes: spatial synchronization of the initial three-dimensional point cloud and the initial two-dimensional image to obtain the initial three-dimensional point cloud and the initial two-dimensional image of the same target scene; time synchronization processing of the initial three-dimensional point cloud and the initial two-dimensional image in the target scene to obtain the initial three-dimensional point cloud and the initial two-dimensional image at the same time in the target scene; determination of any group of initial three-dimensional point cloud and initial two-dimensional image at the same time in the target scene as to-be-labeled information, determination of the initial three-dimensional point cloud in the to-be-labeled information as to-be-labeled point cloud, and determination of the initial two-dimensional image in the to-be-labeled information as to-be-labeled two-dimensional image.

[0038] In step S230, a three-dimensional target frame is determined based on the to-be-labeled three-dimensional point cloud, and the three-dimensional target frame is mapped to obtain a pseudo three-dimensional target frame in the to-be-labeled two-dimensional image.

[0039] In an embodiment of the present application, taking a laser radar scanning device as a three-dimensional point cloud collection device and an optical camera device as a two-dimensional image collection device as an example, after the laser radar scanning device and the optical camera device are erected in the target environment, the mapping relationship between the three-dimensional point cloud information collected by the laser radar scanning device and the two-dimensional image information collected by the optical camera device needs to be determined, and the mapping relationship determination method is as follows: obtaining the extrinsic parameters of the laser radar scanning device, the extrinsic parameters of the optical camera device, and the intrinsic parameters of the optical camera device; determining a first offset and a first rotation of the laser radar scanning device based on the extrinsic parameters of the laser radar scanning device, determining a second offset and a second rotation of the optical camera device based on the extrinsic parameters of the optical camera device, and generating a matrix based on the intrinsic parameters of the optical camera device; calculating the mapping relationship between the to-be-labeled three-dimensional point cloud and the to-be-labeled two-dimensional image according to the first offset, the first rotation, the second offset, the second rotation, and the matrix; after obtaining the mapping relationship between the to-be-labeled three-dimensional point cloud and the to-be-labeled two-dimensional image, a rough three-dimensional target frame is determined in the three-dimensional point cloud based on the approximate position of the target object, and the three-dimensional target frame is mapped to the two-dimensional image according to the calculated mapping relationship to obtain a pseudo three-dimensional target frame.

[0040] It should be understood that the mapping of the three-dimensional target frame to obtain the pseudo three-dimensional target frame in the to-be-labeled two-dimensional image includes: obtaining a mapping relationship between the initial three-dimensional point cloud and the initial two-dimensional image; based on the mapping relationship, mapping the three-dimensional target frame to the to-be-labeled two-dimensional image to obtain a pseudo three-dimensional recognition frame on the to-be-labeled two-dimensional image.

[0041] In an embodiment of the present application, the pseudo three-dimensional target frame obtained above is an irregular frame including 8 vertices, the maximum area surrounded by the 8 vertices is taken as the region of interest, and the image obtained in the region of interest is determined as the region of interest image. In addition, the region where the three-dimensional target frame is located is determined as the three-dimensional region of interest, and the point cloud in the three-dimensional region of interest is determined as the point cloud of interest.

[0042] In step S240, the point cloud recognition information is obtained by performing first classification recognition on the multiple target point clouds in the three-dimensional target frame based on different point cloud classes, and the image recognition information is obtained by performing second classification recognition on the multiple target images in the pseudo three-dimensional target frame based on different image classes.

[0043] Figure 3 It is a weak classifier learning framework shown in an exemplary embodiment of the present application, as shown in Figure 3 The weak classifier learning framework for fusion labeling of target objects includes multiple individual learners such as individual learner 1 and individual learner 2, one decision maker, and one learner.

[0044] The second classification recognition on the multiple target images in the pseudo three-dimensional target frame based on different image classes includes: determining the image region in the pseudo three-dimensional recognition frame as the region of interest image; inputting the region of interest image into a preset image recognition classifier to obtain image recognition parameters and image recognition confidence based on the preset image recognition classifier; traversing multiple preset image recognition classifiers to obtain multiple image recognition parameters and image recognition confidence corresponding to each image recognition parameter; determining any target image recognition parameter and the image recognition confidence corresponding to the target image recognition parameter as a group of image recognition information, and traversing each group of image recognition parameters to obtain multiple groups of image recognition information.

[0045] In one embodiment of the present application, the region of interest image is input into a plurality of different categories of individual learners, and the recognition results of the same region of interest image are obtained based on different individual learners, the recognition results including a plurality of attribute information of the target object, such as volume, shape, color, etc., the information categories of the recognition results output by each individual learner being the same, but the advantages based on different individual learners being different, and the recognition results further including the confidence of each attribute information. Taking three different individual learners as an example, the region of interest image is input into learner 1, learner 2 and learner 3 respectively, and the recognition results of attribute information A and the confidence of the target object in the region of interest image are “A1, confidence a1”, “A2, confidence a2” and “A3, confidence a3” respectively, the recognition results of attribute information B and the confidence are “B1, confidence b1”, “B2, confidence b2” and “B3, confidence b3” respectively, and the recognition results of attribute information C and the confidence are “C1, confidence c1”, “C2, confidence c2” and “C3, confidence c3” respectively. One recognition result and one confidence corresponding thereto are determined as one set of image recognition information, such as “A1, confidence a1”, thereby obtaining a plurality of image recognition information.

[0046] The plurality of target point clouds in the three-dimensional target frame are subjected to first classification recognition based on different point cloud categories, including: determining the point cloud in the three-dimensional recognition frame as a point cloud of interest, and inputting the point cloud of interest into a preset point cloud recognition classifier to obtain a point cloud recognition parameter and a point cloud recognition confidence based on the point cloud recognition classifier; traversing a plurality of preset point cloud recognition classifiers to obtain a plurality of point cloud recognition parameters and a point cloud recognition confidence corresponding to each point cloud recognition parameter; and determining any target point cloud recognition parameter and the point cloud recognition confidence corresponding to the target point cloud recognition parameter as one set of point cloud recognition information, and traversing each set of point cloud recognition parameters to obtain a plurality of sets of point cloud recognition information.

[0047] In one embodiment of the present application, the region of interest image is input into learner 4, learner 5 and learner 6 respectively, and the recognition results of attribute information A and the confidence of the target object in the region of interest image are “A4, confidence a4”, “A5, confidence a5” and “A6, confidence a6” respectively, the recognition results of attribute information B and the confidence are “B4, confidence b4”, “B5, confidence b5” and “B6, confidence b6” respectively, and the recognition results of attribute information C and the confidence are “C4, confidence c4”, “C5, confidence c5” and “C6, confidence c6” respectively. One recognition result and one confidence corresponding thereto are determined as one set of image recognition information, such as “A4, confidence a5”, thereby obtaining a plurality of image recognition information.

[0048] In step S250, the point cloud recognition information and the image recognition information are input into a decision maker for integrated learning, so as to perform fusion labeling according to the output result of the decision maker.

[0049] According to the output result of the decision maker, the fusion labeling is performed, including: obtaining the output result of the decision maker, the output result including a target object three-dimensional recognition box and target object attribute information; mapping the target object three-dimensional recognition box to obtain a target object two-dimensional recognition box of the target object on the to-be-labeled two-dimensional image; and performing fusion labeling on the target object based on the target object two-dimensional recognition box and the target object attribute information.

[0050] In an embodiment of the present application, the recognition results "A1, confidence a1" "A2, confidence a2" "A3, confidence a3" "A4, confidence a4" "A5, confidence a5" "A6, confidence a6", "B1, confidence b1" "B2, confidence b2" "B3, confidence b3" "B4, confidence b4" "B5, confidence b5" "B6, confidence b6", "C1, confidence c1" "C2, confidence c2" "C3, confidence c3" "C4, confidence c4" "C5, confidence c5" "C6, confidence c6" obtained by the above individual learners are all input into the decision maker, the recognition result "A0, B0, C0" and the target object three-dimensional recognition box X are obtained, the target object three-dimensional recognition box X is mapped to the labeled two-dimensional image based on the mapping relationship between the three-dimensional point cloud and the two-dimensional image, the target object two-dimensional recognition box Y is obtained, and then the target object is fusion labeled based on the recognition result "A0, B0, C0" and the target object two-dimensional recognition box Y.

[0051] In an embodiment of the present application, the Stacking algorithm in ensemble learning is used, that is, various classifiers are stacked. The working mechanism of Stacking is to first train multiple different models, that is, multiple target recognition models prepared in advance, and then input the outputs of the previously trained models, that is, the outputs of the various target recognition models, to train a decision model to obtain a final output.

[0052] Figure 4 is the ensemble learning decision maker algorithm network training diagram shown in an exemplary embodiment of the present application, as Figure 4 shown, first, the picture recognition result is obtained based on multiple picture class target recognition algorithms, and multiple point cloud recognition results are obtained based on multiple point cloud class recognition algorithms; then the obtained picture recognition result and point cloud recognition result are input into the decision network, and the mapping relationship between the point cloud and the image (that is, the calibration parameter) is input into the decision network, the decision result of the decision maker network is obtained, the loss value (that is, the loss value) of the decision maker network is calculated based on the world point cloud labeling result (that is, the point cloud labeling GT), and the decision maker network is trained and calibrated based on the loss value.

[0053] It should be understood that after the target object is fusedly labeled, the to-be-labeled point cloud and the to-be-labeled two-dimensional image are trained based on the result of the fused labeling to obtain a target object recognition model, which can be used for recognizing and labeling new to-be-labeled images and to-be-labeled point clouds.

[0054] In an embodiment of the present application, after obtaining the target object recognition model, the target object is manually labeled to obtain target object actual information, and the target object information obtained based on the target object recognition model is determined as target object predicted information; the target object actual information and the target object predicted information are compared, and if the target object actual information and the target object predicted information are different, the image recognition information and the point cloud recognition information corresponding to the target object predicted information are labeled based on the target object actual information.

[0055] Figure 5 is a fusion labeling process diagram based on ensemble learning shown by an exemplary embodiment of the present application, as shown in Figure 5 The two-dimensional image and the three-dimensional point cloud are first obtained based on the data acquisition device; then the obtained data is analyzed, and the two-dimensional image and the three-dimensional point cloud are synchronized; then the point cloud is roughly framed in the labeling system, and the pseudo-three-dimensional target frame is obtained based on the mapping relationship (i.e., the calibration parameter) between the point cloud and the two-dimensional image; the two-dimensional image in the pseudo-three-dimensional target frame is input into multiple weak classifiers of picture class target recognition, the point cloud in the three-dimensional target frame is input into multiple weak classifiers of point cloud class target recognition, and the output results of the above weak classifiers are taken as the input results of the inheritance learning decision maker to obtain a comprehensive recognition result, and the target object is fusedly labeled based on the mapping relationship between the point cloud and the two-dimensional image and the comprehensive recognition result obtained above.

[0056] Figure 6 is a fusion labeling process diagram based on ensemble learning shown by another exemplary embodiment of the present application, as shown in Figure 6 The point cloud is labeled (the point cloud is sparse and cannot determine the attribute, and is roughly framed); then the corresponding rough target frame on the picture is mapped and input into multiple weak classifiers of picture class target recognition, and the rough target frame on the point cloud data is input into multiple weak classifiers of point cloud class target recognition; then the output results of the weak classifiers of the above classifiers are taken as the input results of the inheritance learning decision maker to obtain the labeling result on the point cloud, and the labeling result on the picture is obtained based on the above labeling result on the point cloud, so as to realize the fusion labeling of the target object.

[0057] In an embodiment of the present application, taking labeling a vehicle as an example, the labeler first draws a rough large frame in the target area that may be a vehicle, and the point cloud in the selected area of the frame is directly sent to each point cloud recognition algorithm for target recognition, and each point cloud algorithm gives its own algorithm output; the second step of the labeling system maps the selected rough 3D target frame back to the corresponding position of the picture to form a rough target frame on the 2D picture, the target frame is used for picture cropping, and is input into each picture type target recognition algorithm, and each picture algorithm gives its own algorithm output; the third step inputs the output of each point cloud algorithm and each picture algorithm into the decision algorithm for result decision, and finally outputs the type, attribute and 3D point cloud target frame of the point cloud target, and finally the 3D point cloud target frame is mapped back to the corresponding position of the 2D picture to complete labeling.

[0058] It should be understood that the fusion labeling method proposed in the present application first takes the type, attribute and target frame output by the picture type target recognition algorithm, plus the corresponding confidence (confidence) as an input part, and the point cloud type target recognition algorithm is the same. Since the picture algorithm outputs a pseudo 3D frame and the point cloud algorithm outputs a 3D frame, in order to avoid differences, the calibration parameter is considered as part of the output of the decision maker algorithm. Then, the decision maker takes the output results of various algorithms and the calibration parameter as the input of its own network, and takes the real labeled point cloud labeling result as the label value for training. Finally, a "smart decision maker" that can integrate the respective "advantages" of various target recognition algorithms is obtained through training, so that in the process of fusion labeling of target objects, each target recognition algorithm is no longer trained, and the training resources are reduced.

[0059] Figure 7 is a block diagram of a fusion labeling device shown in an exemplary embodiment of the present application. The device can be applied to Figure 7 the implementation environment shown. The device can also be applied to other exemplary implementation environments and specifically configured in other devices, and the present embodiment does not limit the implementation environment to which the device is applied.

[0060] As Figure 7 shown, the exemplary fusion labeling device includes an information acquisition module 710, a data synchronization module 720, a target determination module 730, a weak classifier module 740 and a fusion labeling module 750.

[0061] The information collection module 710 is configured to acquire an initial three-dimensional point cloud and an initial two-dimensional image; the data synchronization module 720 is configured to perform data synchronization on the initial three-dimensional point cloud and the initial two-dimensional image to obtain a three-dimensional point cloud to be labeled and a two-dimensional image to be labeled; the target determination module 730 is configured to determine a three-dimensional target frame based on the three-dimensional point cloud to be labeled, and perform mapping on the three-dimensional target frame to obtain a pseudo three-dimensional target frame in the two-dimensional image to be labeled; the weak classifier module 740 is configured to perform first classification recognition on a plurality of target point clouds in the three-dimensional target frame based on different point cloud classes to obtain point cloud recognition information, and perform second classification recognition on a plurality of target images in the pseudo three-dimensional target frame based on different image classes to obtain image recognition information; and the fusion labeling module 750 is configured to input the point cloud recognition information and the image recognition information into a decision maker for integrated learning, so as to perform fusion labeling according to an output result of the decision maker.

[0062] In an embodiment of the present application, the weak classifier module further comprises: an image recognition classifier configured to obtain image recognition parameters of the region of interest image and image recognition confidence of the region of interest image based on the input region of interest image; and a point cloud recognition classifier configured to obtain point cloud recognition parameters of the point cloud of interest and point cloud recognition confidence of the point cloud of interest based on the input point cloud of interest.

[0063] It should be noted that the fusion labeling apparatus provided in the above embodiments and the fusion labeling method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here again. In actual application, the functions of the above embodiments can be distributed to different functional modules to complete all or part of the functions described above, i.e., the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, and this will not be limited here.

[0064] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage apparatus configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the fusion labeling method provided in each of the above embodiments.

[0065] Figure 8 The structure schematic diagram of the computer system of the electronic device suitable for implementing the embodiments of the present application is shown. It should be noted that, Figure 8 The computer system 800 of the electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0066] As Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801 which can perform various suitable actions and processes in accordance with programs stored in a read-only memory (ROM) 802 or loaded from a storage section 808 into a random access memory (RAM) 803, such as performing the methods described in the above embodiments. Various programs and data required for the operation of the system are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0067] Connected to the I / O interface 805 are an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as necessary. A removable recording medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 810 as necessary, so that a computer program read therefrom is installed into the storage section 808 as necessary.

[0068] In particular, in accordance with embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising computer programs for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable recording medium 811. When the computer program is executed by the central processing unit (CPU) 801, various functions defined in the system of the present application are performed.

[0069] It should be noted that the computer-readable medium in the embodiments shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable signal medium can include a data signal propagating in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in connection with an instruction execution system, device or component. The computer program contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0070] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0071] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0072] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor of a computer, the computer performs the fusion labeling method as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0073] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the fusion labeling method provided in each of the above embodiments.

[0074] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought of the present application should be covered by the claims of the present application.

Claims

1. A method for fusion annotation, characterized in that, The method comprises: acquiring an initial three-dimensional point cloud and an initial two-dimensional image; synchronizing data of the initial three-dimensional point cloud and the initial two-dimensional image to obtain a three-dimensional point cloud to be labeled and a two-dimensional image to be labeled; determining a three-dimensional target frame based on the three-dimensional point cloud to be labeled, and mapping the three-dimensional target frame to obtain a pseudo three-dimensional target frame in the two-dimensional image to be labeled; performing first classification recognition on a plurality of target point clouds in the three-dimensional target frame based on different point cloud classes to obtain point cloud recognition information, and performing second classification recognition on a plurality of target images in the pseudo three-dimensional target frame based on different image classes to obtain image recognition information; inputting the point cloud recognition information and the image recognition information into a decision maker for integrated learning to obtain corresponding recognition results and a three-dimensional recognition frame, so as to perform fusion labeling according to an output result of the decision maker, wherein the image recognition information comprises image recognition parameters and corresponding image recognition confidence, and the point cloud recognition information comprises point cloud recognition parameters and corresponding point cloud recognition confidence.

2. The fusion labeling method of claim 1, wherein, The data synchronization of the initial three-dimensional point cloud and the initial two-dimensional image comprises: performing spatial synchronization on the initial three-dimensional point cloud and the initial two-dimensional image to obtain the initial three-dimensional point cloud and the initial two-dimensional image of a same target scene; performing time synchronization processing on the initial three-dimensional point cloud and the initial two-dimensional image in the target scene to obtain the initial three-dimensional point cloud and the initial two-dimensional image at a same time in the target scene; determining any group of the initial three-dimensional point cloud and the initial two-dimensional image at the same time in the target scene as labeled information, and determining the initial three-dimensional point cloud in the labeled information as the three-dimensional point cloud to be labeled, and determining the initial two-dimensional image in the labeled information as the two-dimensional image to be labeled.

3. The method of claim 1, wherein, The mapping of the three-dimensional target frame to obtain the pseudo three-dimensional target frame in the two-dimensional image to be labeled comprises: acquiring a mapping relationship between the initial three-dimensional point cloud and the initial two-dimensional image; mapping the three-dimensional target frame to the two-dimensional image to be labeled based on the mapping relationship to obtain the pseudo three-dimensional target frame on the two-dimensional image to be labeled.

4. The method of claim 1, wherein, The second classification recognition of a plurality of target images in the pseudo three-dimensional target frame based on different image classes comprises: determining an image region in the pseudo three-dimensional target frame as a region of interest image; inputting the region of interest image into a preset image recognition classifier to obtain image recognition parameters and image recognition confidence based on the preset image recognition classifier; iterating a plurality of preset image recognition classifiers to obtain a plurality of image recognition parameters and image recognition confidence corresponding to each image recognition parameter; determining any target image recognition parameter and image recognition confidence corresponding to the target image recognition parameter as a group of image recognition information, and iterating each group of image recognition parameters to obtain a plurality of groups of image recognition information.

5. The method of claim 1, wherein, The first classification recognition of a plurality of target point clouds in the three-dimensional target frame based on different point cloud classes comprises: The point cloud in the three-dimensional target frame is determined as a point cloud of interest, and the point cloud of interest is input into a preset point cloud recognition classifier to obtain a point cloud recognition parameter and a point cloud recognition confidence based on the point cloud recognition classifier; A plurality of point cloud recognition parameters and a point cloud recognition confidence corresponding to each point cloud recognition parameter are obtained by traversing a plurality of preset point cloud recognition classifiers; Any target point cloud recognition parameter and the point cloud recognition confidence corresponding to the target point cloud recognition parameter are determined as a set of point cloud recognition information, and a plurality of sets of point cloud recognition information are obtained by traversing each set of point cloud recognition parameter.

6. The method of claim 1-5, wherein, The output result of the decision maker is fused for labeling, including: An output result of the decision maker is obtained, and the output result includes a target object three-dimensional recognition frame and target object attribute information; The target object three-dimensional recognition frame is mapped to obtain a target object two-dimensional recognition frame of the target object on a two-dimensional image to be labeled; The target object is fused and labeled based on the target object two-dimensional recognition frame and the target object attribute information.

7. A fusion labeling apparatus characterized by comprising: The device includes: An information acquisition module is configured to acquire an initial three-dimensional point cloud and an initial two-dimensional image; A data synchronization module is configured to synchronize data of the initial three-dimensional point cloud and the initial two-dimensional image to obtain a three-dimensional point cloud to be labeled and a two-dimensional image to be labeled; A target determination module is configured to determine a three-dimensional target frame based on the three-dimensional point cloud to be labeled, and map the three-dimensional target frame to obtain a pseudo three-dimensional target frame in the two-dimensional image to be labeled; A weak classifier module is configured to perform first classification recognition on a plurality of target point clouds in the three-dimensional target frame based on different point cloud classes to obtain point cloud recognition information, and perform second classification recognition on a plurality of target images in the pseudo three-dimensional target frame based on different image classes to obtain image recognition information; A fusion labeling module is configured to input the point cloud recognition information and the image recognition information into a decision maker for integrated learning to obtain corresponding recognition results and a three-dimensional recognition frame, so that the output result of the decision maker is fused for labeling, wherein the image recognition information includes an image recognition parameter and a corresponding image recognition confidence, and the point cloud recognition information includes a point cloud recognition parameter and a corresponding point cloud recognition confidence.

8. The fusion labeling apparatus according to claim 7, wherein The weak classifier module includes: An image recognition classifier is configured to obtain an image recognition parameter of a region of interest image and an image recognition confidence of the region of interest image based on the input region of interest image, wherein an image region in the pseudo three-dimensional target frame is determined as the region of interest image; A point cloud recognition classifier is configured to obtain a point cloud recognition parameter of a point cloud of interest and a point cloud recognition confidence of the point cloud of interest based on the input point cloud of interest, wherein a point cloud in the three-dimensional target frame is determined as the point cloud of interest.

9. An electronic device, comprising: The electronic device includes: One or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the fusion labeling method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, A computer program product, comprising a computer readable medium having stored thereon a computer program, the computer program comprising instructions which, when executed on at least one processor of a computer, cause the computer to carry out the method according to any one of claims 1 to 6. A computer program product, comprising a computer readable medium having stored thereon a computer program, the computer program comprising instructions which, when executed on at least one processor of a computer,

Images