Data annotation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data annotation method and device, equipment and a storage medium. The method comprises the steps of extracting feature points in a to-be-processed image based on a feature detection algorithm, then obtaining three-dimensional point cloud data corresponding to the feature points based on sensor parameters corresponding to the to-be-processed image, and then obtaining a label of a to-be-recognized object in the to-be-processed image based on the three-dimensional point cloud data. According to the technical scheme provided by the embodiment of the invention, the problem of relatively poor precision of data labeling of the long-tail data in related technologies is solved, the problem of relatively poor labeling precision caused by insufficient sample size of the long-tail data is effectively avoided, and the accuracy and reliability of data labeling are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data labeling method, device, equipment and storage medium. Background Art

[0002] With the rapid development of big data and artificial intelligence technology, data annotation has become a key link in promoting the performance improvement of machine learning models. For example, in the field of image processing, through data annotation, the position and size of the objects to be identified in the image are marked, which can facilitate subsequent modeling, further image recognition and other purposes.

[0003] In related technologies, data annotation is usually achieved based on a large number of existing annotated samples combined with machine learning algorithms. However, this method has the problem of insufficient sample size and poor annotation accuracy when processing long-tail data with low frequency but high importance. Summary of the invention

[0004] The embodiments of the present disclosure provide a data labeling method, apparatus, device and storage medium to solve the problem of poor accuracy in data labeling of long-tail data in the related art.

[0005] In a first aspect, an embodiment of the present disclosure provides a data labeling method, the data labeling method comprising:

[0006] Based on the feature detection algorithm, feature points are extracted from the image to be processed, and the feature points are used to represent image points containing features of the object to be identified in the image to be processed;

[0007] Based on the sensor parameters corresponding to the image to be processed, three-dimensional point cloud data corresponding to the feature points is obtained;

[0008] Based on the three-dimensional point cloud data, the annotation of the object to be identified in the image to be processed is obtained, and the annotation includes a bounding box of the object to be identified.

[0009] Optionally, after extracting feature points in the image to be processed based on a feature detection algorithm, the method further includes: generating descriptors corresponding to the feature points, where the descriptors are used to represent identification information corresponding to the feature points.

[0010] Optionally, the sensor parameters include external parameters and internal parameters; based on the sensor parameters corresponding to the image to be processed, three-dimensional point cloud data corresponding to the feature points are obtained, including: based on the similarity of the descriptors of the feature points in different images to be processed, determining the matching point pairs between different images to be processed, the matching point pairs are used to represent two feature points corresponding to the same three-dimensional space point in two different images to be processed; based on the camera motion model and the coordinates of the matching point pairs, determining the posture of the sensor corresponding to the image to be processed, the posture including the rotation angle and translation position of the sensor between different images to be processed; based on the posture of the sensor, determining the external parameters of the sensor; based on the internal parameters, external parameters of the sensor and the coordinates of the matching point pairs, determining the three-dimensional coordinates of the three-dimensional space point corresponding to the matching point pair; based on all the three-dimensional coordinates in the image to be processed, obtaining the three-dimensional point cloud data.

[0011] Optionally, based on the similarity of the descriptors of the feature points in the different images to be processed, matching point pairs between the different images to be processed are determined, including: based on a sliding window algorithm, determining candidate images to be processed for calculating the similarity from all the images to be processed; based on a nearest neighbor search algorithm, determining candidate matching point pairs from the feature points of the candidate images to be processed; based on the similarity of the descriptors of the candidate matching point pairs, determining matching point pairs from the candidate matching points.

[0012] Optionally, based on the three-dimensional point cloud data, an annotation of the object to be identified in the image to be processed is obtained, including: based on a two-dimensional object detection model, determining the two-dimensional boundary of the object to be identified in the image to be processed, the two-dimensional boundary including the two-dimensional boundary box coordinates of the object to be identified; based on the two-dimensional boundary and the three-dimensional point cloud data, determining the three-dimensional boundary corresponding to the object to be identified, the three-dimensional boundary including the three-dimensional boundary box coordinates of the object to be identified; and optimizing the three-dimensional boundary to obtain the annotation of the object to be identified.

[0013] Optionally, based on a two-dimensional object detection model, determining the two-dimensional boundary of the object to be identified in the image to be processed, including: based on the two-dimensional object detection model, determining the original boundary of the object to be identified in the image to be processed, the original boundary including the original two-dimensional boundary box coordinates of the object to be identified; based on an instance segmentation algorithm and the two-dimensional original boundary of the object to be identified, segmenting the two-dimensional original boundary of the object to be identified in the image to be processed to obtain the two-dimensional boundary corresponding to the object to be identified, the two-dimensional boundary including the segmented boundary box coordinates of the object to be identified.

[0014] Optionally, based on the two-dimensional boundary and the three-dimensional point cloud data, the three-dimensional boundary corresponding to the object to be identified is determined, including: determining the instance identification code of each object to be identified; inputting the instance identification code and the two-dimensional boundary of the object to be identified in different images to be processed into a multi-target tracking algorithm to determine the position change information of the object to be identified, the position change information including the boundary box coordinates of the object to be identified at different times; inputting the three-dimensional point cloud data and the position change information corresponding to the object to be identified into a motion structure recovery algorithm to obtain the three-dimensional boundary corresponding to the object to be identified, the three-dimensional boundary including the three-dimensional boundary box coordinates of the object to be identified.

[0015] Optionally, the three-dimensional boundary is optimized to obtain the annotation of the object to be identified, including: based on the geometric features of the two-dimensional bounding box of the object to be identified, the geometric features of the corresponding three-dimensional bounding box are optimized for the first time, and the geometric features include size, direction and center point coordinates; based on the geometric features of the three-dimensional bounding box after the first optimization, the optimized three-dimensional bounding box coordinates of the object to be identified are obtained; based on the posture of the sensor corresponding to the image to be processed and the two-dimensional bounding box coordinates in the object to be identified, the mapped three-dimensional bounding box coordinates corresponding to the object to be identified are determined; based on the vertex coordinates in the mapped three-dimensional bounding box coordinates that are not included in the optimized three-dimensional bounding box coordinates, the coordinates are supplemented to the optimized three-dimensional bounding box coordinates to obtain the supplemented three-dimensional bounding box coordinates; based on the supplemented three-dimensional bounding box coordinates, the two-dimensional projection image of the three-dimensional bounding box of the object to be identified in the corresponding image to be processed is determined; with the goal of minimizing the difference between the vertex coordinates of the two-dimensional projection image and the vertex coordinates of the two-dimensional bounding box, the optimized three-dimensional bounding box is optimized for the second time; and the three-dimensional bounding box coordinates obtained by the second optimization are used as the annotation of the object to be identified.

[0016] Optionally, based on the feature detection algorithm, before extracting feature points in the image to be processed, the method further includes: acquiring image data of the image to be processed, and preprocessing the image data, wherein the preprocessing includes deduplication processing of the image data.

[0017] Optionally, after obtaining the image data of the image to be processed and preprocessing the image data, the method further includes: encrypting the image data and configuring permissions based on permission requirements corresponding to the image to be processed.

[0018] Optionally, after obtaining the annotations of the objects to be identified in the image to be processed based on the three-dimensional point cloud data, the method further includes: verifying the annotations based on at least two set verification algorithms, and determining whether the annotations pass all types of verification.

[0019] In a second aspect, an embodiment of the present disclosure provides a data labeling device, the data labeling device comprising:

[0020] An extraction module, used for extracting feature points in the image to be processed based on a feature detection algorithm, wherein the feature points are used for representing image points containing features of the object to be identified in the image to be processed;

[0021] A determination module, used for obtaining three-dimensional point cloud data corresponding to the feature points based on sensor parameters corresponding to the image to be processed;

[0022] The processing module is used to obtain the annotation of the object to be identified in the image to be processed based on the three-dimensional point cloud data, and the annotation includes the boundary box of the object to be identified.

[0023] In a third aspect, an embodiment of the present disclosure further provides a control device, the control device comprising:

[0024] at least one processor;

[0025] and a memory communicatively coupled to the at least one processor;

[0026] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the control device to perform the data labeling method as described in the first aspect of the present disclosure.

[0027] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the data labeling method as described in the first aspect of the present disclosure.

[0028] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the data labeling method as in the first aspect of the present disclosure.

[0029] The data annotation method, device, equipment and storage medium provided by the embodiments of the present disclosure extract feature points in the image to be processed based on a feature detection algorithm, then obtain three-dimensional point cloud data corresponding to the feature points based on the sensor parameters corresponding to the image to be processed, and then obtain the annotation of the object to be identified in the image to be processed based on the three-dimensional point cloud data. In this way, the object to be identified in the image to be processed can be automatically identified and annotated, and the identification process is not affected by the number of samples of the object to be identified in the image to be processed, effectively avoiding the problem of poor annotation accuracy caused by insufficient long-tail data samples, and improving the accuracy and reliability of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] Figure 1An application scenario diagram of the data annotation method provided in the embodiment of the present disclosure;

[0032] Figure 2 A flowchart of a data annotation method provided by an embodiment of the present disclosure;

[0033] Figure 3a A flowchart of a data annotation method provided by yet another embodiment of the present disclosure;

[0034] Figure 3b for Figure 3a A flow chart of a method for determining matching point pairs provided in the illustrated embodiment;

[0035] Figure 3c for Figure 3a A flow chart of a method for determining a two-dimensional boundary provided in the illustrated embodiment;

[0036] Figure 3d for Figure 3a A flow chart of a method for determining a three-dimensional boundary provided in the illustrated embodiment;

[0037] Figure 3e for Figure 3a A flow chart of a method for optimizing a three-dimensional boundary provided in the illustrated embodiment;

[0038] Figure 4 A flowchart of a data annotation method provided by yet another embodiment of the present disclosure;

[0039] Figure 5 A schematic diagram of the structure of a data annotation device provided by another embodiment of the present disclosure;

[0040] Figure 6 A schematic diagram of the structure of a control device provided in one embodiment of the present disclosure.

[0041] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0044] With the rapid development of big data and artificial intelligence technology, data annotation has become a key link in promoting the performance improvement of machine learning models. For example, in the field of image processing, through data annotation, the position and size of the objects to be identified in the image are marked, which can facilitate subsequent modeling, further image recognition and other purposes.

[0045] In related technologies, data labeling is usually achieved based on a large number of existing labeled samples combined with machine learning algorithms. However, the data labeling accuracy of machine learning algorithms is related to the number of samples. The more samples there are, the higher the labeling accuracy. Therefore, this method has the problem of poor labeling accuracy in processing long-tail data with low frequency but high importance. In addition, the scenarios corresponding to long-tail data vary greatly. Even if a large number of samples corresponding to long-tail data are collected, there is still the problem of insufficient labeling accuracy of machine learning algorithms due to the differences in scenarios between different samples.

[0046] In order to solve this problem, the embodiment of the present disclosure provides a data labeling method, which extracts feature points based on image data, then obtains corresponding three-dimensional point cloud data, and obtains the labeling of the object to be identified based on the three-dimensional point cloud data, thereby effectively solving the problem of limited image labeling accuracy based on machine learning algorithms.

[0047] The application scenarios of the embodiments of the present disclosure are explained below:

[0048] Figure 1 This is an application scenario diagram of the data annotation method provided by the embodiment of the present disclosure. Figure 1 As shown, during the data labeling process, the data collected by the sensor 100 will be sent to the image processing server 110, and the image processing server 110 will process and label the data to obtain labeled data.

[0049] It should be noted that Figure 1 In the scenario shown, only one sensor and one image processing server are used for illustration, but the present disclosure is not limited thereto, that is, the number of sensors and image processing servers can be arbitrary.

[0050] The data annotation method provided by the present disclosure is described in detail below through specific embodiments.

[0051] Figure 2 A flowchart of a data annotation method provided by an embodiment of the present disclosure. Figure 2As shown, the data labeling method provided in this embodiment includes the following steps:

[0052] Step S201: extracting feature points in the image to be processed based on a feature detection algorithm.

[0053] The feature points are used to represent image points containing features of the object to be identified in the image to be processed.

[0054] Specifically, the image to be processed is image data that needs to be annotated. The image to be processed comes from the shooting of an image sensor (such as a camera). The image to be processed can be a single frame image in the video data, or multiple scene images of the same scene.

[0055] The image to be processed may be data extracted from a file in a video format or data in a picture format.

[0056] The image to be processed contains objects to be identified, such as human bodies, traffic lights, license plates, etc. These objects to be identified may differ from surrounding objects in terms of color, brightness, etc. Therefore, the image points on such objects can be identified through feature detection algorithms. The identified image points are also the feature points in the image to be processed.

[0057] This type of image recognition algorithm does not require a large number of images to be processed as samples for training, but can directly adopt already trained image recognition algorithms, such as the SIFT algorithm (Scale Invariant Feature Transform), ORB algorithm (Oriented FAST and Rotated BRIEF, an algorithm improved based on the FAST feature extraction algorithm and the BRIEF algorithm), and other existing image feature extraction algorithms.

[0058] In some embodiments, if the type of object to be identified is predetermined, a feature point extraction algorithm optimized for the corresponding type of object identification may also be selected. Such feature point extraction algorithms are existing algorithms in the art and will not be described in detail here.

[0059] Step S202: obtaining three-dimensional point cloud data corresponding to the feature points based on the sensor parameters corresponding to the image to be processed.

[0060] Specifically, by extracting feature points from multiple frames of images to be processed or multiple images to be processed, the posture changes of the sensor between different images to be processed can be further evaluated based on the relationship between the feature points, and then the corresponding parameters of the sensor can be obtained. Then, combined with the relationship between the feature points and based on the multi-view geometry algorithm, three-dimensional point cloud reconstruction based on the feature points is achieved, that is, the three-dimensional point cloud data corresponding to the feature points is obtained.

[0061] The specific 3D point cloud reconstruction algorithm may be a multi-view geometry algorithm (such as an epipolar geometry constraint algorithm based on iterative least squares method), or may be a stereo vision algorithm, a structured light algorithm, a laser scanning algorithm or other 3D point cloud reconstruction algorithms, which will not be described in detail here.

[0062] Step S203: obtaining annotations of the objects to be identified in the image to be processed based on the three-dimensional point cloud data.

[0063] The annotation includes the bounding box of the object to be identified.

[0064] Specifically, the obtained three-dimensional point cloud data is the three-dimensional point cloud corresponding to the object to be identified. Therefore, the bounding box of the object to be identified can be determined based on these three-dimensional point cloud data. The specific method for determining the bounding box is mainly to obtain the two-dimensional bounding box of the object to be identified through the existing two-dimensional object detection algorithm, and then determine the three-dimensional bounding box of the object to be identified based on the correspondence between the three-dimensional point cloud and the two-dimensional feature points.

[0065] The annotations of the objects to be identified obtained through the above steps do not rely on a large number of images to be processed containing the objects to be identified as samples, but only need to perform feature extraction and three-dimensional reconstruction on the images to be identified, which significantly reduces the demand for the number of samples for data annotation, and can effectively ensure the accurate identification and annotation of long-tail data with a small number of samples, thereby improving the accuracy and reliability of data annotation.

[0066] The data annotation method provided by the embodiment of the present disclosure pre-processes the topological path in the topological map to be processed, determines the nodes to be processed in the topological map to be processed, and then determines the potential mapping area of ​​the topological edge corresponding to the nodes to be processed based on the type of the nodes to be processed, and then segment the mapping area based on the optimization algorithm, and finally obtains the processed topological map based on the target mapping area obtained after the segmentation. In this way, the mapping area corresponding to the topological path in the topological map can be automatically determined, and the topological edges adjacent to the nodes can be jointly processed according to the different types of nodes, and the mapping area can be optimized by segmenting the mapping area, so that the mapping speed and reliability of the mapping area can be balanced, and the mapping efficiency and reliability of the topological map can be maximized.

[0067] Figure 3a A flowchart of a data annotation method provided by an embodiment of the present disclosure. Figure 3a As shown, the data labeling method provided in this embodiment includes the following steps:

[0068] Step S301: extracting feature points in the image to be processed based on a feature detection algorithm.

[0069] The feature points are used to represent image points containing features of the object to be identified in the image to be processed.

[0070] Specifically, feature points are usually points of the object to be identified in the image to be processed that have significant features relative to other objects to be identified, such as edge points of the object to be identified, contour points of specific structures in the object to be identified, etc.

[0071] Step S302: Generate a descriptor corresponding to the feature point.

[0072] The descriptor is used to represent the identification information corresponding to the feature point.

[0073] Specifically, for each feature point, a descriptor corresponding to the feature point needs to be calculated. The descriptor may be a numerical vector to describe the features of a local image formed by image points around the feature point.

[0074] The descriptor is robust to changes such as rotation, scaling, and brightness changes of the image, so that even if the image to be processed is processed by resizing, brightness adjustment, etc., its descriptor can remain unchanged, so as to ensure subsequent processing based on the descriptor.

[0075] Step S303: determining matching point pairs between different images to be processed based on the similarities of the descriptors of the feature points in the different images to be processed.

[0076] The matching point pair is used to represent two feature points corresponding to the same three-dimensional space point in two different images to be processed.

[0077] Specifically, based on the numerical vectorized descriptors of different feature points, the similarities of the descriptors in different images to be processed can be calculated, and the feature points corresponding to the two descriptors with the highest similarity can be determined as matching point pairs.

[0078] The specific similarity matching calculation method may adopt a brute force matching algorithm (corresponding to Brute-ForceMatcher in English), a FLANN algorithm, and the like.

[0079] Furthermore, if Figure 3b As shown, it is a flow chart of a method for determining a matching point pair, which includes the following steps:

[0080] Step S3031: Based on the sliding window algorithm, determine candidate images to be processed for calculating similarity from all images to be processed.

[0081] Specifically, in video data or in a sequence of multiple consecutive images, usually only the current frame and the image data of the adjacent frames before and after will have the same object to be identified, therefore, only these frames / frames of images to be processed need to be matched. These images to be processed for matching are candidate images to be processed, and the specific method for determining candidate images to be processed is the sliding window algorithm (the principle is the process in the above analysis), and the width of the sliding window is the number of candidate images to be processed for matching, such as images to be processed corresponding to three, five or seven consecutive frames.

[0082] Through the sliding window algorithm, all candidate images to be processed corresponding to each image to be processed can be determined, and matching point pairs corresponding to feature points in the image to be processed can be determined from the candidate images to be processed.

[0083] Step S3032: Based on a nearest neighbor search algorithm, determine candidate matching point pairs from the feature points of the candidate image to be processed.

[0084] Specifically, after obtaining the candidate image to be processed, the corresponding candidate matching points can be preliminarily selected from the candidate image to be processed based on the feature point descriptors in the image to be processed, and the candidate matching points and the feature points in the image to be processed are used as candidate matching point pairs. The candidate matching point pairs with poor matching degrees are then screened out to obtain the final matching point pairs.

[0085] The method for initially selecting candidate matching point pairs is usually a neighbor search algorithm, such as a K-nearest neighbor algorithm (or nearest neighbor search algorithm), combined with data structures such as KD trees and ball trees to quickly find candidate matching point pairs. These algorithms are commonly used matching algorithms in this field and will not be described in detail here.

[0086] Step S3033: Determine a matching point pair from the candidate matching points based on the similarity of the descriptors of the candidate matching point pairs.

[0087] Specifically, after obtaining the candidate matching point pairs, the candidate matching points with the highest similarity to the feature point descriptors in the image to be processed (for the convenience of description, the feature points in the image to be processed are referred to as target feature points below) can be preferentially selected as target matching points.

[0088] In some embodiments, the target matching point may be screened by setting a distance threshold. If the distance between the target matching point and the target feature point is greater than the distance threshold, the target matching point is filtered out and the target matching point is re-determined among the candidate matching points.

[0089] In some embodiments, the candidate matching points can also be filtered, such as screening based on the above-mentioned distance threshold, or calculating the ratio of the closest distance between the target feature point and the candidate matching point and the second closest distance between the target feature point and the candidate matching point, and determining the candidate matching point whose closest distance ratio is greater than the set ratio threshold as the target matching point.

[0090] In some embodiments, outliers in the candidate matching points may be excluded by a random sampling consistency algorithm, or the candidate matching points may be filtered based on whether the candidate matching point pairs satisfy a set geometric relationship (such as collinearity).

[0091] In some embodiments, the process of determining candidate matching point pairs may also be optimized based on a graph optimization algorithm (such as a g2o algorithm or a Levenberg-Marquardt algorithm).

[0092] In some embodiments, if the application scenario of the data annotation method is to process a real-time acquired video stream, the similarity between the overall image data corresponding to the latest acquired video frame and the existing image frames can also be calculated, and a set number of specific image frames with the highest similarity can be selected to extract matching point pairs corresponding to the latest acquired video frames without considering all historical frames, thereby reducing resource consumption in the process of determining matching point pairs and improving the efficiency of determining matching point pairs.

[0093] Through the above method, the accuracy and reliability of the determined matching point pairs can be effectively guaranteed.

[0094] Step S304: determining the posture of the sensor corresponding to the image to be processed based on the camera motion model and the coordinates of the matching point pairs.

[0095] The posture includes the rotation angle and translation position of the sensor between different images to be processed.

[0096] Specifically, the sensor corresponding to the image to be processed is actually a camera of various models. Therefore, the calculation from feature points to three-dimensional point cloud can be realized based on the camera motion model.

[0097] Sensor parameters include external parameters and internal parameters. Internal parameters are inherent configurations of the sensor and can be directly obtained from the sensor. External parameters include information such as the position and posture of the sensor during the process of shooting the image to be processed. Therefore, based on the image point coordinate relationship of the corresponding matching point pairs in different frames of image data, combined with the camera motion model, the rotation angle change and translation position change of the camera between different frames can be calculated. Calculating the camera posture based on the relationship between image points and the camera motion model is an existing technology in the field of image acquisition and processing, which will not be explained here.

[0098] In specific calculations, abnormal values ​​in the calculated posture can be filtered out through methods such as iterative least squares method and random sampling consistency algorithm to improve the accuracy of the posture calculation results, thereby ensuring the accuracy and reliability of subsequent calculation results.

[0099] Step S305: determining external parameters of the sensor based on the posture of the sensor.

[0100] Specifically, based on the sensor postures of different frames, the external parameters of the sensor, that is, the camera extrinsic parameters, can be further calculated. Calculating extrinsic parameters based on camera postures is common knowledge in the art and will not be described in detail here.

[0101] Step S306 , determining the three-dimensional coordinates of the three-dimensional space point corresponding to the matching point pair based on the internal parameters, external parameters of the sensor and the coordinates of the matching point pair.

[0102] Specifically, by combining the internal and external parameters of the camera (i.e., sensor) and the triangulation algorithm, the three-dimensional coordinates corresponding to the two-dimensional coordinates of each matching point pair can be calculated. At this time, the two image points in this matching point pair are the projections of the three-dimensional space points corresponding to this three-dimensional coordinate in different images to be processed.

[0103] Step S307, obtaining three-dimensional point cloud data based on all three-dimensional coordinates in the image to be processed.

[0104] Specifically, by calculating the three-dimensional coordinates corresponding to all matching point pairs respectively, it is possible to obtain a three-dimensional point cloud (ie, three-dimensional point cloud data) in the scene corresponding to the image to be processed.

[0105] These 3D point clouds mark the important features (of the objects to be identified) in the scene. Therefore, the objects to be identified can be labeled based on these 3D point clouds.

[0106] In some embodiments, after obtaining the three-dimensional point cloud data, the three-dimensional point cloud data obtained by the camera external parameters can be optimized based on nonlinear optimization algorithms such as bundle adjustment to improve the accuracy of the annotation of the object to be identified obtained by subsequent calculations.

[0107] Step S308: determining the two-dimensional boundary of the object to be identified in the image to be processed based on the two-dimensional object detection model.

[0108] The two-dimensional boundary includes the two-dimensional boundary box coordinates of the object to be identified.

[0109] Specifically, after obtaining the 3D point cloud data, the annotation of the object to be identified can be further obtained based on the 3D point cloud data. However, this process first determines the 2D boundary of the object to be identified, and then further determines the 3D boundary of the object to be identified based on the 2D boundary.

[0110] In addition to the two-dimensional bounding box coordinates of the object to be identified, the two-dimensional boundary may also include the center point coordinates of the object to be identified.

[0111] Furthermore, if Figure 3c As shown, it is a flow chart of a method for determining a two-dimensional boundary, which includes the following steps:

[0112] Step S3081: Based on the two-dimensional object detection model, determine the original boundary of the object to be identified in the image to be processed.

[0113] The original boundary includes the original two-dimensional bounding box coordinates of the object to be identified.

[0114] Specifically, a pre-trained two-dimensional object detection model (such as the Faster R-CNN model or the YOLO model) can identify and locate the category of the object to be identified (such as vehicles, cargo boxes, and pedestrians) and the original boundary from the image to be processed.

[0115] The original boundary obtained at this time is usually the bounding box corresponding to the object to be identified, which is marked with a rectangle, quadrilateral or simple geometric figure (that is, the original two-dimensional bounding box coordinates). This original boundary is relatively rough and needs to be further optimized on this basis.

[0116] Step S3082: Based on the instance segmentation algorithm and the two-dimensional original boundary of the object to be identified, segment the two-dimensional original boundary of the object to be identified in the processed image to obtain the two-dimensional boundary corresponding to the object to be identified.

[0117] The two-dimensional boundary includes the segmented boundary box coordinates of the object to be identified.

[0118] Specifically, based on the original boundary, the bounding box of the object to be identified can be further segmented by an instance segmentation algorithm (such as the Mask R-CNN algorithm) to make it closer to the contour of the object to be identified. Depending on the instance segmentation algorithm selected, the segmentation accuracy can reach the pixel level, thereby maximizing the accuracy of the obtained two-dimensional boundary, so as to ensure the accuracy of the three-dimensional boundary of the object to be identified.

[0119] While obtaining the two-dimensional boundaries of the objects to be identified through the instance segmentation algorithm, an instance identification code is also assigned to each object to be identified, so as to track the changes of each object to be identified in different images to be processed, and then generate the corresponding three-dimensional boundaries.

[0120] Step S309: determining the three-dimensional boundary corresponding to the object to be identified based on the two-dimensional boundary and the three-dimensional point cloud data.

[0121] The three-dimensional boundary includes the three-dimensional boundary box coordinates of the object to be identified.

[0122] Specifically, by combining the coordinates in the two-dimensional boundary and the three-dimensional point cloud data corresponding to each image point, the three-dimensional boundary corresponding to the object to be identified can be further obtained.

[0123] Furthermore, if Figure 3d As shown, it is a flow chart of a method for determining a three-dimensional boundary, which includes the following steps:

[0124] Step S3091: Determine the instance identification code of each object to be identified.

[0125] Specifically, for each object to be identified, its corresponding two-dimensional boundary needs to be tracked between different images to be processed based on its instance identification code, so as to generate its corresponding three-dimensional boundary based on the two-dimensional boundaries in different images to be processed.

[0126] Step S3092: input the instance identification codes and two-dimensional boundaries of the objects to be identified in different images to be processed into a multi-target tracking algorithm to determine the position change information of the objects to be identified.

[0127] The position change information includes the bounding box coordinates of the object to be identified at different times.

[0128] Specifically, through a multi-target tracking algorithm (such as the deepSORT algorithm or the MOTR algorithm), it is possible to track the position changes of the same object in different frames and output the position change information corresponding to each object to be identified, that is, the corresponding two-dimensional boundaries of each object to be identified in different images to be processed.

[0129] Step S3093: input the three-dimensional point cloud data and position change information corresponding to the object to be identified into a motion structure recovery algorithm to obtain the three-dimensional boundary corresponding to the object to be identified.

[0130] The three-dimensional boundary includes the three-dimensional boundary box coordinates of the object to be identified.

[0131] Specifically, based on the motion structure recovery algorithm (the algorithm corresponds to Structure from Motion, also known as SfM algorithm in English), the three-dimensional boundary of the object to be identified can be obtained through the three-dimensional point cloud data corresponding to the image to be processed involving position change information and position transformation information.

[0132] To ensure the accuracy of the obtained three-dimensional boundary, the SfM algorithm can be optimized based on the geometric constraint relationship and the machine learning algorithm.

[0133] Step S310, optimizing the three-dimensional boundary to obtain the label of the object to be identified.

[0134] Specifically, since the obtained three-dimensional boundary is generated based on the two-dimensional boundary, the three-dimensional boundary needs to be optimized to ensure the accuracy of the obtained annotation of the object to be identified.

[0135] Furthermore, if Figure 3e As shown, it is a flow chart of a method for optimizing a three-dimensional boundary, which includes the following steps:

[0136] Step S3101: Based on the geometric features of the two-dimensional bounding box of the object to be identified, the geometric features of the corresponding three-dimensional bounding box are optimized for the first time.

[0137] The geometric features include size, direction and center point coordinates.

[0138] Specifically, by projecting the three-dimensional boundary onto the two-dimensional boundary corresponding to the image to be recognized and comparing the projection with the two-dimensional boundary, the bounding box size of the three-dimensional boundary can be optimized based on the projection consistency principle, that is, the first optimization.

[0139] Step S3102: based on the geometric features of the three-dimensional bounding box after the first optimization, obtain the optimized three-dimensional bounding box coordinates of the object to be identified.

[0140] Specifically, based on the geometric features of the optimized projection (ie, adjusting the coordinates in the projection to the coordinates corresponding to the two-dimensional boundary) and the projection relationship, the bounding box coordinates of the optimized three-dimensional boundary can be obtained.

[0141] Step S3103: Determine the mapped three-dimensional bounding box coordinates corresponding to the object to be identified based on the posture of the sensor corresponding to the image to be processed and the two-dimensional bounding box coordinates of the object to be identified.

[0142] Specifically, in addition to corrections based on two-dimensional boundaries, it is also necessary to consider two-dimensional boundary vertices that may be contained in three-dimensional boundaries. For example, the two-dimensional boundary of the same object to be identified in some images to be processed contains a vertex, while in other images to be processed, the vertex is not included (this situation usually occurs when the object to be identified is an object that may change, such as a human body). In this case, the three-dimensional boundary obtained based on the previous steps may not contain the vertex, but the vertex still needs to be added to the three-dimensional boundary box coordinates.

[0143] In this case, the mapped three-dimensional bounding box coordinates corresponding to the two-dimensional bounding box coordinates can be determined based on the posture of the sensor corresponding to the image to be processed and all the two-dimensional bounding box coordinates corresponding to the object to be identified.

[0144] Step S3104: based on the vertex coordinates in the mapped three-dimensional bounding box coordinates that are not included in the optimized three-dimensional bounding box coordinates, the coordinates are supplemented into the optimized three-dimensional bounding box coordinates to obtain supplemented three-dimensional bounding box coordinates.

[0145] Specifically, the mapped 3D bounding box coordinates are compared with the bounding box coordinates in the determined 3D boundary, and the mapped 3D bounding box coordinates not included in the bounding box coordinates are added to the 3D boundary to obtain the added 3D bounding box coordinates. In this way, the comprehensiveness and accuracy of the obtained 3D boundary can be guaranteed as much as possible, and at the same time, the 3D model of the object to be identified can be conveniently established based on the obtained annotations.

[0146] Step S3105: Based on the supplemented three-dimensional bounding box coordinates, determine a two-dimensional projection image of the three-dimensional bounding box of the object to be identified in the corresponding image to be processed.

[0147] Specifically, in order to ensure the accuracy of the supplemented three-dimensional bounding box coordinates, it is necessary to repeat the process from step S3101 to step S3102 for the supplemented three-dimensional bounding box coordinates, that is, project them into the image to be processed (that is, obtain a two-dimensional projection image), and compare them with the two-dimensional boundaries in the image to be processed.

[0148] Step S3106 , with the goal of minimizing the difference between the vertex coordinates of the two-dimensional projection image and the vertex coordinates of the two-dimensional bounding box, perform a second optimization on the optimized three-dimensional bounding box.

[0149] Specifically, by comparison and with the goal of minimizing the difference between the two, the bounding box coordinates of the three-dimensional boundary are optimized again (i.e., the second optimization) to minimize the projection error, thereby ensuring that the final three-dimensional boundary can closely correspond to the object to be identified in the image to be identified.

[0150] Step S3107: Use the three-dimensional bounding box coordinates obtained by the second optimization as the label of the object to be identified.

[0151] Specifically, the bounding box coordinates of the three-dimensional boundary obtained through the above steps can be used as the annotation of the object to be identified, and the accuracy and reliability of the annotation can be effectively guaranteed.

[0152] The data annotation method provided by the embodiment of the present disclosure generates corresponding descriptors by extracting feature points in the image to be processed, and determines matching point pairs in the image to be processed based on the descriptors, and then establishes a three-dimensional point cloud corresponding to the image to be processed based on the matching point pairs and camera extrinsics, and then combines the two-dimensional boundary and the three-dimensional point cloud of the object to be identified in the image to be processed to obtain the three-dimensional boundary in the image to be processed, and obtains the annotation of the object to be identified in the image to be processed through optimization. In this way, the accuracy and reliability of the annotation of the object to be identified can be guaranteed by repeatedly mining and analyzing the image features in the image to be processed without the need for a large number of sample inputs and learning, thereby effectively solving the problem of poor annotation accuracy caused by insufficient samples faced by long-tail data.

[0153] Figure 4 A flowchart of a data annotation method provided by an embodiment of the present disclosure. Figure 4 As shown, the data labeling method provided in this embodiment includes the following steps:

[0154] Step S401, obtaining image data of an image to be processed, and preprocessing the image data.

[0155] The preprocessing includes removing duplicates from the image data.

[0156] Specifically, in the actual process of data annotation, it is first necessary to obtain the image data of the image to be processed. Since the image data comes from different sources and has different data formats, it is necessary to receive this data through a unified interface and perform reading, parsing, data cleaning and integration (i.e. preprocessing).

[0157] To receive data, you can use the adapter pattern or middleware technology to achieve the interface connection between the data source and the system. For the API interface, you can use the HTTP protocol to obtain data.

[0158] Data preprocessing includes cleaning the received raw data of the image to be processed during the data access process, removing duplicate, noisy and erroneous data, and merging and integrating the raw data from multiple data sources to form a unified data format and unit, thereby obtaining the image data of the image to be processed for subsequent processing, thereby improving the accuracy and reliability of the data.

[0159] Step S402: encrypting and configuring permissions for the image data based on the permission requirements corresponding to the image to be processed.

[0160] Specifically, for images to be processed that contain sensitive data (such as image data involving personal privacy, such as data captured by bank surveillance), such data will contain corresponding processing authority requirements when received. At this time, encryption algorithms (such as AES, RSA and other encryption algorithms) are required to encrypt and protect the image data during transmission, processing and storage of the image to be processed, and set up appropriate authority control mechanisms to ensure that only authorized users or systems can access and process these data, thereby ensuring the security of data transmission, processing and storage, and preventing data leakage and illegal access. And through the authority control mechanism, the complexity of data access and management is simplified, and maintenance costs are reduced.

[0161] In some embodiments, since the data labeling method requires a large amount of computation, a suitable transmission protocol (such as TCP / IP, HTTP / 2) and transmission method (such as streaming transmission, block transmission) are selected according to the data size and transmission frequency, and distributed caching technology (such as Redis, Memcached) is used to cache data to improve data access and processing speed.

[0162] Step S403: extracting feature points in the image to be processed based on a feature detection algorithm.

[0163] The feature points are used to represent image points containing features of the object to be identified in the image to be processed.

[0164] Step S404: obtaining three-dimensional point cloud data corresponding to the feature points based on the sensor parameters corresponding to the image to be processed.

[0165] Step S405: obtaining annotations of the objects to be identified in the image to be processed based on the three-dimensional point cloud data.

[0166] The annotation includes the bounding box of the object to be identified.

[0167] Specifically, steps S403 to S405 are the same as the corresponding steps in the aforementioned embodiment and are not described again here.

[0168] Step S406: verify the annotation based on at least two set verification algorithms, and determine whether the annotation passes all types of verification.

[0169] Specifically, after obtaining the data annotation, the annotation and the image to be processed can be verified for data integrity, data accuracy, and data consistency to ensure the reliability of the processing process.

[0170] Data integrity verification can be performed through algorithms such as hash verification and record number verification. Data accuracy verification can be performed through algorithms such as business rule verification and cross-verification. Consistency verification can be performed through duplicate data checking and data format verification.

[0171] On the basis of the above, statistical methods (such as Z-score, IQR, etc.) can also be used to detect abnormal values ​​in the labeled data, and when an abnormality or error is detected, an alarm mechanism is triggered to notify relevant personnel to handle it, thereby maximizing the accuracy and reliability of the obtained data annotation.

[0172] The data labeling method provided by the embodiment of the present disclosure pre-processes the received data, then labels it, and verifies the labeling result. Thus, the obtained labeled data can guarantee the accuracy and reliability to the maximum extent, and is not dependent on the number of samples, effectively ensuring the practicality of the data labeling method.

[0173] Figure 5 This is a schematic diagram of the structure of a data annotation device provided by an embodiment of the present disclosure. Figure 5 As shown, the data labeling device 500 includes: an extraction module 510, a determination module 520 and a processing module 530. Among them:

[0174] An extraction module 510 is used to extract feature points in the image to be processed based on a feature detection algorithm, where the feature points are used to represent image points containing features of the object to be identified in the image to be processed;

[0175] A determination module 520, configured to obtain three-dimensional point cloud data corresponding to the feature points based on sensor parameters corresponding to the image to be processed;

[0176] The processing module 530 is used to obtain the annotation of the object to be identified in the image to be processed based on the three-dimensional point cloud data, and the annotation includes the bounding box of the object to be identified.

[0177] Optionally, the extraction module 510 is further used to generate descriptors corresponding to the feature points after extracting the feature points in the image to be processed based on a feature detection algorithm, where the descriptors are used to represent identification information corresponding to the feature points.

[0178] Optionally, the determination module 520 is specifically used for: if the sensor parameters include external parameters and internal parameters; based on the similarity of the descriptors of the feature points in different images to be processed, determining the matching point pairs between different images to be processed, the matching point pairs are used to represent two feature points corresponding to the same three-dimensional space point in two different images to be processed; based on the camera motion model and the coordinates of the matching point pairs, determining the posture of the sensor corresponding to the image to be processed, the posture including the rotation angle and translation position of the sensor between different images to be processed; based on the posture of the sensor, determining the external parameters of the sensor; based on the internal parameters, external parameters of the sensor and the coordinates of the matching point pairs, determining the three-dimensional coordinates of the three-dimensional space point corresponding to the matching point pair; based on all the three-dimensional coordinates in the image to be processed, obtaining three-dimensional point cloud data.

[0179] Optionally, the determination module 520 is specifically used to determine, based on a sliding window algorithm, candidate images to be processed for calculating similarity from all images to be processed; based on a nearest neighbor search algorithm, determine candidate matching point pairs from feature points of the candidate images to be processed; and based on the similarity of the descriptors of the candidate matching point pairs, determine matching point pairs from the candidate matching points.

[0180] Optionally, the processing module 530 is specifically used to determine the two-dimensional boundary of the object to be identified in the image to be processed based on the two-dimensional object detection model, the two-dimensional boundary including the two-dimensional boundary box coordinates of the object to be identified; determine the three-dimensional boundary corresponding to the object to be identified based on the two-dimensional boundary and the three-dimensional point cloud data, the three-dimensional boundary including the three-dimensional boundary box coordinates of the object to be identified; and optimize the three-dimensional boundary to obtain the annotation of the object to be identified.

[0181] Optionally, the processing module 530 is specifically used to determine the original boundaries of the object to be identified in the image to be processed based on the two-dimensional object detection model, where the original boundaries include the original two-dimensional boundary box coordinates of the object to be identified; based on the instance segmentation algorithm and the two-dimensional original boundaries of the object to be identified, segment the two-dimensional original boundaries of the object to be identified in the image to be processed to obtain the two-dimensional boundaries corresponding to the object to be identified, where the two-dimensional boundaries include the segmented boundary box coordinates of the object to be identified.

[0182] Optionally, the processing module 530 is specifically used to determine the instance identification code of each object to be identified; input the instance identification code and two-dimensional boundaries of the objects to be identified in different images to be processed into the multi-target tracking algorithm to determine the position change information of the objects to be identified, and the position change information includes the bounding box coordinates of the objects to be identified at different times; input the three-dimensional point cloud data and position change information corresponding to the objects to be identified into the motion structure recovery algorithm to obtain the three-dimensional boundaries corresponding to the objects to be identified, and the three-dimensional boundaries include the three-dimensional bounding box coordinates of the objects to be identified.

[0183] Optionally, the processing module 530 is specifically used to perform a first optimization on the geometric features of the corresponding three-dimensional bounding box based on the geometric features of the two-dimensional bounding box of the object to be identified, where the geometric features include size, direction and center point coordinates; obtain the optimized three-dimensional bounding box coordinates of the object to be identified based on the geometric features of the three-dimensional bounding box after the first optimization; determine the mapped three-dimensional bounding box coordinates corresponding to the object to be identified based on the posture of the sensor corresponding to the image to be processed and the two-dimensional bounding box coordinates in the object to be identified; supplement the vertex coordinates in the mapped three-dimensional bounding box coordinates that are not included in the optimized three-dimensional bounding box coordinates to obtain the supplemented three-dimensional bounding box coordinates; determine the two-dimensional projection image of the three-dimensional bounding box of the object to be identified in the corresponding image to be processed based on the supplemented three-dimensional bounding box coordinates; perform a second optimization on the optimized three-dimensional bounding box with the goal of minimizing the difference between the vertex coordinates of the two-dimensional projection image and the vertex coordinates of the two-dimensional bounding box; and use the three-dimensional bounding box coordinates obtained from the second optimization as the label of the object to be identified.

[0184] Optionally, the extraction module 510 is further used to obtain image data of the image to be processed and preprocess the image data before extracting feature points in the image to be processed based on the feature detection algorithm, and the preprocessing includes deduplication processing of the image data.

[0185] Optionally, the extraction module 510 is further used to obtain image data of the image to be processed, and after preprocessing the image data, encrypt the image data and configure permissions based on permission requirements corresponding to the image to be processed.

[0186] Optionally, the processing module 530 is further used to, after obtaining the annotations of the objects to be identified in the image to be processed based on the three-dimensional point cloud data, verify the annotations based on at least two set verification algorithms, and determine whether the annotations pass all types of verification.

[0187] In this embodiment, the data labeling device solves the problem of poor accuracy in data labeling of long-tail data in related technologies through the combination of various modules, effectively avoids the problem of poor labeling accuracy caused by insufficient long-tail data samples, and improves the accuracy and reliability of data labeling.

[0188] Figure 6 A schematic diagram of the structure of a control device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the control device 600 includes: a memory 610 and a processor 620 .

[0189] The memory 610 stores a computer program that can be executed by at least one processor 620. The computer program is executed by at least one processor 620 to enable the control device to implement the material removal method provided in any of the above embodiments or the data labeling method provided in any of the above embodiments.

[0190] The memory 610 and the processor 620 may be connected via a bus 630 .

[0191] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the method embodiments, which will not be repeated here.

[0192] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the method embodiments, which will not be repeated here.

[0193] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement a material removal method provided in any of the above method embodiments or a data labeling method provided in any of the above embodiments.

[0194] Among them, the computer readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0195] An embodiment of the present disclosure provides a computer program product, which includes computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the material removal method in the above method embodiment or the data labeling method provided in any of the above embodiments.

[0196] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0197] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0198] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data labeling method, characterized in that: The method comprises: Extracting feature points in the image to be processed based on a feature detection algorithm, wherein the feature points are used to represent image points containing features of the object to be identified in the image to be processed; Based on the sensor parameters corresponding to the image to be processed, obtaining three-dimensional point cloud data corresponding to the feature points; Based on the three-dimensional point cloud data, a label of the object to be identified in the image to be processed is obtained, and the label includes a bounding box of the object to be identified.

2. The method according to claim 1, characterized in that After extracting feature points from the image to be processed based on the feature detection algorithm, the method further includes: A descriptor corresponding to the feature point is generated, where the descriptor is used to represent identification information corresponding to the feature point.

3. The method according to claim 2, characterized in that The sensor parameters include external parameters and internal parameters; Based on the sensor parameters corresponding to the image to be processed, obtaining three-dimensional point cloud data corresponding to the feature points includes: Determine matching point pairs between the different images to be processed based on similarities of the descriptors of the feature points in the different images to be processed, wherein the matching point pairs are used to represent two feature points corresponding to the same three-dimensional space point in the two different images to be processed; Determine the posture of the sensor corresponding to the image to be processed based on the camera motion model and the coordinates of the matching point pair, wherein the posture includes a rotation angle and a translation position of the sensor between different images to be processed; Determining external parameters of the sensor based on the posture of the sensor; Determine the three-dimensional coordinates of the three-dimensional space point corresponding to the matching point pair based on the internal parameters and external parameters of the sensor and the coordinates of the matching point pair; The three-dimensional point cloud data is obtained based on all three-dimensional coordinates in the image to be processed.

4. The method according to claim 3, characterized in that The determining of matching point pairs between different images to be processed based on similarities of descriptors of feature points in the different images to be processed includes: Based on the sliding window algorithm, a candidate image to be processed for calculating similarity is determined from all images to be processed; Determine candidate matching point pairs from the feature points of the candidate image to be processed based on a nearest neighbor search algorithm; Based on the similarity of the descriptors of the candidate matching point pair, the matching point pair is determined from the candidate matching points.

5. The method according to claim 3, characterized in that: The obtaining, based on the three-dimensional point cloud data, a label of the object to be identified in the image to be processed comprises: Determining a two-dimensional boundary of an object to be identified in the image to be processed based on a two-dimensional object detection model, wherein the two-dimensional boundary includes two-dimensional boundary box coordinates of the object to be identified; Based on the two-dimensional boundary and the three-dimensional point cloud data, determining a three-dimensional boundary corresponding to the object to be identified, wherein the three-dimensional boundary includes the three-dimensional boundary box coordinates of the object to be identified; The three-dimensional boundary is optimized to obtain a label of the object to be identified.

6. The method according to claim 5, characterized in that The step of determining the two-dimensional boundary of the object to be identified in the image to be processed based on the two-dimensional object detection model includes: Based on the two-dimensional object detection model, determining the original boundary of the object to be identified in the image to be processed, wherein the original boundary includes the original two-dimensional boundary box coordinates of the object to be identified; Based on the instance segmentation algorithm and the two-dimensional original boundary of the object to be identified, the two-dimensional original boundary of the object to be identified in the image to be processed is segmented to obtain the two-dimensional boundary corresponding to the object to be identified, and the two-dimensional boundary includes the segmented boundary box coordinates of the object to be identified.

7. The method according to claim 6, characterized in that The determining, based on the two-dimensional boundary and the three-dimensional point cloud data, the three-dimensional boundary corresponding to the object to be identified includes: Determine an instance identification code for each object to be identified; Inputting instance identification codes and two-dimensional boundaries of the objects to be identified in different images to be processed into a multi-target tracking algorithm to determine position change information of the objects to be identified, wherein the position change information includes the coordinates of the boundary boxes of the objects to be identified at different times; The three-dimensional point cloud data and position change information corresponding to the object to be identified are input into a motion structure recovery algorithm to obtain a three-dimensional boundary corresponding to the object to be identified, wherein the three-dimensional boundary includes the three-dimensional boundary box coordinates of the object to be identified.

8. A data labeling device, characterized in that: The data labeling device comprises: An extraction module, used for extracting feature points in the image to be processed based on a feature detection algorithm, wherein the feature points are used to represent image points containing features of the object to be identified in the image to be processed; A determination module, configured to obtain three-dimensional point cloud data corresponding to the feature points based on sensor parameters corresponding to the image to be processed; A processing module is used to obtain a label of the object to be identified in the image to be processed based on the three-dimensional point cloud data, wherein the label includes a bounding box of the object to be identified.

9. A control device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the control device to perform the data labeling method as described in any one of claims 1 to 7.

10. A computer readable storage medium or a computer program product, characterized in that: The computer-readable storage medium or program product stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the data labeling method according to any one of claims 1 to 7.

Images