Position indexing method and device
By acquiring image and point cloud data, determining the attribute information and position relationship of the target object, the problem of low accuracy and recall in the prior art is solved, and the training performance of the ranging model is improved.
Patent Information
- Application Number
- CN202310820869.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-05
- Publication Date
- 2025-01-14
AI Technical Summary
The accuracy and recall of the existing position labeling methods are low, which cannot meet the training requirements of the deep learning end-to-end ranging algorithm, especially in the case of complex obstacles and sparse point cloud data.
By obtaining image data and point cloud data, determining the attribute information of the target object, establishing the correspondence between the target object and the image data, and using attribute information and bounding box matching to determine the position information of the target object to be marked, and then accurately performing the image data. Mark.
It improves the accuracy and recall of the labeling results, meets the training requirements of the ranging model, and improves the performance of the ranging model.
Smart Images
Figure CN119314138A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, specifically to the field of intelligent driving, and more particularly to a position indexing method and device. Background Art
[0002] In the field of intelligent driving, end-to-end distance measurement algorithms based on deep learning are often used to predict the position of obstacles around the vehicle relative to the vehicle itself. They have the advantages of high prediction accuracy and wide application scenarios. However, the distance measurement model corresponding to the end-to-end distance measurement algorithm usually needs to be trained using image data labeled with the position of obstacles relative to the vehicle itself, while the accuracy and recall rate of the annotation results of the existing position annotation methods are low, which cannot meet the training requirements of the distance measurement model. Summary of the invention
[0003] In view of this, the object of the present invention is to provide a position labeling method and device to improve the accuracy and recall rate of the labeling results, thereby meeting the training requirements of the ranging model and improving the performance of the trained ranging model.
[0004] In a first aspect, an embodiment of the present invention provides a location indexing method, the method comprising:
[0005] Obtain image data and corresponding point cloud data;
[0006] Determining attribute information of at least one target object according to the point cloud data, wherein the attribute information is used to characterize the position and shape of the target object in three-dimensional space;
[0007] Determine, according to the attribute information, a correspondence between at least one target object to be labeled in the image data and each of the target objects;
[0008] For each of the target objects to be labeled, determining first position information of the target object to be labeled according to attribute information of a target object corresponding to the target object to be labeled;
[0009] The position of each of the target objects to be marked relative to the current object is marked in the image data at least according to the first position information.
[0010] In a second aspect, an embodiment of the present invention provides a position indexing device, the device comprising:
[0011] An acquisition unit, used for acquiring image data and corresponding point cloud data;
[0012] an attribute information determining unit, configured to determine attribute information of at least one target object according to the point cloud data, wherein the attribute information is used to characterize a position and a shape of the target object in a three-dimensional space;
[0013] an associating unit, configured to determine a corresponding relationship between at least one target object to be annotated in the image data and each of the target objects according to the attribute information;
[0014] A position information determining unit, configured to determine, for each of the target objects to be labeled, first position information of the target object to be labeled according to attribute information of a target object corresponding to the target object to be labeled;
[0015] A labeling unit is used to label the position of each target object to be labeled relative to the current object in the image data at least according to the first position information.
[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method as described in any one of the first aspects is implemented.
[0017] In a fourth aspect, an embodiment of the present invention provides an electronic device, the device comprising:
[0018] a memory for storing one or more computer program instructions;
[0019] A processor, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of the first aspects.
[0020] In a fifth aspect, an embodiment of the present invention provides a computer program product, which, when executed on a computer, enables the computer to execute the method as described in any one of the first aspects.
[0021] The position labeling method of the embodiment of the present invention determines the attribute information of at least one target object based on the point cloud data, and determines the correspondence between at least one target object to be labeled and each target object in the image data based on the attribute information, and then determines the first position information of each target object to be labeled based on the attribute information of the target object corresponding to each target object to be labeled, and then labels the position of each target object to be labeled relative to the current object in the image data at least based on the first position information. The attribute information is used to characterize the position and shape of the target object in three-dimensional space. The method can improve the accuracy and recall rate of the labeling results, thereby meeting the training requirements of the ranging model and improving the performance of the trained ranging model. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
[0023] Figure 1is a flow chart of a location indexing method according to an embodiment of the present invention;
[0024] Figure 2 is a flow chart of an object association method according to an embodiment of the present invention;
[0025] Figure 3 is a schematic diagram of a first bounding box in image data according to an embodiment of the present invention;
[0026] Figure 4 is a flowchart of a second bounding box determination method according to an embodiment of the present invention;
[0027] Figure 5 is a schematic diagram of a three-dimensional bounding box in image data according to an embodiment of the present invention;
[0028] Figure 6 is a schematic diagram of a second bounding box in image data according to an embodiment of the present invention;
[0029] Figure 7 is a flow chart of a method for determining target location information according to an embodiment of the present invention;
[0030] Figure 8 A schematic diagram of selecting preset marking points of a vehicle according to an embodiment of the present invention;
[0031] Fig. 9 is a flow chart of a method for determining second location information according to an embodiment of the present invention;
[0032] Fig.10 A flowchart of a method for determining a spatial position of a preset annotation point according to an embodiment of the present invention;
[0033] Fig.11 A schematic diagram of a process for determining target location information according to an embodiment of the present invention;
[0034] Fig.12 is a schematic diagram of a position indexing device according to an embodiment of the present invention;
[0035] Fig.13 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.
[0037] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.
[0038] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".
[0039] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.
[0040] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.
[0041] In the related art, the position marking method usually selects a fixed point in the image for the obstacle to be marked, and then associates the fixed point with the point cloud data of the obstacle to determine the position of the fixed point relative to the own vehicle, and then determines the position of the fixed point relative to the own vehicle as the position of the obstacle relative to the own vehicle, but this method can only mark relatively simple obstacles. If the obstacle is too complex, for example, the obstacle is at an angle to the own vehicle, the obstacle is too close or too far from the own vehicle, the obstacle is truncated, or the obstacle is blocked, there will be a large deviation between the position result marked by the fixed point and the true value. At the same time, when the point cloud data is too sparse, the related technology will also have the situation of misassociation, which will also cause a large deviation between the marked position result and the true value. The above reasons make the accuracy and recall rate of the marking results of the current position marking method low, and cannot meet the training requirements of the ranging model.
[0042] In this regard, the present invention provides a position labeling method and device to improve the accuracy and recall rate of the labeling results, thereby meeting the training requirements of the ranging model and improving the performance of the trained ranging model.
[0043] Figure 1 FIG. 1 is a flow chart of a location indexing method according to an embodiment of the present invention. Figure 1 As shown, the position indexing method may specifically include the following steps:
[0044] It should be understood that the execution subject of the location indexing method can be any general data processing device, including but not limited to a tablet computer or a computer.
[0045] S100: Acquire image data and corresponding point cloud data.
[0046] Specifically, the processing device may acquire image data and point cloud data corresponding to the image data.
[0047] Among them, the image data may include at least one target object to be annotated, that is, the target object to be annotated. It should be understood that in this embodiment, the target object may specifically refer to other vehicles traveling around the own vehicle. In some embodiments, the target object can also be set to other obstacles on the road according to actual needs, such as pedestrians, traffic facilities, trees or buildings. The point cloud data is a data set composed of a large number of spatial points, that is, points in three-dimensional space. The spatial points in the point cloud data can be used to characterize the position of the surface points of related objects in a certain direction around the own vehicle in three-dimensional space.
[0048] It should be understood that the correspondence between the point cloud data and the image data may specifically mean that the acquisition time of the two is the same. Specifically, the processing device will use the point cloud data in the subsequent steps to mark the position of each target object to be marked in the image data relative to the own vehicle, and the position of the target object and the own vehicle usually changes in real time. In order to ensure the accuracy of the marking result, the point cloud data and the image data need to be data collected at the same time.
[0049] Optionally, the image data and point cloud data can be acquired by at least one image acquisition device and at least one radar sensor installed on the vehicle. The image acquisition device can be set at a corresponding position of the vehicle body, such as the front, rear or both sides of the vehicle, and the radar sensor can be set on the roof. Furthermore, the image acquisition device and radar sensor can be used to respectively acquire image data and point cloud data corresponding to the image data during the driving process of the vehicle. Optionally, the image acquisition device can be specifically a camera or a video camera, and the radar sensor can be specifically a lidar sensor.
[0050] Optionally, in order to enable the image acquisition device and the radar sensor to acquire image data and point cloud data having a corresponding relationship, the image acquisition device and the radar sensor may maintain time synchronization in acquisition frequency.
[0051] S200: Determine attribute information of at least one target object according to the point cloud data.
[0052] Specifically, the spatial points in the point cloud data belong to multiple objects, including at least one target object. After acquiring the point cloud data, the processing device can determine the attribute information of each target object based on the point cloud data. The attribute information is used to characterize the position and shape of the target object in the three-dimensional space. It should be understood that the attribute information can also be used to characterize the orientation of the target object in the three-dimensional space, the speed of the target object, and the object category information to which the target object belongs.
[0053] In an optional implementation, the processing device can determine the attribute information of each object based on the point cloud data, and then the processing device can determine the target object among the objects based on the attribute information. It should be understood that each object has a shape corresponding to it, and when determining the target object, the processing device can determine the object whose shape matches the target object as the target object based on the attribute information.
[0054] Optionally, in step S200, the processing device may input the point cloud data into a corresponding algorithm model to determine the attribute information of each target object through the algorithm model. Optionally, the algorithm model may be a model constructed based on a CenterPoint algorithm. The CenterPoint algorithm is a three-dimensional target detection algorithm based on a laser radar point cloud. It should be understood that the algorithm model may also be a model constructed based on other three-dimensional target detection algorithms.
[0055] S300: Determine a correspondence between at least one target object to be labeled in the image data and each of the target objects according to the attribute information.
[0056] Specifically, after determining the attribute information of at least one target object according to the point cloud data, the processing device may determine the corresponding relationship between each of the target objects and each to-be-annotated target object in the image data according to the attribute information.
[0057] Optionally, the processing device can determine a first bounding box of each target object to be labeled in the image data, and project each target object in the three-dimensional space to the plane where the image data is located according to the attribute information to determine a second bounding box of each target object in the image data, and then determine the correspondence between each target object and each target object to be labeled by matching the first bounding box with the second bounding box.
[0058] Figure 2 FIG. 1 is a flow chart of an object association method according to an embodiment of the present invention. Figure 2 As shown, the object association method may specifically include the following steps:
[0059] It should be understood that the object association method can be specifically used to implement the above step S300.
[0060] S310: Determine a first bounding box of each target object to be labeled in the image data.
[0061] Specifically, the processing device may determine a first bounding box of each target object to be labeled in the image data.
[0062] Optionally, the bounding box may be specifically determined by a corresponding bounding box regression model. The bounding box regression model may automatically mark the bounding box of each target object to be marked in the image without user operation through a bounding box regression algorithm, such as the YOLOv5 (You Only Look Once version 5) algorithm, thereby improving the marking efficiency.
[0063] Figure 3 FIG. 1 is a schematic diagram of a first boundary box in image data according to an embodiment of the present invention. Figure 3 As shown, the image data includes target objects 32, 33 and 34 to be labeled. The processing device can determine first bounding boxes 321, 331 and 341 of the target objects 32, 33 and 34 to be labeled, respectively.
[0064] S320: Determine a second bounding box of each of the target objects in the image data according to the attribute information.
[0065] Specifically, the processing device may also project each target object in the three-dimensional space to the plane where the image data is located according to the attribute information to determine a second bounding box of each target object in the image data.
[0066] Optionally, the processing device may first project each target object in the three-dimensional space to the plane where the image data is located according to the attribute information to determine a stereoscopic bounding box of each target object in the image data, and then determine a second bounding box of each target object according to the stereoscopic bounding box.
[0067] Figure 4 FIG. 4 is a flow chart of a second bounding box determination method according to an embodiment of the present invention. Figure 4 As shown, the second bounding box determination method may specifically include the following steps:
[0068] It should be understood that the second bounding box determination method can be specifically used to implement the above step S320.
[0069] S321. Project each of the target objects onto a plane where the image data is located according to the attribute information to determine a three-dimensional bounding box of each of the target objects in the image data.
[0070] Specifically, the processing device may project each of the target objects in the three-dimensional space onto the plane where the image data is located according to the attribute information to determine a three-dimensional bounding box of each of the target objects in the image data.
[0071] Optionally, for each target object, the processing device may project the target object from the three-dimensional space to the plane where the image data is located according to the attribute information and the parameter information of the image acquisition device that acquires the image data. The parameter information may specifically include the intrinsic parameter information and the extrinsic parameter information of the image acquisition device. Specifically, the processing device may select a plurality of related points corresponding to the target object in the three-dimensional space according to the attribute information. After selecting the related points, the processing device may perform an extrinsic parameter matrix transformation on the position of each of the related points in the three-dimensional space according to the extrinsic parameter information to obtain the position of each of the related points in the camera coordinate system. After determining the position of each of the related points in the camera coordinate system, the processing device may perform an intrinsic parameter matrix transformation on the position of the related points in the camera coordinate system according to the intrinsic parameter information to determine the position of each of the related points in the image data. After determining the position of each of the related points in the image data, the processing device may determine the stereoscopic bounding box of the target object in the image data.
[0072] Optionally, for different target objects, the selected related points may be different. Taking a vehicle as an example, the related points may include eight corner points on a stereoscopic detection frame of the vehicle in three-dimensional space. It should be understood that the selection of the related points will be specifically set and adjusted according to actual conditions.
[0073] Figure 5 FIG. 1 is a schematic diagram of a three-dimensional bounding box in image data according to an embodiment of the present invention. Figure 5 As shown, the processing device can project the target objects a, b and c in the three-dimensional space to the plane where the image data is located to determine the three-dimensional bounding boxes 521, 531 and 541 of the target objects a, b and c in the image data respectively. It should be understood that in the current step, the three-dimensional bounding boxes 521, 531 and 541 cannot yet correspond to the target objects to be labeled in the image data. It should be understood that Figure 5 The positions of the eight solid black points (as shown by point 5311) in the image are the positions of the eight corner points of the vehicle determined in the image data.
[0074] S322: For each target object, determine a second bounding box of the target object according to the three-dimensional bounding box of the target object.
[0075] Specifically, after determining the stereoscopic bounding box of each target object in the image data, the processing device may determine the second bounding box of each target object according to the stereoscopic bounding box.
[0076] Optionally, in step S322, each 3D bounding box covers a certain pixel area in the image data, and for each of the 3D bounding boxes, the processing device may determine each pixel point at the boundary of the pixel area covered by the 3D bounding box, and then stitch the pixel points together to determine a second bounding box corresponding to the target object. The second bounding box may completely surround the pixel area covered by the 3D bounding box in the image data.
[0077] Figure 6 FIG. 4 is a schematic diagram of a second boundary box in image data according to an embodiment of the present invention. Figure 6 As shown, in Figure 5 On the basis of the above, the processing device may further determine second bounding boxes 621 , 631 , and 641 in the image data according to the stereo bounding boxes 521 , 531 , and 541 .
[0078] It should be understood that Figure 3 and Figure 6 The shapes of the first boundary box and the second boundary box are shown for illustration only. In actual application, the shapes of the boundary boxes will be set and adjusted according to actual conditions.
[0079] S330: Match each of the first bounding boxes with each of the second bounding boxes to determine a corresponding relationship between each of the target objects and each of the target objects to be labeled.
[0080] Specifically, after determining the first bounding box of each target object to be labeled and the second bounding box of each target object in the image data, the processing device can match each of the first bounding boxes with each of the second bounding boxes to determine the correspondence between each first bounding box and each second bounding box. Then, the processing device can determine the correspondence between each target object and each target object to be labeled based on the correspondence between each first bounding box and each second bounding box.
[0081] like Figure 3 and Figure 6 As shown, the processing device can Figure 3 The first bounding boxes 321, 331 and 341 in Figure 6The second bounding boxes 621, 631, and 641 in the image data are matched to determine the first bounding box 321 corresponding to the second bounding box 621, the first bounding box 331 corresponding to the second bounding box 631, and the first bounding box 341 corresponding to the second bounding box 641. Further, after determining the correspondence between each first bounding box and each second bounding box, the processing device can determine that the target object 32 to be labeled in the image data corresponds to the target object a, the target object 33 to be labeled in the image data corresponds to the target object b, and the target object 34 to be labeled in the image data corresponds to the target object c according to the determined correspondence between the bounding boxes.
[0082] It should be understood that during the box association process, the number of target objects determined by the processing device based on the point cloud data is usually greater than the number of target objects to be labeled in the image data. When determining the correspondence between each first bounding box and each second bounding box, the processing device can only determine the second bounding box corresponding to each first bounding box.
[0083] Optionally, the processing device may determine the second bounding boxes corresponding to the first bounding boxes by using a corresponding matching algorithm, and the matching algorithm may be a Hungarian matching algorithm. The Hungarian matching algorithm may determine the second bounding boxes matching the first bounding boxes from among the plurality of second bounding boxes.
[0084] Optionally, for any first bounding box and second bounding box, whether the first bounding box matches the second bounding box can be determined by the processing device by calculating the intersection-and-union ratio of pixel areas covered by the two. Specifically, when the intersection-and-union ratio of the two is within a corresponding threshold range, the processing device will determine that the two are associated.
[0085] S400: For each target object to be labeled, determine first position information of the target object to be labeled according to attribute information of a target object corresponding to the target object to be labeled.
[0086] Specifically, after determining the correspondence between each target object and each target object to be labeled, for each target object to be labeled, the processing device may determine the first position information of the target object to be labeled according to the attribute information of the target object corresponding to the target object to be labeled.
[0087] It should be understood that the position information in this embodiment, such as the first position information, the second position information, the third position information and the target position information, may refer to the position of the target object to be marked relative to the current object, that is, the vehicle itself, in three-dimensional space.
[0088] S500: Mark the position of each of the target objects to be marked relative to the current object in the image data at least according to the first position information.
[0089] Specifically, after determining the first position information of each target object to be labeled, the processing device may mark the position of each target object to be labeled in the image data at least according to the first position information. It should be understood that in this embodiment, marking the position of each target object to be labeled may specifically refer to marking the position of each target object to be labeled relative to the current object in the image data.
[0090] It should be understood that in step S500, the processing device can directly mark the position of each target object to be marked in the image data according to the first position information; or, in order to avoid a decrease in the recall rate and accuracy of subsequent marking results due to mis-association or missed association problems in the process of determining the first position information, the processing device can further determine multiple position information of the target object relative to the current object by other means, and then jointly determine the target position information of the target object relative to the current object based on each of the position information, and then mark the position of each target object to be marked in the image data according to the target position information.
[0091] Figure 7 FIG. 1 is a flow chart of a method for determining target location information according to an embodiment of the present invention. Figure 7 As shown, the target location information determination method may specifically include the following steps:
[0092] S510: For each of the target objects to be labeled, determine second position information of the target object to be labeled according to the point cloud data and a preset labeling point corresponding to the target object to be labeled.
[0093] Specifically, each target object to be annotated in the image data may also have at least one corresponding preset annotation point. For each target object to be annotated, the processing device may determine the second position information of the target object to be annotated based on the point cloud data and the preset annotation point corresponding to the target object to be annotated. The preset annotation point may be pre-annotated by an operator, or may be annotated by the processing device through a corresponding annotation model. It should be understood that this embodiment can improve the annotation accuracy and annotation flexibility in the face of complex scenes by manually participating in the annotation process of the preset annotation points.
[0094] It should be understood that for different target objects, the selected preset marking points are different. Figure 8 FIG. 1 is a schematic diagram of selecting preset marking points of a vehicle according to an embodiment of the present invention. Figure 8 As shown, taking vehicle 81 as an example, the preset annotation points may include four points 821, 823, 822 and 824 around vehicle 81, four corner points 811, 812, 813 and 814 of vehicle 81, and a point 831 on the front windshield of vehicle 81. It should be understood that Figure 8The preset marking points shown are for illustration only. In actual application, the preset marking points will be determined according to actual needs and applicable target object types.
[0095] Fig. 9 FIG. 4 is a flow chart of a second location information determination method according to an embodiment of the present invention. Fig. 9 As shown, the second location information determination method may specifically include the following steps:
[0096] It should be understood that the second location information determination method can be specifically used to implement the above step S510.
[0097] S511. Determine the spatial position information of each of the preset annotation points according to the point cloud data.
[0098] Specifically, the processing device may determine the spatial position information of each preset annotation point by matching each preset annotation point with each spatial point in the point cloud data, wherein the spatial position information is used to represent the position in the three-dimensional space.
[0099] Fig.10 The present invention is a flowchart of a method for determining a spatial position of a preset annotation point according to an embodiment of the present invention.
[0100] like Fig.10 As shown, the method for determining the spatial position of the preset annotation point may specifically include the following steps:
[0101] It should be understood that the method for determining the spatial position of the preset annotation point can be specifically used to implement the above step S511.
[0102] S5111. Project the point cloud data onto the plane where the image data is located to determine a set of projection points in the image data.
[0103] Specifically, the processing device may project each spatial point in the point cloud data onto the plane where the image data is located to determine a set of projection points in the image data.
[0104] It should be understood that the method of projecting the spatial points to the plane where the image data is located is the same as the method of projecting the relevant points corresponding to the target object to the plane where the image data is located in step S321, and will not be repeated here.
[0105] Optionally, in order to reduce the computational pressure, the spatial points projected in step S5111 may be spatial points belonging to each target object.
[0106] S5112: For each of the preset marking points, determine the spatial position information of the projection point in the projection point set that is closest to the preset marking point as the spatial position information of the preset marking point.
[0107] Specifically, after determining the projection point set in the image data, for each preset annotation point, the processing device may determine the spatial position information of the projection point in the projection point set that is closest to the preset annotation point as the spatial position information of the preset annotation point.
[0108] It should be understood that the projection point closest to the preset annotation point specifically refers to the projection point closest to the preset annotation point on the two-dimensional plane where the image data is located.
[0109] S512: Determine second position information of the target object to be marked according to the spatial position information and relative position information of each of the preset marking points.
[0110] Specifically, after determining the spatial position information of each preset annotation point, the processing device can determine the second position information of the target object to be annotated according to the spatial position information and the relative position information of each preset annotation point, wherein the relative position information is used to characterize the position of the preset annotation point on the corresponding target object to be annotated.
[0111] Optionally, in order to determine more accurate second position information, the processing device may further correct the determined second position information in combination with the shape of the target object to be marked, ie, the length, width, height, and heading angle, during the process of determining the second position information.
[0112] S520: For each target object to be labeled, determine third position information of the target object to be labeled according to spatial position information of a target spatial point of the target object to be labeled.
[0113] Specifically, for each of the target objects to be labeled, the processing device may also first determine a target space point of each of the target objects to be labeled. The target space point is a space point in the point cloud data corresponding to the target object to be labeled that is closest to the current object. After determining the target space point of each of the target objects to be labeled, the processing device may determine the third position information of each of the target objects to be labeled based on the spatial position information of each of the target space points.
[0114] It should be understood that the spatial point closest to the current object specifically refers to the spatial point closest to the current object in the three-dimensional space.
[0115] Optionally, when determining the target space point of each target object to be labeled, for each target object to be labeled, the processing device can determine the point cloud data corresponding to the target object to be labeled based on the attribute information of the target object corresponding to the target object to be labeled, and then determine the target space point of the target object to be labeled in the point cloud data corresponding to the target object to be labeled.
[0116] Optionally, in order to determine more accurate third position information, the processing device may also correct the determined third position information in combination with the shape of the target object to be marked, ie, the length, width, height, and heading angle during the process of determining the third position information.
[0117] S530: For each of the target objects to be labeled, determine the target position information of the target object to be labeled according to the first position information, the second position information and the third position information of the target object to be labeled.
[0118] Specifically, after determining the first position information, the second position information and the third position information of the target object to be marked, the processing device may jointly determine the target position information of the target object to be marked according to the first position information, the second position information and the third position information.
[0119] Optionally, for each of the target objects to be labeled, the processing device may perform a weighted sum of the first position information, the second position information, and the third position information of the target object to be labeled to determine the target position information of the target object to be labeled. Wherein, when performing the weighted sum, the weight values corresponding to the first position information, the second position information, and the third position information may be determined by the processing device according to the confidence of each position information. The confidence may be used to characterize the credibility of the position information. The lower the confidence, the lower the credibility of the position information, that is, the greater the deviation between the position information and the true value.
[0120] Optionally, in the process of determining the target location information, in order to prevent location information with too low confidence from affecting the target location information, the processing device may also directly set the weight value of the first location information, the second location information, or the third location information whose confidence is less than or equal to the preset confidence threshold to zero when detecting that the confidence of any one of the first location information, the second location information, or the third location information is less than or equal to the preset confidence threshold. The preset confidence threshold will be set and adjusted according to actual conditions.
[0121] S540: Mark the position of each target object to be marked relative to the current object in the image data according to the target position information.
[0122] Specifically, after determining the target position information of each target object to be marked, the processing device may mark the position of each target object to be marked in the image data according to the target position information.
[0123] Therefore, the processing device determines the first position information, second position information and third position information of the target object to be labeled by using multiple methods, and then fuses the first position information, the second position information and the third position information to determine the target position information, and then labels the position of the target object to be labeled according to the target position information, thereby further improving the accuracy and recall rate of the labeling results.
[0124] Fig.11 FIG. 1 is a schematic diagram of a process for determining target location information according to an embodiment of the present invention. Fig.11 As shown, the process of determining the target location information includes:
[0125] Specifically, after acquiring the image data 112 and the corresponding point cloud data 113 , the processing device may determine the attribute information 111 of at least one target object according to the point cloud data 113 .
[0126] Further, the processing device may determine a first bounding box of each target object to be labeled in the image data 112, and project each target object from the three-dimensional space to the plane where the image data 112 is located according to the attribute information 111 of each target object to determine a second bounding box of each target object in the image data 112. After determining the first bounding box of each target object to be labeled and the second bounding box of each target object, the processing device may match each first bounding box with each second bounding box to determine the corresponding relationship between each target object and each target object to be labeled. After determining the corresponding relationship between each target object to be labeled and each target object, for each target object to be labeled, the processing device may determine the first position information 114 of the target object to be labeled according to the attribute information 111 of the target object corresponding to the target object to be labeled.
[0127] Therefore, the processing device can determine the first position information 114 of each target object to be marked by performing frame association.
[0128] Furthermore, for each target object to be labeled in the image data 112, the processing device can determine the spatial position information of each preset annotation point by matching the preset annotation points corresponding to each target object to be labeled with each spatial point in the point cloud data 113, and then determine the second position information 115 of each target object to be labeled based on the spatial position information.
[0129] Therefore, the processing device can determine the second position information 115 of each target object to be marked by performing point association.
[0130] Further, after determining the correspondence between each target object to be labeled and the target object according to the attribute information 111 and the image data 112, the processing device may also determine the third position information 116 of each target object to be labeled according to the target space point of each target object to be labeled. The target space point is the space point closest to the current object in the point cloud data 113 corresponding to the target object to be labeled.
[0131] Therefore, the processing device can determine the third position information 116 of each target object to be marked by first performing frame association and then performing point association.
[0132] Furthermore, after determining the first position information 114 , the second position information 115 and the third position information 116 of each target object to be labeled, the processing device can determine the target position information 117 of each target object to be labeled based on the first position information 114 , the second position information 115 and the third position information 116 .
[0133] The position labeling method of the embodiment of the present invention determines the attribute information of at least one target object based on the point cloud data, and determines the correspondence between at least one target object to be labeled and each target object in the image data based on the attribute information, and then determines the first position information of each target object to be labeled based on the attribute information of the target object corresponding to each target object to be labeled, and then labels the position of each target object to be labeled relative to the current object in the image data at least based on the first position information. The attribute information is used to characterize the position and shape of the target object in three-dimensional space. The method can improve the accuracy and recall rate of the labeling results, thereby meeting the training requirements of the ranging model and improving the performance of the trained ranging model.
[0134] Fig.12 FIG. 1 is a schematic diagram of a position indexing device according to an embodiment of the present invention. Fig.12 As shown, the position labeling device of the embodiment of the present invention includes an acquiring unit 121 , an attribute information determining unit 122 , an associating unit 123 , a position information determining unit 124 and a labeling unit 125 .
[0135] Specifically, the acquisition unit 121 is used to acquire image data and corresponding point cloud data;
[0136] an attribute information determining unit, configured to determine attribute information of at least one target object according to the point cloud data, wherein the attribute information is used to characterize a position and a shape of the target object in a three-dimensional space;
[0137] The association unit 122 is used to determine the corresponding relationship between at least one target object to be labeled in the image data and each of the target objects according to the attribute information;
[0138] The position information determining unit 123 is used to determine, for each target object to be labeled, first position information of the target object to be labeled according to attribute information of the target object corresponding to the target object to be labeled;
[0139] The labeling unit 124 is used to label the position of each target object to be labeled relative to the current object in the image data at least according to the first position information.
[0140] The position labeling device of the embodiment of the present invention determines the attribute information of at least one target object based on the point cloud data, and determines the correspondence between at least one target object to be labeled and each target object in the image data based on the attribute information, and then determines the first position information of each target object to be labeled based on the attribute information of the target object corresponding to each target object to be labeled, and then labels the position of each target object to be labeled relative to the current object in the image data at least based on the first position information. The attribute information is used to characterize the position and shape of the target object in three-dimensional space. The device can improve the accuracy and recall rate of the labeling results, thereby meeting the training requirements of the distance measurement model and improving the performance of the trained distance measurement model.
[0141] Fig.13 Schematic diagram of an electronic device according to an embodiment of the present invention. Fig.13 As shown, Fig.13 The electronic device shown is a general data processing device, which includes a general computer hardware structure, which includes at least a processor 131 and a memory 132. The processor 131 and the memory 132 are connected via a bus 133. The memory 132 is suitable for storing instructions or programs executable by the processor 131. The processor 131 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 131 executes the instructions stored in the memory 132, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 133 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 134 and the display device and the input / output (I / O) device 135. The input / output (I / O) device 135 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 135 is connected to the system through an input / output (I / O) controller 136.
[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0143] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.
[0144] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.
[0145] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.
[0146] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.
[0147] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0148] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A location indexing method, characterized in that: The method comprises: Obtain image data and corresponding point cloud data; Determining attribute information of at least one target object according to the point cloud data, wherein the attribute information is used to characterize the position and shape of the target object in three-dimensional space; Determine, according to the attribute information, a correspondence between at least one target object to be labeled in the image data and each of the target objects; For each of the target objects to be labeled, determining first position information of the target object to be labeled according to attribute information of a target object corresponding to the target object to be labeled; The position of each of the target objects to be marked relative to the current object is marked in the image data at least according to the first position information.
2. The method according to claim 1, characterized in that Each target object to be annotated in the image data has at least one corresponding preset annotation point; Marking the position of each of the target objects to be marked relative to the current object in the image data at least according to the first position information includes: For each of the target objects to be labeled, determining second position information of the target object to be labeled according to the point cloud data and a preset labeling point corresponding to the target object to be labeled; For each of the target objects to be labeled, determining the third position information of the target object to be labeled according to the spatial position information of the target spatial point of the target object to be labeled, wherein the target spatial point is the spatial point closest to the current object in the point cloud data corresponding to the target object to be labeled; For each of the target objects to be labeled, determining the target position information of the target object to be labeled according to the first position information, the second position information and the third position information of the target object to be labeled; The position of each target object to be marked relative to the current object is marked in the image data according to the target position information.
3. The method according to claim 2, characterized in that Determining the second position information of the target object to be annotated according to the point cloud data and the preset annotation points corresponding to the target object to be annotated includes: Determining spatial position information of each of the preset annotation points according to the point cloud data, wherein the spatial position information is used to represent the position in the three-dimensional space; The second position information of the target object to be marked is determined according to the spatial position information and relative position information of each of the preset marking points, wherein the relative position information is used to represent the position of the preset marking point on the corresponding target object to be marked.
4. The method according to claim 3, characterized in that Determining the spatial position information of each of the preset annotation points according to the point cloud data includes: Projecting the point cloud data onto a plane where the image data is located to determine a set of projection points in the image data; For each of the preset marking points, the spatial position information of the projection point in the projection point set that is closest to the preset marking point is determined as the spatial position information of the preset marking point.
5. The method according to claim 2, characterized in that: Before determining the third position information of the target object to be marked according to the target space point of the target object to be marked, the method includes: Determine the target space point of each target object to be marked.
6. The method according to claim 5, characterized in that Determining the target space point of each target object to be marked includes: For each of the target objects to be labeled, determining the point cloud data corresponding to the target object to be labeled according to the attribute information of the target object corresponding to the target object to be labeled; and A target space point of the target object to be labeled is determined in the point cloud data corresponding to the target object to be labeled.
7. The method according to claim 2, characterized in that Determining the target position information of the target object to be marked according to the first position information, the second position information and the third position information of the target object to be marked includes: For each of the target objects to be labeled, a weighted sum is performed on the first position information, the second position information and the third position information of the target object to be labeled to determine the target position information of the target object to be labeled.
8. The method according to claim 7, characterized in that Performing a weighted sum of the first position information, the second position information, and the third position information of the target object to be marked to determine the target position information of the target object to be marked includes: Determining confidence levels of the first location information, the second location information, and the third location information; Determining weight values corresponding to the first location information, the second location information, and the third location information according to the confidence level; A weighted sum is performed on the first position information, the second position information and the third position information of the target object to be marked based on the weight value to determine the target position information of the target object to be marked.
9. The method according to claim 8, characterized in that The method further comprises: In response to detecting that the confidence of any one of the first location information, the second location information or the third location information is less than or equal to a preset confidence threshold, the weight value of the first location information, the second location information or the third location information whose confidence is less than or equal to the preset confidence threshold is set to zero.
10. The method according to claim 1, characterized in that Determining the correspondence between at least one target object to be labeled in the image data and each of the target objects according to the attribute information includes: Determining a first bounding box of each of the target objects to be labeled in the image data; determining a second bounding box of each of the target objects in the image data according to the attribute information; Each of the first bounding boxes is matched with each of the second bounding boxes to determine a corresponding relationship between each of the target objects and each of the target objects to be labeled.
11. The method according to claim 10, characterized in that Determining a second bounding box of each of the target objects in the image data according to the attribute information includes: Projecting each of the target objects onto a plane where the image data is located according to the attribute information to determine a three-dimensional bounding box of each of the target objects in the image data; For each of the target objects, a second bounding box of the target object is determined according to the three-dimensional bounding box of the target object.
12. A position indexing device, characterized in that: The device comprises: An acquisition unit, used for acquiring image data and corresponding point cloud data; an attribute information determining unit, configured to determine attribute information of at least one target object according to the point cloud data, wherein the attribute information is used to characterize a position and a shape of the target object in a three-dimensional space; an associating unit, configured to determine a corresponding relationship between at least one target object to be annotated in the image data and each of the target objects according to the attribute information; A position information determining unit, configured to determine, for each of the target objects to be labeled, first position information of the target object to be labeled according to attribute information of a target object corresponding to the target object to be labeled; A labeling unit is used to label the position of each target object to be labeled relative to the current object in the image data at least according to the first position information.
13. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 11 when executed by a processor.
14. An electronic device, characterized in that: The device comprises: a memory for storing one or more computer program instructions; A processor, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 11.
15. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.