Obstacle labeling method and device, equipment and storage medium

Abstract

The invention provides an obstacle marking method and device, equipment and a storage medium, and relates to the technical field of data processing and image processing, in particular to the fields of artificial intelligence, big data, automatic driving, auxiliary driving, autonomous parking and the like. According to the specific implementation scheme, a 2D detection result is obtained, and the 2D detection result is obtained based on a plurality of video frames collected by a target vehicle and at least comprises 2D feature information of a 2D target obstacle in a target environment where the target vehicle is located and a preset label of the 2D target obstacle; a 3D detection result is obtained, and the 3D detection result is obtained based on multi-frame point cloud data collected by the target vehicle and comprises 3D feature information of a 3D target obstacle in the target environment; and obtaining a target label of each 3D target obstacle based on 2D feature information contained in the 2D detection result, a preset label of the 2D target obstacle and 3D feature information contained in the 3D detection result.

Description

Technical Field

[0001] The present disclosure relates to the fields of data processing and image processing technology, and in particular to the fields of artificial intelligence, big data, autonomous driving, and autonomous parking. Background Art

[0002] In autonomous driving or assisted driving systems, highly accurate environmental perception is the key to ensuring safe and efficient navigation. In this environmental perception process, accurate identification and positioning of obstacles are crucial for path planning and obstacle avoidance strategies. Accurate identification and positioning of obstacles (collectively referred to as obstacle detection) rely on a large amount of labeled data. Therefore, how to efficiently complete data labeling has become the current focus. Summary of the invention

[0003] The present invention provides a method, device, equipment and storage medium for marking obstacles.

[0004] According to one aspect of the present disclosure, a method for marking obstacles is provided, comprising:

[0005] Obtaining a 2D detection result, wherein the 2D detection result is obtained based on a plurality of video frames collected by the target vehicle, and at least includes 2D feature information of a 2D target obstacle in a target environment where the target vehicle is located and a preset label of the 2D target obstacle;

[0006] Obtaining a 3D detection result, wherein the 3D detection result is obtained based on multiple frames of point cloud data collected by the target vehicle, and includes 3D feature information of the 3D target obstacle in the target environment;

[0007] Based on the 2D feature information and the preset label of the 2D target obstacle included in the 2D detection result, and the 3D feature information included in the 3D detection result, the target label of each 3D target obstacle included in the 3D detection result is obtained.

[0008] According to another aspect of the present disclosure, there is provided a device for marking obstacles, comprising:

[0009] A result determination unit is used to obtain a 2D detection result, wherein the 2D detection result is obtained based on a plurality of video frames collected by the target vehicle, and at least includes 2D feature information of a 2D target obstacle in a target environment where the target vehicle is located and a preset label of the 2D target obstacle; and obtain a 3D detection result, wherein the 3D detection result is obtained based on a plurality of frames of point cloud data collected by the target vehicle, and includes 3D feature information of a 3D target obstacle in the target environment;

[0010] The labeling unit is used to obtain a target label of each 3D target obstacle based on the 2D feature information and the preset label of the 2D target obstacle included in the 2D detection result, and the 3D feature information included in the 3D detection result.

[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0017] In this way, the disclosed solution provides a specific solution for labeling 3D target obstacles using 2D detection results to obtain target labels of 3D target obstacles. This solution does not require manual labeling and can be automatically implemented through offline data (for example, multiple offline video frames and multiple frames of point cloud data). In this way, the labeling efficiency is effectively improved, thereby providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 The schematic flow chart of the obstacle marking method according to the embodiment of the present disclosure is as follows: Figure 1 ;

[0021] Figure 2 The schematic flow chart of the obstacle marking method according to the embodiment of the present disclosure is as follows: Figure 2 ;

[0022] Figure 3 The schematic flow chart of the obstacle marking method according to the embodiment of the present disclosure is as follows: Figure 3 ;

[0023] Figure 4 is a flowchart of a method for marking obstacles according to an embodiment of the present disclosure in a specific example;

[0024] Figure 5 is a schematic flow chart of static obstacle detection according to an embodiment of the present disclosure;

[0025] Figure 6 is a schematic flow chart of a sight distance improvement module according to an embodiment of the present disclosure;

[0026] Figure 7 is a schematic flow chart of a false detection filtering module according to an embodiment of the present disclosure;

[0027] Figure 8 is a schematic diagram of the structure of an obstacle marking device according to an embodiment of the present disclosure;

[0028] Fig. 9 It is a block diagram of an electronic device used to implement the obstacle marking method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0031] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0032] The following is a description of the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be arbitrarily combined with the technical solutions of the embodiments of the present disclosure, and they all belong to the protection scope of the embodiments of the present disclosure.

[0033] Labeled data usually includes two categories: dynamic obstacle labeling data and static obstacle labeling data. Here, static obstacles involve various construction areas and special scenes. The accuracy and efficiency of labeling often affect the actual application effect of autonomous driving or assisted driving technology.

[0034] Based on this, the disclosed scheme proposes an offline automatic labeling method, and in particular provides a method for offline automatic labeling of stationary obstacles. This method achieves efficient, accurate, and easily expandable automatic labeling of stationary obstacles without relying on three-dimensional (Dimensional, D) point cloud detection, such as 4D (including three-dimensional space and one-dimensional time) automatic labeling. Moreover, it can also effectively improve the labeling line of sight (for example, to 200 meters). In this way, while improving the labeling efficiency, the overall labeling effect of stationary obstacles is improved.

[0035] Specifically, Figure 1 This is a schematic flow chart of a method for marking obstacles according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0036] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0037] Step S101: Obtain 2D detection results.

[0038] Here, the 2D detection result is obtained based on multiple video frames collected by the target vehicle (such as an image acquisition component on the target vehicle). Further, the 2D detection result may at least include 2D feature information of the 2D target obstacle in the target environment where the target vehicle is located and a preset label of the 2D target obstacle (for example, the preset label includes a preset type, that is, the 2D detection result may specifically include a preset category of the 2D target obstacle).

[0039] Further, in one example, the 2D detection result may include 2D feature information of all 2D obstacles detected based on multiple video frames. For example, in one example, the multiple video frames collected by the target vehicle in the target environment may be input into the target detection model to obtain a 2D detection result including 2D feature information of all 2D obstacles. Here, the 2D target obstacle is one of all 2D obstacles.

[0040] Here, in one example, the 2D feature information may include at least one of the following: the position of the 2D target obstacle (for example, it may be represented by a 2D target box), the size of the 2D target obstacle, the center point of the 2D target obstacle, etc.

[0041] It is understandable that the 2D feature information includes but is not limited to the above information. In practical applications, other 2D feature information is obtained based on detection accuracy or detection requirements, and the present disclosure does not limit this.

[0042] Furthermore, it should be noted that, in order to facilitate the distinction from the obstacles represented by the subsequent point cloud data, in this example, the target obstacle detected based on the video frame is referred to as a 2D target obstacle. Accordingly, the target frame of the 2D target obstacle can be referred to as a 2D target frame.

[0043] Furthermore, the target obstacle represented by the point cloud data is called a 3D target obstacle; accordingly, the target frame of the 3D target obstacle can be called a 3D target frame.

[0044] Step S102: Obtain 3D detection results.

[0045] Here, the 3D detection result is obtained based on multiple frames of point cloud data collected by the target vehicle (eg, a laser radar on the target vehicle). Further, the 3D detection result may include 3D feature information of the 3D target obstacle in the target environment.

[0046] For example, in one example, the 3D feature information may include at least one of the following: the position of the 3D target obstacle (for example, represented by a 3D target box), the center point of the point cloud data of the 3D target obstacle, the orientation of the 3D target obstacle, the size of the 3D target obstacle, etc.

[0047] It is understandable that the 3D feature information includes but is not limited to the above information. In actual applications, other 3D feature information is obtained based on other needs, and the present disclosure does not limit this.

[0048] Step S103: obtaining a target label of each 3D target obstacle based on the 2D feature information and preset labels of the 2D target obstacles included in the 2D detection result, and the 3D feature information included in the 3D detection result.

[0049] That is to say, in this example, the target label of each 3D target obstacle can be obtained based on the 2D feature information, the preset label of the 2D target obstacle, and the 3D feature information; further, in one example, after obtaining the target label of each 3D target obstacle, the 3D detection result can also be updated, so that the 3D detection result not only includes the 3D feature information of each 3D target obstacle, but also includes the target label of each 3D target obstacle.

[0050] In this way, the disclosed solution provides a specific solution for labeling 3D target obstacles using 2D detection results to obtain target labels of 3D target obstacles. This solution does not require manual labeling and can be automatically implemented through offline data (for example, multiple offline video frames and multiple frames of point cloud data). In this way, the labeling efficiency is effectively improved, thereby providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0051] Moreover, the labeling process of 3D target obstacles in the disclosed solution does not rely on the 3D point cloud detection method, so it can effectively solve the problem of inaccurate labeling results caused by sparse point clouds in the distance, and at the same time, it has strong scalability. For example, existing 3D point cloud detection often relies on point cloud detection models. At this time, if a new category of obstacles is detected, it is necessary to label the new category of obstacles and retrain the point cloud detection model. This method has problems such as high cost, poor scalability, and long iteration cycle, and the disclosed solution can effectively avoid such problems.

[0052] Furthermore, in a specific example, the target obstacle may be a static obstacle, such as a cone barrel, a crash barrier, a triangle sign, etc. on the road. In this way, the disclosed solution can be used to automatically mark the static obstacles on the road, thereby improving the overall marking effect of the static obstacles and providing reliable data support for the subsequent improvement of the actual application effect of the autonomous driving technology.

[0053] For example, in one example, the target obstacles to be labeled can also be determined in the following manner, thus enriching the application scenarios of the disclosed solution. In other words, the disclosed solution is both practical and applicable, and at the same time, it is also scalable, thereby providing strong support for the subsequent implementation of targeted labeling and the improvement of the actual application effect of autonomous driving technology. Specifically, after checking and obtaining all 2D obstacles in the target environment where the target vehicle is located (for example, the multiple video frames collected by the target vehicle in the target environment are input into the target detection model to obtain all 2D obstacles in the target environment where the target vehicle is located), static obstacles are determined from all 2D obstacles in the target environment where the target vehicle is located (for example, the preset labels of each 2D obstacle can also be detected); from all the determined static obstacles, static obstacles whose preset labels meet the label requirements are selected to determine the 2D target obstacles, for example, the selected static obstacles whose preset labels meet the label requirements are used as the 2D target obstacles. Furthermore, a static obstacle whose preset tag is a designated tag may be selected and used as the 2D target obstacle, thereby effectively enriching the application scenarios of the disclosed solution.

[0054] The following is a detailed description of the specific scheme for obtaining 3D detection results:

[0055] Specifically, Figure 2 This is a schematic flow chart of a method for marking obstacles according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0056] Further, the method includes at least part of the following contents. Figure 2 As shown, including:

[0057] Step S201: Obtain 2D detection results.

[0058] Here, the 2D detection result is obtained based on multiple video frames collected by the target vehicle, and at least includes 2D feature information of the 2D target obstacle in the target environment where the target vehicle is located and a preset label of the 2D target obstacle.

[0059] It should be noted that the relevant examples of 2D detection results can be found in the above description, which will not be repeated here.

[0060] Step S202: Based on the 2D detection result, determine the target point corresponding to the 2D target obstacle in the 2D detection result from the multi-frame point cloud data collected by the target vehicle (for example, the laser radar of the target vehicle).

[0061] Step S203: Based on all the determined target points, a target global map of the target environment where the target vehicle is located is obtained.

[0062] Here, in an example, considering that the point cloud intensity of the point cloud data corresponding to the small 2D targets is relatively strong, after obtaining the target global map, the target points in the target global map can also be processed based on the point cloud intensity to filter out the target points whose point cloud intensity is less than the preset intensity threshold, and then update the target global map, thereby improving the accuracy of the obtained target global map.

[0063] Step S204: Determine the 3D target obstacles in the target global map to obtain the 3D detection result.

[0064] Here, the 3D detection result is obtained based on multiple frames of point cloud data collected by the target vehicle (eg, a laser radar on the target vehicle). Further, the 3D detection result may include 3D feature information of the 3D target obstacle in the target environment.

[0065] It should be noted that the relevant examples of 3D detection results can be found in the above description, which will not be repeated here.

[0066] Step S205: obtaining a target label of each 3D target obstacle based on the 2D feature information and preset labels of the 2D target obstacles included in the 2D detection result, and the 3D feature information included in the 3D detection result.

[0067] In this way, the disclosed solution provides a refined solution for obtaining 3D detection results using 2D detection results. The solution is simple, efficient, and does not rely on 3D point cloud detection methods. In this way, it can effectively solve the problem of inaccurate labeling results caused by sparse point clouds in the distance. At the same time, it has strong scalability. In this way, it provides strong support for subsequent automated labeling, model training, and improving the actual application effect of autonomous driving technology.

[0068] Further, in a specific example, the target point corresponding to the 2D target obstacle in the 2D detection result may be obtained in the following manner. Specifically, the above-described step of determining the target point corresponding to the 2D target obstacle in the 2D detection result from the multi-frame point cloud data collected by the target vehicle based on the 2D detection result (for example, the above-described step S202) may specifically include:

[0069] Step S202 - 1 : Projecting points in each frame of point cloud data in the multiple frames of point cloud data into each video frame of the multiple video frames to obtain initial projection points of each point in each frame of point cloud data.

[0070] For example, in one example, based on the above method, the initial projection point of each point in each frame of point cloud data on each video frame can be obtained.

[0071] For example, in a specific example, the initial projection point of each point in each frame of point cloud data can be obtained in the following manner. Specifically, the above-mentioned projecting the points in each frame of point cloud data in the multiple frames of point cloud data into each video frame of the multiple video frames to obtain the initial projection point of each point in each frame of point cloud data (for example, the above-mentioned step S202-1) includes:

[0072] Step S202-1-1: Determine the video frame corresponding to each frame of point cloud data.

[0073] For example, based on the timestamp, the video frame corresponding to each frame of point cloud data is determined. For example, in one example, a frame of point cloud data and a video frame belonging to the same moment can be determined based on the timestamp. At this time, a frame of point cloud data and a video frame belonging to the same moment correspond to each other.

[0074] Step S202-1-2: Project the points in each frame of point cloud data to the video frame corresponding to each frame of point cloud data to obtain the initial projection points of each point in each frame of point cloud data.

[0075] That is to say, the points in each frame of point cloud data are projected onto the video frame belonging to the same time as the frame of point cloud data, so as to obtain the initial projection points of the frame of point cloud data on the video frame belonging to the same time.

[0076] In this way, the disclosed scheme provides a refined scheme for obtaining the initial projection points, which is simple, efficient, and highly interpretable. In this way, the projection method can be used to quickly determine the points in the 3D point cloud data corresponding to the 2D target obstacles, thereby providing strong support for quickly obtaining the target global map and then obtaining the 3D detection results.

[0077] Step S202-2: The point whose initial projection point in each frame of point cloud data falls within the range of the 2D target obstacle in the video frame is taken as the target point.

[0078] That is to say, for each frame of point cloud data, the points whose initial projection points fall within the range of the 2D target obstacle in the projected video frame (for example, within the 2D target box) can be used as target points. At this time, the target point can be considered as the point corresponding to the 2D target obstacle in the 3D point cloud data (that is, the multi-frame point cloud data described above). In this way, using the above mapping method, all target points corresponding to all 2D target obstacles can be obtained, and then a target global map of the target environment where the target vehicle is located, which is mainly composed of all target points, can be obtained.

[0079] For example, in one example, after obtaining the initial projection points of each point in each frame of point cloud data on each video frame, video frames with 2D target obstacles are selected from all video frames, and then it is determined whether the initial projection points of each point in each frame of point cloud data in the video frame with the 2D target obstacle fall within the 2D target frame of the video frame with the 2D target obstacle. If so, the point corresponding to the initial projection point falling within the 2D target frame is used as the target point.

[0080] In this way, the disclosed solution provides a solution for quickly determining the refined target points corresponding to 2D target obstacles in 3D point cloud data. It is simple and efficient, and provides strong support for quickly obtaining a global map of the target and then obtaining 3D detection results. Moreover, the solution does not rely on the 3D point cloud detection method, so it can effectively solve the problem of inaccurate annotation results caused by sparse point clouds in the distance, and at the same time, it has strong scalability. In this way, it provides strong support for subsequent automated annotation, and then for model training, and improving the actual application effect of autonomous driving technology.

[0081] Further, in a specific example, the 3D target obstacle may be determined in the following manner to obtain the 3D detection result. The above-mentioned determination of the 3D target obstacle in the target global map to obtain the 3D detection result (for example, the above-mentioned step S204) may specifically include:

[0082] Step S204 - 1 : Projecting each target point in the target global map into a video frame of a 2D target obstacle to obtain a target projection point of each target point.

[0083] That is, each target point is projected into a video frame where a 2D target obstacle exists to obtain a target projection point of each target point.

[0084] For example, in one example, the above-mentioned projecting each target point in the target global map into the video frame (ie) of the 2D target obstacle to obtain the target projection point of each target point (eg, step S204-1) may specifically include:

[0085] The target global map is converted to the target coordinate system corresponding to the video frame, for example, the target global map is converted to the coordinate system of the image acquisition component corresponding to the video frame, and then the target global map in the coordinate system of the image acquisition component is obtained, and further, each target point in the target global map after the coordinate system conversion is projected into the video frame in the 2D target obstacle to obtain the target projection point of each target point. In this way, it provides strong support for quickly and accurately determining 3D target obstacles using 2D inspection results, and then provides strong support for subsequent automated labeling, model training, and improving the actual application effect of autonomous driving technology.

[0086] Step S204-2: Based on the target projection points falling within the range of the 2D target obstacle (for example, falling within the 2D target frame of the 2D target obstacle), determine the target point group of the 3D target obstacle in the target global map to obtain the 3D detection result.

[0087] For example, in one example, after obtaining the target projection points that fall within the range of the 2D target obstacle, the target projection points that fall within the range of the 2D target obstacle can also be filtered, so as to further improve the accuracy of the obtained 3D detection results, thereby providing strong support for subsequent accurate and automated labeling, model training, and improving the actual application effect of autonomous driving technology. Specifically, the above-mentioned determination of the target point group of the 3D target obstacle in the target global map based on the target projection points that fall within the range of the 2D target obstacle to obtain the 3D detection result (for example, step S2024-2) can specifically include:

[0088] Step S204-2-1: taking the target projection points that fall within the range of the 2D target obstacle (for example, fall within the 2D target frame of the 2D target obstacle) as the initial point group of the 3D target obstacle corresponding to the 2D target obstacle.

[0089] Step S204-2-2: Perform at least one of noise filtering and preset point cloud computing on the initial point group of the 3D target obstacle corresponding to the 2D target obstacle to determine the target point group of the 3D target obstacle in the target global map and obtain the 3D detection result.

[0090] For example, in one example, a density-based spatial clustering application (DBSCAN) algorithm can be used to perform noise filtering on an initial point group of target projection points that fall within a 2D target box of a 2D target obstacle, and a random sampling consensus (RANSAC) algorithm is used to process the initial point group after noise filtering to estimate the 3D feature information of the 3D target obstacle, thereby obtaining the 3D detection result.

[0091] Thus, the present disclosure provides a refined solution for obtaining 3D target obstacles and then obtaining 3D detection results. The solution is simple and efficient, and the accuracy of the obtained 3D detection results is high. Moreover, the solution does not rely on the 3D point cloud detection method, so it can effectively solve the problem of inaccurate labeling results caused by sparse point clouds in the distance, and at the same time, it has strong scalability. In this way, it provides strong support for subsequent automatic labeling, and then for model training, and improving the actual application effect of autonomous driving technology.

[0092] The following is a detailed description of the specific scheme for obtaining the target label of the 3D target obstacle:

[0093] Specifically, Figure 3 Schematic flow of the obstacle marking method according to an embodiment of the present application Figure 3 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 or Figure 2 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0094] Further, the method includes at least part of the following contents. Figure 3 As shown, including:

[0095] Step S301: Obtain 2D detection results.

[0096] Here, the 2D detection result is obtained based on multiple video frames collected by the target vehicle, and at least includes 2D feature information of the 2D target obstacle in the target environment where the target vehicle is located and a preset label of the 2D target obstacle.

[0097] It should be noted that the relevant examples of 2D detection results can be found in the above description, which will not be repeated here.

[0098] Step S302: Obtain 3D detection results.

[0099] Here, the 3D detection result is obtained based on the multi-frame point cloud data collected by the target vehicle, and includes the 3D feature information of the 3D target obstacles in the target environment.

[0100] It should be noted that the relevant examples of 3D detection results can be found in the above description, which will not be repeated here.

[0101] Step S303: Obtaining target association information based on the 2D feature information of the 2D target obstacle included in the 2D detection result and the 3D feature information of the 3D target obstacle included in the 3D detection result.

[0102] Here, the target association information at least represents a mapping relationship between the 2D target obstacle and the 3D target obstacle.

[0103] Step S304: When it is determined based on the target association information that there is a 2D target obstacle associated with the 3D target obstacle, a preset label of the 2D target obstacle is assigned to the 3D target obstacle associated with the 2D target obstacle to obtain a target label of at least part of the 3D target obstacle included in the 3D detection result.

[0104] In this way, the disclosed scheme provides a scheme for quickly obtaining the refined target labels of at least some 3D target obstacles contained in the 3D detection results by utilizing the target association relationship. The scheme is simple and efficient, and can quickly determine the target labels of 3D target obstacles with associated relationships. Moreover, the scheme does not rely on the 3D point cloud detection method, and thus can effectively solve the problem of inaccurate labeling results caused by sparse point clouds in the distance. At the same time, it has strong scalability, and thus provides strong support for subsequent accurate and automated labeling, model training, and improving the actual application effect of autonomous driving technology.

[0105] Further, in a specific example, the target association information may be obtained in the following manner. Specifically, the above-described obtaining the target association information based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result (for example, the above-described step S303) may specifically include:

[0106] Step S303 - 1 : Based on the 2D feature information of the 2D target obstacle included in the 2D detection result and the 3D feature information of the 3D target obstacle included in the 3D detection result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

[0107] Step S303 - 2 : obtaining the target association information based on the feature value representing the similarity between the 2D target obstacle and the 3D target obstacle.

[0108] That is to say, the disclosed solution provides a method for quickly determining the association relationship between the 2D target obstacle and the 3D target obstacle by utilizing the feature value representing the similarity between the 2D target obstacle and the 3D target obstacle. For example, in one example, the 2D target obstacle and the 3D target obstacle whose feature value representing the similarity is greater than a preset value can be determined to have an association relationship. Otherwise, it can be determined that there is no association relationship between the two. In this way, the target association relationship is quickly obtained. This method is simple, efficient, and highly accurate, and provides strong support for the subsequent use of target association information to quickly obtain the target labels of at least some of the 3D target obstacles contained in the 3D detection results.

[0109] Further, in one example, the following method can be used to obtain a feature value representing the similarity, which can provide strong support for quickly obtaining the target association relationship. Moreover, the method is simple, versatile, and interpretable, while also taking into account efficiency. Specifically, the above-mentioned 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result to obtain a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle (for example, the above-mentioned step S303-1) can specifically include:

[0110] Step S303-1-1: Based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, obtain at least one of the following: intersection ratio, shape similarity, area difference value, and effective distance.

[0111] Step S303-1-2: Based on at least one of the calculated intersection-over-union ratio, shape similarity, and area difference value, a feature value characterizing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

[0112] That is to say, based on the 2D feature information of the 2D target obstacle and the 3D feature information of the 3D target obstacle, at least one of the intersection-and-union ratio, shape similarity, area difference value and effective distance between the 2D target obstacle and the 3D target obstacle can be obtained. In this way, using at least one of the obtained intersection-and-union ratio, shape similarity and area difference value, a feature value characterizing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

[0113] For example, in one example, the feature value representing the similarity may be obtained in the following manner. Specifically, the feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained based on at least one of the calculated intersection-over-union ratio, shape similarity, and area difference value (for example, step S303-1-2 described above), which may specifically include:

[0114] Performing weighted processing on at least two of the intersection-over-union ratio, the shape similarity, the area difference value, and the effective distance to obtain a weighted result;

[0115] Based on the weighted result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

[0116] In other words, the disclosed solution provides a more flexible solution for determining the characteristic value of similarity. The solution can select appropriate parameters to obtain the characteristic value of similarity based on actual scenario requirements. It is highly flexible and applicable, and can effectively improve the accuracy of the obtained target association relationship.

[0117] The following is a detailed description of a specific scheme for obtaining trajectory information and obtaining or optimizing a target label of a 3D target obstacle based on the trajectory information:

[0118] Specifically, in one example, after obtaining the target global map, the 3D target obstacles included in the target global map may be tracked and associated between frames to obtain the trajectory information of each 3D target obstacle included in the target global map.

[0119] Here, the trajectory information of the 3D target obstacle may include at least one of the following: the position of the 3D target obstacle in the target frame point cloud data. Furthermore, the target frame point cloud data may specifically be any frame in the multi-frame point cloud data.

[0120] Here, it should be noted that, for the convenience of description, the radar frame where the 3D target obstacle is located can be referred to as target frame point cloud data. Further, in an example, assuming that the same 3D target obstacle exists in M ​​(a natural number greater than or equal to 1) target frame point cloud data, at this time, the trajectory information of the 3D target obstacle can be mainly composed of the 3D target frame identifying the 3D target obstacle in each of the M target frame point cloud data, that is, composed of M 3D target frames.

[0121] Furthermore, after obtaining the trajectory information of the 3D target obstacle, the trajectory information can also be associated with the target association relationship obtained above. For example, in one example, the target association relationship can specifically represent the mapping relationship between the 2D target obstacle and the 3D target obstacle, and the trajectory information of the 3D target obstacle; in other words, in one example, the target association information can specifically represent the mapping relationship between the 2D target obstacle, the 3D target obstacle, and the trajectory information of the 3D target obstacle.

[0122] In this way, the disclosed solution provides a refined solution for obtaining the trajectory information of 3D target obstacles using the target global map, thereby providing strong support for the subsequent use of the trajectory information to perform labeling consistency constraints, thereby improving the accuracy of the target labels of the obtained 3D target obstacles.

[0123] Furthermore, in a specific example, after obtaining the trajectory information of the 3D target obstacle, the trajectory information can also be used to mark the 3D target obstacles that have no associated relationship. In this way, on the one hand, the accuracy of the marking results can be effectively improved, and on the other hand, the overall marking rate can be effectively improved, thereby providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0124] Specifically, after assigning the preset label of the 2D target obstacle to the 3D target obstacle associated with the 2D target obstacle based on the target association information, for example, after the above step S304, the obstacle labeling method may further include:

[0125] Step S305: If the 2D target obstacle associated with the 3D target obstacle is not determined based on the target association information, a label prediction is performed on the 3D target obstacle not associated with the 2D target obstacle based on at least the trajectory information of the 3D target obstacle to obtain a prediction result of the 3D target obstacle not associated with the 2D target obstacle. Here, the prediction result includes a prediction label and a probability value of the prediction label.

[0126] Step S306: Based on the predicted labels whose probability values ​​are greater than or equal to the target values, obtain the target labels of the 3D target obstacles that are not associated with the 2D target obstacles.

[0127] It should be noted that in actual applications, due to reasons such as missed detection or occlusion of 2D target obstacles, there are some 3D target obstacles that are not associated with 2D target obstacles. At this time, the trajectory information obtained by the disclosed solution can be used. For example, based on the occlusion information, label prediction can be performed on the 3D target obstacles that are not associated with the 2D target obstacles, and then the prediction result determines the target label of the 3D target obstacle that is not associated with the 2D target obstacle. In this way, the problem of being unable to label the 3D target obstacles due to missed detection or occlusion of 2D target obstacles can be effectively avoided, thereby further improving the overall labeling rate.

[0128] Here, it should be noted that the above-mentioned occlusion information can be obtained based on the following video frames and / or point cloud data: the occlusion of the 3D target obstacle in the video frame corresponding to the 3D target obstacle not associated with the 2D target obstacle, and / or the occlusion of the 3D target obstacle in one or more frames of point cloud data corresponding to the 3D target obstacle not associated with the 2D target obstacle. In this way, data support is provided for improving the accuracy of the prediction results.

[0129] Furthermore, the trajectory information may be used to further verify the annotation result; for example, in one example, after obtaining the target label of the 3D target obstacle, or, or, after the above-mentioned step S306, the obstacle annotation method may further include:

[0130] Step S307: based on the label consistency principle of the same 3D target obstacle in the trajectory information, verify the target label of each 3D target obstacle in the 3D inspection result to obtain a label verification result.

[0131] For example, in one example, the target labels of the same 3D target obstacle in different frames of point cloud data in the trajectory information are cross-validated to obtain the label verification result.

[0132] Step S308: adjusting the target label of the 3D target obstacle in the 3D inspection result based on the label verification result, so that the target label of the same 3D target obstacle in the trajectory information is consistent.

[0133] In this way, the disclosed scheme provides a refined scheme for label verification using trajectory information, so as to achieve target labels for the same 3D target obstacles on the same trajectory information, such as the unification of target categories. This scheme can effectively improve the accuracy of the labeling results, and thus provide reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0134] The following is a detailed description of the specific scheme for using trajectory information to ensure global consistency and improve the annotation sight distance:

[0135] Specifically, in one example, after obtaining the trajectory information of the 3D target obstacle, the trajectory information can also be used to ensure the global position invariance and global existence of the same obstacle, thereby effectively improving the accuracy of the annotation results.

[0136] For example, in one example, after obtaining the trajectory information of the 3D target obstacle, or, further, after obtaining the target label of each 3D target obstacle, or, further, after verifying the target label of the 3D target obstacle (for example, step S308 described above), the method further includes:

[0137] Step S309: obtaining the global position information of the 3D target obstacle based on the position information of the same 3D target obstacle in different target frame point cloud data in the trajectory information of the 3D target obstacle.

[0138] For example, the position information of the same 3D target obstacle in different target frame point cloud data is averaged, such as the point group, to obtain the global position information of the 3D target obstacle.

[0139] For example, in one example, the global position information of the 3D target obstacle can be obtained in the following manner, so as to quickly and accurately determine the global position information of the 3D target obstacle, providing strong support for achieving global position invariance and thereby improving the accuracy of the overall labeling results.

[0140] Specifically, the above-mentioned obtaining the global position information of the 3D target obstacle based on the position information of the same 3D target obstacle in different target frame point cloud data in the trajectory information of the 3D target obstacle (for example, step S309) may specifically include:

[0141] Step S309-1: convert the 3D target obstacle on each target frame point cloud data in the trajectory information of the 3D target obstacle to a preset coordinate system (eg, a world coordinate system) to obtain the position information of the 3D target obstacle on the target frame point cloud data in the preset coordinate system.

[0142] Step S309 - 2 : Based on the position information of the same 3D target obstacle in the preset coordinate system, the global position information of the 3D target obstacle on the target frame point cloud data is obtained.

[0143] Step S310: using the global position information of the 3D target obstacle, at least adjusting the 3D feature information of the 3D target obstacle in the 3D detection result.

[0144] Furthermore, the global position information of the 3D target obstacle can be used to adjust the position of the 3D target obstacle in the target global map, so as to further improve the accuracy of the marked 3D target obstacle and effectively ensure the invariance of the global position.

[0145] In this way, the disclosed scheme provides a refined scheme for using trajectory information to achieve global position invariance of the same obstacle. This scheme can effectively improve the global consistency of the annotation results, and at the same time, improve the accuracy of the annotation results, thereby providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0146] Further, in an example, the global position information described above can also be used to optimize the 2D detection result. For example, missed or blocked 2D target obstacles can be recalled to optimize the 2D detection result based on the recall result. For example, after obtaining the global position information of the 3D target obstacle, for example, after the step S309 described above, the method can further include:

[0147] Step S311: obtaining the projection position of the global position information of the 3D target obstacle on each of the multiple video frames.

[0148] It is understandable that step S311 and step S310 may be executed simultaneously or sequentially, and the disclosed solution does not limit the execution order of the two.

[0149] Furthermore, in one example, the projection position can be obtained in the following manner, which can quickly and accurately obtain the projection position of the global position information of the 3D target obstacle on the video frame, thereby providing strong support for the subsequent recall of missed or occluded 2D target obstacles, achieving global consistency of annotation results, and improving the accuracy of annotation results, and further providing strong support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0150] Specifically, the above-mentioned obtaining of the projection position of the global position information of the 3D target obstacle on each of the multiple video frames includes:

[0151] Step S311 - 1 : converting the global position information of the 3D target obstacle into the coordinate system of the image acquisition device that acquires the video frame (eg, the camera coordinate system) to obtain the global position information of the 3D target obstacle in the coordinate system of the image acquisition device.

[0152] Step S311 - 1 : Projecting the obtained global position information of the 3D target obstacle in the coordinate system of the image acquisition device onto the video frame to obtain the projection position of the global position information of the 3D target obstacle on the video frame.

[0153] That is to say, the global position information of the 3D target obstacle is converted to the camera coordinate system and then back-projected onto the video frame. In this way, the projection position of the 3D target obstacle in the video frame is obtained, thereby providing data support for the subsequent recall of missed or occluded 2D target obstacles.

[0154] Step S312: Based on the projection position of the global position information of the 3D target obstacle on each video frame, it is determined whether there is a 2D target obstacle corresponding to the 3D target obstacle in the video frame to adjust the 2D detection result.

[0155] In this way, the disclosed scheme provides a refined scheme for optimizing 2D detection results using global position information. This scheme can effectively use the global consistency principle to recall missed 2D target obstacles, thereby achieving an improvement in the annotated line of sight, and thus providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0156] Further, in a specific example, the following method can be used to realize the recall of 2D obstacles. Specifically, the above-mentioned projection position based on the global position information of the 3D target obstacle on each video frame determines whether there is a 2D target obstacle corresponding to the 3D target obstacle in the video frame to adjust the 2D detection result (for example, step S312), which can specifically include:

[0157] Step S312-1: when it is determined that the projection position of the global position information of the 3D target obstacle on the video frame falls within a preset range in the video frame, determine whether there is a 2D target obstacle corresponding to the 3D target obstacle whose projection position falls within the preset range in the 2D inspection result.

[0158] Step S312-2: if it is determined that the 2D target obstacle does not exist, the 2D target obstacle corresponding to the 3D target obstacle whose projection position in the video frame falls within a preset range is recalled, and the 2D detection result is updated based on the recalled 2D target obstacle.

[0159] That is to say, first determine whether the projection position of the global position information of the 3D target obstacle on the video frame falls within the preset range of the video frame (the preset range can be set based on the actual recall requirements and can be an empirical value), and then determine if it falls within the preset range of the video frame, determine whether there is a 2D target obstacle corresponding to the 3D target obstacle whose projection position falls within the preset range in the 2D inspection result, and then, if it does not exist, recall the 2D target obstacle corresponding to the 3D target obstacle whose projection position falls within the preset range. In this way, the global consistency principle is used to recall the missed 2D target obstacles, thereby achieving an improvement in the labeled sight distance, and providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0160] The following is a detailed description of the specific scheme for filtering false detection of trajectory information:

[0161] Specifically, in one example, after obtaining the trajectory information of the 3D target obstacle, the following method can be used to perform false detection filtering on the trajectory information of the 3D target obstacle to filter out falsely detected trajectory information, specifically including:

[0162] Obtain target confidence of trajectory information of each 3D target obstacle;

[0163] Based on the target confidence of the trajectory information, the trajectory information is verified to obtain a trajectory verification result, so as to adjust the obtained trajectory information of each 3D target obstacle based on the trajectory verification result.

[0164] It should be noted that, in one example, after obtaining the trajectory information of each 3D target obstacle, the obtained trajectory information can be filtered for false detection, and then the trajectory information after false detection filtering can be used to determine the global position information and improve the annotation visibility. In this way, the accuracy of the annotation results can be further improved.

[0165] In this way, the disclosed scheme provides a refined scheme for filtering false detections of trajectory information, thereby further providing strong support for improving the accuracy of annotation results, and further providing reliable data support for subsequent model training and improving the actual application effect of autonomous driving technology.

[0166] Further, in an example, false detection filtering of trajectory information may be implemented in the following manner. Specifically, the target confidence based on the trajectory information described above verifies the trajectory information to obtain a trajectory verification result, and adjusts the obtained trajectory information of each 3D target obstacle based on the trajectory verification result, which may specifically include:

[0167] When the target confidence of the trajectory information is greater than or equal to a first preset threshold, retaining the trajectory whose target confidence is greater than or equal to the first preset threshold;

[0168] or,

[0169] When the target confidence of the trajectory information is less than the first preset threshold, the trajectory information with the target confidence less than the first preset threshold is determined as a false detection trajectory.

[0170] In other words, this example provides a refined solution for using the target confidence of trajectory information to filter false detections of trajectory information, thereby further providing strong support for improving the accuracy of annotation results.

[0171] Furthermore, in one example, when the target confidence of the trajectory information is less than the first preset threshold, the single-frame confidence of the corresponding video frame can be further obtained. In this way, the false detection granularity can be further improved, and false deletion can be effectively avoided, thereby providing strong support for further improving the accuracy of the annotation results. Specifically, for the determined false detection trajectory, the video frames corresponding to the 3D target obstacles in the false detection trajectory can be further determined; and the single-frame confidence of each video frame corresponding to the 3D target obstacle in the false detection trajectory can be obtained; here, the single-frame confidence represents the confidence of the existence of the 2D target obstacle corresponding to the 3D target obstacle in the video frame; and then based on the single-frame confidence of each video frame corresponding to the 3D target obstacle, it is determined whether to delete the false detection trajectory. For example, in one example, if the single-frame confidence of the video frame corresponding to the 3D target obstacle is greater than or equal to the second preset threshold, the 3D target obstacle on the false detection trajectory with a single-frame confidence greater than or equal to the second preset threshold is retained, and the false detection trajectory is updated until all 3D target obstacles in the false detection trajectory are detected, and then the trajectory information is updated. At this time, the updated trajectory information obtained is the trajectory information after false detection filtering. In this way, on the one hand, false detection is effectively avoided, and on the other hand, mistaken deletion is also effectively avoided, thereby providing strong support for improving the accuracy of the annotation results.

[0172] The following is a further detailed description of the disclosed solution with reference to specific examples:

[0173] In the first part, the basic principle of offline automatic annotation of stationary obstacles in the target environment is briefly described.

[0174] First, the image detection model is used to perform 2D detection on the 2D images collected from the target environment to obtain 2D detection results. The global map detection algorithm is used to obtain the target global map, and the offline detection of stationary obstacles (also called 3D stationary obstacles) is implemented based on the target global map to obtain offline detection results to provide initial values ​​for subsequent optimization.

[0175] Secondly, the stationary obstacles that need to be marked offline in the offline detection results are tracked to obtain the trajectory information of the stationary obstacles, and the trajectory information is further constrained to be globally consistent.

[0176] Again, the stationary obstacles in the 2D detection results (also called 2D stationary obstacles) and the stationary obstacles in the offline detection results (also called 3D stationary obstacles) are associated, and a voting algorithm is used to mark the 3D stationary obstacles in the target global map. For example, the 3D stationary obstacles in the target global map are assigned the best category (such as cone barrels, anti-collision barrels, triangle signs, etc.).

[0177] Finally, according to the static obstacle and the trajectory information corresponding to the static obstacle, the frame without marking the static obstacle within a certain range is projected back to recall the missed static obstacle, thus improving the marked sight distance.

[0178] In the second part, the modules and specific processes involved in the offline automatic labeling of stationary obstacles in the disclosed solution are described in detail.

[0179] like Figure 4 As shown, the offline automatic labeling method of the static obstacle in the disclosed solution mainly involves five modules and their specific processing steps are as follows:

[0180] Module 1: Stationary obstacle detection module

[0181] The goal of the stationary obstacle detection module is to achieve the initial detection function of stationary obstacles to obtain 3D detection results; Figure 5 As shown, the specific implementation process is as follows:

[0182] Step 1.1: Obtain 2D detection results of all video frames (also referred to as video frame images, for example, in one example, one frame may include multiple (such as 7) surround view images) collected by the image acquisition component in the target vehicle in the target environment. For example, all video frame images may be input into a visual 2D detection model to obtain 2D detection results.

[0183] Here, in this example, the 2D detection result may mainly include the positions of all detected 2D obstacles (for example, it can be represented by a target box, here, in order to distinguish it from the target box of the following 3D small target, the target box of the 2D small target may be called a 2D target box), size, center point and other 2D feature information, and preset labels of 2D obstacles (for example, in one example, the preset label may specifically include a preset category of the 2D obstacle).

[0184] Here, it should be noted that, in order to facilitate the distinction from obstacles represented by subsequent point cloud data, in this example, obstacles detected based on video frames are referred to as 2D obstacles.

[0185] Step 1.2: Based on the preset labels of each 2D obstacle in the 2D detection results, select the target obstacle (also referred to as the 2D target obstacle) whose preset label is the specified label. For example, in one example, a static obstacle with a specified label (also referred to as a 2D static obstacle) can be selected, or, further, a static small target (hereinafter referred to as a 2D small target) with a specified label (such as a crash barrier, a construction sign, a triangle sign, etc.) can be selected from the static obstacles.

[0186] Step 1.3: Execute a global map detection algorithm, for example, project the acquired 3D point cloud data for the target environment (for example, multiple frames of point cloud data collected by the laser radar in the target vehicle under the target environment) into each video frame (for example, projected into the video frame of each 2D small target) to obtain the projection points of each point in the 3D point cloud data on the video frame (corresponding to the initial projection points described above); based on the projection points of each point in the 3D point cloud data on the video frame, obtain a global map of the target.

[0187] For example, in one example, for each frame of point cloud data, each point therein can be projected into a video frame associated with the frame of point cloud data to obtain the projection points of each point in the frame of point cloud data. Here, it should be noted that the video frame and radar frame (here, a frame of point cloud data can be called a radar frame) collected at the same time and in the same target environment can be called associated.

[0188] Furthermore, in one example, after obtaining the projection points of the points in each frame of point cloud data on the video frame, it can be determined whether the projection points of the points in each frame of point cloud data fall within the 2D target box of the 2D small target in the video frame; if so, the points whose projection points fall within the 2D target box of the 2D small target in the video frame are determined as static points (corresponding to the target points described above); in this way, after processing all the frame point cloud data, all the static points can be obtained. At this time, based on all the static points, a global map mainly composed of static points can be obtained, which can be called a target global map.

[0189] Furthermore, considering that the point cloud intensity of the point cloud data corresponding to the small 2D targets is relatively strong, the obtained target global map can be further processed to filter out points with point cloud intensity less than a certain threshold, and update the target global map. In this way, the accuracy of the obtained target global map can be improved.

[0190] It should be noted that, in an example, the above steps 1.2 and 1.3 may also be implemented in the following manner:

[0191] Projecting the acquired 3D point cloud data for the target environment (e.g., multiple frames of point cloud data collected by the laser radar in the target vehicle in the target environment) into each video frame to obtain the projection point of each point in the 3D point cloud data on each video frame;

[0192] For each video frame, determine whether there is a 2D image with a specified label in the video frame.

[0193] If a small target exists, the points falling into the 2D target box of the 2D small target are taken as static points. In this way, all static points are obtained to obtain the global map of the target.

[0194] Step 1.4: For each 2D small target in the 2D detection result, the target global map obtained above is converted to radar coordinates, and each static point in the target global map under the radar coordinate system is projected onto the video frame corresponding to the 2D small target to obtain the projection point of each static point (corresponding to the target projection point described above); determine all static points whose projection points fall into the target frame of the 2D small target to obtain the 3D small target corresponding to the 2D small target; after enumerating all 2D small targets, the 3D small target (corresponding to the 3D target obstacle described above) of each 2D small target in the 3D space (e.g., point cloud data) can be obtained according to all static points whose projection points fall into the target frame of the 2D small target.

[0195] It can be understood that the 3D small target in the 3D space is mainly composed of all static points falling into the target box of the 2D small target.

[0196] Step 1.5: Use the DBSCAN algorithm to filter the noise of the point cloud data of the 3D small target obtained above, and then process the point cloud data of the 3D small target after noise filtering through the random sampling consensus (RANSAC) algorithm to estimate the inner points, planes and normal vectors of the point cloud data of the 3D small target, and then obtain the 3D detection result of the target global map.

[0197] Here, the 3D detection result mainly includes the position of each 3D small target (for example, it can be represented by a target frame. Here, in order to distinguish it from the target frame of a 2D small target, the target frame of the 3D small target can be called a 3D target frame), center point, angle, direction, size and other 3D feature information.

[0198] Module 2: Target Tracking Module

[0199] The goal of the target tracking module is to achieve trajectory tracking of small 3D targets. The specific steps include:

[0200] Step 2.1: After obtaining the 3D detection results, use Kalman filtering to track and associate the 3D small targets detected based on the target global map between frames to obtain the trajectory information of each 3D small target contained in the 3D detection results.

[0201] Here, in one example, the trajectory information of the 3D small target may mainly consist of 3D target frames in different radar frames where the 3D small target is located.

[0202] Furthermore, it should be noted here that, for the sake of ease of description, the radar frame where the 3D small target is located can be called target frame point cloud data. Further, assuming that the same 3D small target exists in M ​​target frame point cloud data, at this time, the trajectory information of the 3D small target can be mainly composed of the 3D target frame identifying the 3D small target in each target frame point cloud data in the M target frame point cloud data, that is, composed of M 3D target frames.

[0203] Module 3: Association Module

[0204] The goal of this association module is to associate the 2D small targets in the 2D detection results with the 3D small targets in the 3D detection results based on the characteristics of the small targets, so as to provide a more fine-grained category for the 3D small targets and improve the accuracy of the 3D detection results. At the same time, the types of the 3D small targets in the trajectory information are consistent.

[0205] Specifically, the main steps include:

[0206] Step 3.1: Based on the 2D feature information of the 2D small target in the video frame and the 3D feature information of the 3D small target in the target global map, the 2D small target is associated with the 3D small target to obtain target association information.

[0207] Here, the target association information at least records the mapping relationship between the 2D small target and the 3D small target. Furthermore, the target association relationship may further record the mapping relationship between the 2D small target and the 3D small target, and record the trajectory information of the 3D small target; in other words, in one example, the target association information may specifically record the mapping relationship between the 2D small target, the 3D small target, and the trajectory information.

[0208] Furthermore, in one example, the target association information described above may be obtained in the following manner:

[0209] Step 3.1.1: Calculate the area difference between the area of ​​the 2D small target on the video frame and the area of ​​the 3D small target in the target global map, calculate the intersection over union (IOU) of the 2D small target and the 3D small target, and calculate the shape similarity between the 2D small target and the 3D small target; further, weight at least two of the area difference value, IOU and shape similarity to obtain a feature value that can characterize the similarity between the 2D small target and the 3D small target.

[0210] Here, it should be noted that the weights in the weighted processing process can be set based on the implementation scenario requirements, for example, can be specifically an empirical value, and the present disclosure does not limit this.

[0211] In this way, weighted processing can, on the one hand, increase the corresponding weight for static obstacles with a smaller distance or area, and on the other hand, suppress static obstacles with a larger distance or area, thus making the associated information more accurate.

[0212] Furthermore, the feature value representing the similarity can be calculated in the following way:

[0213] Method 1: For small objects with a large area, the IOU and shape similarity can be considered to obtain the similarity feature value between the 2D small object and the 3D small object. For example, the IOU and shape similarity are weighted to obtain the feature value representing the similarity between the 2D small object and the 3D small object.

[0214] Method 2: For small targets with low IOU (for example, IOU is less than a preset value) (considering that the position of the small target may be inaccurate), the shape similarity of the small target and the similarity of the center points of the two can be considered to obtain the similarity between the 2D small target and the 3D small target. For example, the shape similarity and the distance between the center points are weighted to obtain a feature value that characterizes the similarity between the 2D small target and the 3D small target.

[0215] Method 3: Calculate the projection frame of the 3D target frame of the 3D small target in the video frame, and calculate the distance between the 2D target frame of the 2D small target and the projection frame corresponding to the 3D small target, as well as the area difference between the 2D target frame and the projection frame corresponding to the 3D small target, to obtain a feature value representing the similarity between the 2D small target and the 3D small target. For example, weight the distance between the 2D target frame of the 2D small target and the projection frame corresponding to the 3D small target, and the area difference between the 2D target frame and the projection frame corresponding to the 3D small target, to obtain a feature value representing the similarity between the 2D small target and the 3D small target.

[0216] It should be noted that, in practical applications, considering the problem that the IOU accuracy of larger small targets is poor, an IOU threshold can also be set. When the IOU is less than a certain IOU threshold, no association is required.

[0217] In addition, in another example, discriminative association can also be performed, for example, small targets are classified (for example, based on the size of the small targets), and the small targets in each category are associated, and there is no need to associate between different categories. In this way, false associations can be effectively avoided, thereby further improving the accuracy of target association information.

[0218] It should be noted that the above is only an exemplary description. In actual applications, other methods can be used to associate 2D small targets with 3D small targets, and the present disclosure does not limit this.

[0219] Step 3.2: Based on the target association information of the 2D small target and the 3D small target, determine the target label of the 3D small target with an associated relationship. For example, in one example, the target category of the 3D small target with an associated relationship can be specifically determined. Specifically, based on the target association relationship, determine the 2D small target with an associated relationship with the 3D small target, assign the preset label of the 2D small target with an associated relationship with the 3D small target to the 3D small target, so as to obtain the target label of the 3D small target.

[0220] In this way, the target labels of all small 3D targets with associated relationships can be obtained based on the target association relationship.

[0221] It should be noted that in actual applications, due to missed detection or occlusion of 2D small targets, there are some 3D small targets that are not associated with 2D small targets. In the existing scheme, the category of the unrelated 3D small targets will be forcibly assigned to the position class (unknown), which will lead to the problem of incorrect category of 3D small targets in the labeling results.

[0222] Based on this, the disclosed scheme provides a category voting algorithm to solve the problem of category errors in the labeling results; for example, based on the above step 2.1, the trajectory information of the 3D small target is obtained (or, it can also include occlusion information in the video frame and / or radar frame), and the label (for example, category) of the 3D small target that is not associated with the 2D small target is predicted, for example, the predicted label and its probability value are obtained, and the probability values ​​are sorted, and then the label with the largest probability value is selected as the target label of the 3D small target that is not associated with the 2D small target.

[0223] Furthermore, the target labels of the 3D small targets in different frames of point cloud data can be cross-validated using the label consistency principle of the same 3D small target in the trajectory information obtained above. Alternatively, in another example, the preset labels of the 2D small targets and the target labels of the 3D small targets can be cross-validated to obtain label verification results, and then the target labels of the 3D small targets can be adjusted based on the label verification results, thereby ensuring that the target labels of the same 3D small targets on the same trajectory information are consistent.

[0224] Module 4: Sight Improvement Module

[0225] The purpose of this sight range improvement module is to ensure the global position invariance and global existence of stationary obstacles to improve the sight range and stability of obstacle detection.

[0226] like Figure 6 As shown, the main steps include:

[0227] Step 4.1: Convert the 3D small target on each frame of point cloud data in the trajectory information obtained above to a preset coordinate system (such as a world coordinate system). At this time, for the same 3D small target in different frames of point cloud data on the same trajectory information, the position information of the 3D small target in the preset coordinate system may be different. At this time, different position information can be used and the global average position of the 3D small target (corresponding to the global position information described above) can be obtained by averaging. Furthermore, the average orientation can also be obtained. In this way, the invariance of the global position can be effectively ensured.

[0228] Step 4.2: Traverse all video frames. For example, for each video frame, convert the global average position of the small 3D target to the coordinate system of the image acquisition device corresponding to the video frame (for example, the camera coordinate system), and back-project it into the video frame to obtain the projection position of the global average position of the small 3D target in the camera coordinate system of the video frame.

[0229] Step 4.3: Based on the projection position, determine whether there is a 2D small target corresponding to the 3D small target in the video frame. For example, determine whether there is a 2D small target associated with the 3D small target in the video frame. If so, execute step 4.4; otherwise, execute step 4.5.

[0230] Step 4.4: Based on the projection position corresponding to the global average position of the 3D small target, determine the position and orientation of the 2D target frame of the 2D small target corresponding to the 3D small target in the video frame to achieve global position unification.

[0231] That is to say, in order to solve the problem of sparse point clouds of stationary obstacles (such as small 3D targets) at long viewing distances (such as more than 100 meters) and missed detection of 2D small targets caused by low image pixel resolution, in this example, global back-projection processing is performed on the stationary obstacles (such as small 3D targets). For example, the global average position is converted to the camera coordinate system corresponding to the video frame to project the position of the 2D small target corresponding to the 3D small target in the video frame. In this way, the recall of stationary obstacles at long viewing distances can be effectively achieved.

[0232] Step 4.5: Determine whether the projection position corresponding to the global average position of the 3D small target is within the preset range in the video frame. If so, determine the projection position as the position of the 2D small target in the video frame to achieve the recall of the 2D small target. In this way, the recall of stationary obstacles at long viewing distances can be effectively achieved.

[0233] Module 5: False detection filtering algorithm module (also known as the false detection filtering algorithm module for single frame and trajectory fusion)

[0234] The purpose of the false detection filtering algorithm module is to filter out false detections. Here, it should be noted that in order to avoid false detections and false recalls, in actual applications, non-maximum suppression (NMS) can also be used to remove overlapping frames, and then the false detection filtering algorithm module can be used to filter false detections, thereby filtering out falsely recalled obstacles and obtaining stable, ultra-long-distance static obstacle labeling results.

[0235] Specifically, Figure 7 As shown, the main steps include:

[0236] Step 5.1: Calculate the target confidence of each track information obtained in step 2.1 above. For example, in one example, the target confidence can be mainly composed of the following dimensions: the confidence of the 2D target box of the 2D small target, the feature value of the similarity between the associated 3D small target and the 2D small target, and the occlusion ratio. For example, in one example, the formula of the target confidence of the track information (which can be recorded as track_similarity) can be specifically as follows:

[0237]

[0238] in,

[0239] Here, N represents the number of small 3D objects with the same trajectory information; score i,camera Represents the confidence of the 2D target box of the i-th 3D small target in the trajectory information under the perspective of the image acquisition component (camara perspective), similarity i,cameraThe feature value representing the similarity between the 3D target frame of the i-th 3D small target in the trajectory information and its associated 2D target frame. i,camera Indicates the occlusion rate of the i-th small 3D target in the trajectory information.

[0240] Step 5.2: Determine whether the target confidence track_similarity of the track information is greater than or equal to the first preset threshold. If so, execute step 5.3; otherwise, execute step 5.4.

[0241] Step 5.3: Save the trajectory information.

[0242] It can be understood that if the target confidence track_similarity of the track information is greater than a preset threshold, the track information is considered to be a valid track, and therefore, the track information is retained.

[0243] Step 5.4: Determine the track information whose target confidence track_similarity is less than the first preset threshold as a false detection track. Traverse the video frames corresponding to the 3D small target in the false detection track, and calculate the single frame confidence of the 3D small target in the false detection track on the video frame corresponding to the 3D small target.

[0244] Here, the calculation method of the single frame confidence is similar to the calculation method of the above target confidence, which will not be repeated here.

[0245] Step 5.5: Determine whether the single-frame confidence of the 3D small target in the false detection trajectory on the video frame corresponding to the 3D small target is greater than or equal to the second preset threshold, so that it can effectively avoid being mistakenly filtered out; if so, execute step 5.5; otherwise, execute step 5.6.

[0246] Step 5.5: Keep the small 3D target in the misdetected trajectory.

[0247] Step 5.6: Delete the small 3D target in the false detection trajectory, update the false detection trajectory, and finally output the labeling result, for example, output the labeling result for the small 3D target.

[0248] In this way, the disclosed solution realizes the automatic labeling of various static obstacles such as construction signs and crash barrels in the bird's eye view (BEV) image without relying on 3D point cloud detection. At the same time, experiments have proved that the disclosed solution can also increase the labeling sight distance of static obstacles in construction scenes such as cone barrels, crash barrels, construction signs, and triangle signs to 200 meters. In this way, using the labeled data of the disclosed solution to drive the vehicle-side BEV obstacle detection model can greatly improve the obstacle detection sight distance, thereby providing strong support for the subsequent timely obstacle avoidance of the main vehicle.

[0249] The disclosed solution also provides an obstacle marking device, such as Figure 8 As shown, including:

[0250] The result determination unit 801 is used to obtain a 2D detection result, wherein the 2D detection result is obtained based on a plurality of video frames collected by the target vehicle, and at least includes 2D feature information of a 2D target obstacle in a target environment where the target vehicle is located and a preset label of the 2D target obstacle; and obtain a 3D detection result, wherein the 3D detection result is obtained based on a plurality of frames of point cloud data collected by the target vehicle, and includes 3D feature information of a 3D target obstacle in the target environment;

[0251] The labeling unit 802 is used to obtain a target label of each 3D target obstacle based on the 2D feature information included in the 2D detection result and the preset label of the 2D target obstacle, and the 3D feature information included in the 3D detection result.

[0252] In a specific example of the disclosed solution, the result determination unit is specifically used to:

[0253] Based on the 2D detection result, determining a target point corresponding to the 2D target obstacle in the 2D detection result from the multi-frame point cloud data collected by the target vehicle;

[0254] Based on all the determined target points, a target global map of the target environment where the target vehicle is located is obtained;

[0255] Determine the 3D target obstacles in the target global map to obtain the 3D detection result.

[0256] In a specific example of the disclosed solution, the result determination unit is specifically used to:

[0257] Projecting points in each frame of point cloud data in the multiple frames of point cloud data into each video frame of the multiple video frames to obtain initial projection points of each point in each frame of point cloud data;

[0258] The point in each frame of point cloud data where the initial projection point falls within the range of the 2D target obstacle in the video frame is taken as the target point.

[0259] In a specific example of the disclosed solution, the result determination unit is specifically used to:

[0260] Determine the video frame corresponding to each frame of point cloud data;

[0261] The points in each frame of point cloud data are projected onto the video frame corresponding to each frame of point cloud data to obtain the initial projection points of each point in each frame of point cloud data.

[0262] In a specific example of the disclosed solution, the result determination unit is further used to:

[0263] Determining static obstacles from all detected 2D obstacles in the target environment where the target vehicle is located;

[0264] From all the determined static obstacles, static obstacles whose preset labels meet the label requirements are selected to determine the 2D target obstacle.

[0265] In a specific example of the disclosed solution, the result determination unit is specifically used to:

[0266] Projecting each target point in the target global map into a video frame of the 2D target obstacle to obtain a target projection point of each target point;

[0267] Based on the target projection points falling within the range of the 2D target obstacle, a target point group of the 3D target obstacle in the target global map is determined to obtain the 3D detection result.

[0268] In a specific example of the disclosed solution, the result determination unit is specifically used to:

[0269] The target global map is converted into a target coordinate system corresponding to the video frame, and each target point in the target global map after the coordinate system conversion is projected into the video frame in the 2D target obstacle to obtain a target projection point of each target point.

[0270] In a specific example of the disclosed solution, the result determination unit is specifically used to:

[0271] The target projection points falling within the range of the 2D target obstacle are used as the initial point group of the 3D target obstacle corresponding to the 2D target obstacle;

[0272] The initial point group of the 3D target obstacle corresponding to the 2D target obstacle is subjected to at least one of noise filtering and preset point cloud computing to determine the target point group of the 3D target obstacle in the target global map and obtain the 3D detection result.

[0273] In a specific example of the disclosed solution, the marking unit is further used to:

[0274] Obtaining target association information based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, wherein the target association information at least represents a mapping relationship between the 2D target obstacle and the 3D target obstacle;

[0275] When it is determined based on the target association information that there is a 2D target obstacle associated with the 3D target obstacle, a preset label of the 2D target obstacle is assigned to the 3D target obstacle associated with the 2D target obstacle to obtain target labels of at least part of the 3D target obstacles included in the 3D detection result.

[0276] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0277] Based on the 2D feature information of the 2D target obstacle included in the 2D detection result and the 3D feature information of the 3D target obstacle included in the 3D detection result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained;

[0278] The target association information is obtained based on a feature value representing a similarity between the 2D target obstacle and the 3D target obstacle.

[0279] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0280] Based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, at least one of the following is obtained: intersection over union ratio, shape similarity, area difference value, and effective distance;

[0281] Based on at least one of the calculated intersection-over-union ratio, the shape similarity, and the area difference value, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

[0282] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0283] Performing weighted processing on at least two of the intersection-over-union ratio, the shape similarity, the area difference value, and the effective distance to obtain a weighted result;

[0284] Based on the weighted result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

[0285] In a specific example of the disclosed solution, the marking unit is further used to:

[0286] Performing frame-by-frame tracking and association on the 3D target obstacles included in the target global map to obtain trajectory information of each 3D target obstacle included in the target global map;

[0287] The trajectory information of the 3D target obstacle includes at least one of the following: the position of the 3D target obstacle in the target frame point cloud data, where the target frame point cloud data is any frame in the multi-frame point cloud data.

[0288] In a specific example of the disclosed solution, the marking unit is further used to:

[0289] In a case where a 2D target obstacle associated with the 3D target obstacle is not determined based on the target association information, a label prediction is performed on the 3D target obstacle not associated with the 2D target obstacle based at least on the trajectory information of the 3D target obstacle to obtain a prediction result of the 3D target obstacle not associated with the 2D target obstacle, wherein the prediction result includes a prediction label and a probability value of the prediction label;

[0290] Based on the predicted labels whose probability values ​​are greater than or equal to the target values, the target labels of the 3D target obstacles that are not associated with the 2D target obstacles are obtained.

[0291] In a specific example of the disclosed solution, the marking unit is further used to:

[0292] Based on the label consistency principle of the same 3D target obstacle in the trajectory information, verify the target label of the 3D target obstacle in the 3D inspection result to obtain a label verification result;

[0293] The target label of the 3D target obstacle in the 3D inspection result is adjusted based on the label verification result, so that the target label of the same 3D target obstacle in the trajectory information is consistent.

[0294] In a specific example of the disclosed solution, the marking unit is further used to:

[0295] Based on the position information of the same 3D target obstacle in the trajectory information of the 3D target obstacle in different target frame point cloud data, the global position information of the 3D target obstacle is obtained;

[0296] The global position information of the 3D target obstacle on the target frame point cloud data is used to at least adjust the 3D feature information of the 3D target obstacle in the 3D detection result.

[0297] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0298] The 3D target obstacle on each target frame point cloud data in the trajectory information of the 3D target obstacle is converted into a preset coordinate system to obtain the position information of the 3D target obstacle on the target frame point cloud data in the preset coordinate system;

[0299] Based on the position information of the same 3D target obstacle in a preset coordinate system, the global position information of the 3D target obstacle on the target frame point cloud data is obtained.

[0300] In a specific example of the disclosed solution, the marking unit is further used to:

[0301] Obtaining a projection position of global position information of a 3D target obstacle on each of the multiple video frames;

[0302] Based on the projection position of the global position information of the 3D target obstacle on each video frame, it is determined whether the video frame has a 2D target obstacle corresponding to the 3D target obstacle, so as to adjust the 2D detection result.

[0303] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0304] Converting the global position information of the 3D target obstacle to the coordinate system of the image acquisition device that acquires the video frame to obtain the global position information of the 3D target obstacle in the coordinate system of the image acquisition device;

[0305] The global position information of the 3D target obstacle in the coordinate system of the image acquisition device is projected onto the video frame to obtain the projection position of the global position information of the 3D target obstacle on the video frame.

[0306] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0307] When it is determined that the projection position of the global position information of the 3D target obstacle on the video frame falls within a preset range in the video frame, determining whether there is a 2D target obstacle corresponding to the 3D target obstacle whose projection position falls within the preset range in the 2D inspection result;

[0308] If it is determined that the obstacle does not exist, the 2D target obstacle corresponding to the 3D target obstacle whose projection position in the video frame falls within the preset range is recalled, and the 2D detection result is updated based on the recalled 2D target obstacle.

[0309] In a specific example of the disclosed solution, the marking unit is further used to:

[0310] Obtain target confidence of trajectory information of each 3D target obstacle;

[0311] Based on the target confidence of the trajectory information, the trajectory information is verified to obtain a trajectory verification result, so as to adjust the obtained trajectory information of each 3D target obstacle based on the trajectory verification result.

[0312] In a specific example of the disclosed solution, the marking unit is specifically used to:

[0313] When the target confidence of the trajectory information is greater than or equal to a first preset threshold, retaining the trajectory whose target confidence is greater than or equal to the first preset threshold;

[0314] or,

[0315] When the target confidence of the trajectory information is less than the first preset threshold, the trajectory information with the target confidence less than the first preset threshold is determined as a false detection trajectory.

[0316] In a specific example of the disclosed solution, the marking unit is further used to:

[0317] Determine each video frame corresponding to the 3D target obstacle in the false detection trajectory;

[0318] The single-frame confidence of each video frame corresponding to the 3D target obstacle in the false detection trajectory is obtained; the single-frame confidence represents the confidence that there is a 2D target obstacle corresponding to the 3D target obstacle in the video frame;

[0319] Based on the single frame confidence of each video frame corresponding to the 3D target obstacle, determine whether to delete the misdetected trajectory.

[0320] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0321] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0322] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0323] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0324] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0325] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0326] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the marking method of obstacles. For example, in some embodiments, the marking method of obstacles may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the marking method of obstacles described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the marking method of obstacles in any other appropriate manner (e.g., by means of firmware).

[0327] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0328] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0329] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0330] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0331] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0332] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0333] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0334] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for marking obstacles, comprising: Obtaining a 2D detection result, wherein the 2D detection result is obtained based on a plurality of video frames collected by the target vehicle, and at least includes 2D feature information of a 2D target obstacle in a target environment where the target vehicle is located and a preset label of the 2D target obstacle; Obtaining a 3D detection result, wherein the 3D detection result is obtained based on multiple frames of point cloud data collected by the target vehicle, and includes 3D feature information of the 3D target obstacle in the target environment; Based on the 2D feature information and preset labels of the 2D target obstacles included in the 2D detection results, and the 3D feature information included in the 3D detection results, a target label of each 3D target obstacle is obtained.

2. The method according to claim 1, wherein: The obtaining of the 3D detection result includes: Based on the 2D detection result, determining a target point corresponding to the 2D target obstacle in the 2D detection result from the multi-frame point cloud data collected by the target vehicle; Based on all the determined target points, a target global map of the target environment where the target vehicle is located is obtained; Determine the 3D target obstacles in the target global map to obtain the 3D detection result.

3. The method according to claim 2, wherein: The determining, based on the 2D detection result, from the multi-frame point cloud data collected by the target vehicle, a target point corresponding to the 2D target obstacle in the 2D detection result includes: Projecting points in each frame of point cloud data in the multiple frames of point cloud data into each video frame of the multiple video frames to obtain initial projection points of each point in each frame of point cloud data; The point in each frame of point cloud data where the initial projection point falls within the range of the 2D target obstacle in the video frame is taken as the target point.

4. The method according to claim 3, wherein: The projecting of the points in each frame of the point cloud data in the multiple frames of the point cloud data into each video frame of the multiple video frames to obtain the initial projection points of each point in each frame of the point cloud data includes: Determine the video frame corresponding to each frame of point cloud data; The points in each frame of point cloud data are projected onto the video frame corresponding to each frame of point cloud data to obtain the initial projection points of each point in each frame of point cloud data.

5. The method according to claim 3, further comprising: Determining static obstacles from all detected 2D obstacles in the target environment where the target vehicle is located; From all the determined static obstacles, static obstacles whose preset tags meet the tag requirements are selected to determine the 2D target obstacle.

6. The method according to claim 2, wherein: The determining of the 3D target obstacle in the target global map to obtain the 3D detection result includes: Projecting each target point in the target global map into a video frame of the 2D target obstacle to obtain a target projection point of each target point; Based on the target projection points falling within the range of the 2D target obstacle, a target point group of the 3D target obstacle in the target global map is determined to obtain the 3D detection result.

7. The method according to claim 6, wherein: The projecting each target point in the target global map into a video frame in the 2D target obstacle to obtain a target projection point of each target point includes: The target global map is converted into a target coordinate system corresponding to the video frame, and each target point in the target global map after the coordinate system conversion is projected into the video frame in the 2D target obstacle to obtain a target projection point of each target point.

8. The method according to claim 6, wherein: The step of determining a target point group of a 3D target obstacle in the target global map based on the target projection point falling within the range of the 2D target obstacle to obtain the 3D detection result includes: The target projection points falling within the range of the 2D target obstacle are used as the initial point group of the 3D target obstacle corresponding to the 2D target obstacle; The initial point group of the 3D target obstacle corresponding to the 2D target obstacle is subjected to at least one of noise filtering and preset point cloud computing to determine the target point group of the 3D target obstacle in the target global map and obtain the 3D detection result.

9. The method according to claim 1, further comprising: Obtaining target association information based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, wherein the target association information at least represents a mapping relationship between the 2D target obstacle and the 3D target obstacle; The step of obtaining the target label of each 3D target obstacle based on the 2D feature information and the preset label of the 2D target obstacle contained in the 2D detection result, and the 3D feature information contained in the 3D detection result includes: When it is determined based on the target association information that there is a 2D target obstacle associated with the 3D target obstacle, a preset label of the 2D target obstacle is assigned to the 3D target obstacle associated with the 2D target obstacle to obtain target labels of at least part of the 3D target obstacles included in the 3D detection result.

10. The method according to claim 9, wherein: The obtaining target association information based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result includes: Based on the 2D feature information of the 2D target obstacle included in the 2D detection result and the 3D feature information of the 3D target obstacle included in the 3D detection result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained; The target association information is obtained based on a feature value representing a similarity between the 2D target obstacle and the 3D target obstacle.

11. The method according to claim 10, wherein: The obtaining, based on the 2D feature information of the 2D target obstacle included in the 2D detection result and the 3D feature information of the 3D target obstacle included in the 3D detection result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle includes: Based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, at least one of the following is obtained: intersection over union ratio, shape similarity, area difference value, and effective distance; Based on at least one of the calculated intersection-over-union ratio, the shape similarity, and the area difference value, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

12. The method according to claim 11, wherein: The step of obtaining a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle based on at least one of the calculated intersection-over-union ratio, shape similarity, and area difference value includes: Performing weighted processing on at least two of the intersection-over-union ratio, the shape similarity, the area difference value, and the effective distance to obtain a weighted result; Based on the weighted result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

13. The method according to any one of claims 9 to 12, further comprising: Performing frame-by-frame tracking and association on the 3D target obstacles included in the target global map to obtain trajectory information of each 3D target obstacle included in the target global map; The trajectory information of the 3D target obstacle includes at least one of the following: the position of the 3D target obstacle in the target frame point cloud data, where the target frame point cloud data is any frame in the multi-frame point cloud data.

14. The method according to claim 13, further comprising: In a case where a 2D target obstacle associated with the 3D target obstacle is not determined based on the target association information, a label prediction is performed on the 3D target obstacle not associated with the 2D target obstacle based at least on the trajectory information of the 3D target obstacle to obtain a prediction result of the 3D target obstacle not associated with the 2D target obstacle, wherein the prediction result includes a prediction label and a probability value of the prediction label; Based on the predicted labels whose probability values ​​are greater than or equal to the target values, the target labels of the 3D target obstacles that are not associated with the 2D target obstacles are obtained.

15. The method according to claim 14, further comprising: Based on the label consistency principle of the same 3D target obstacle in the trajectory information, verify the target label of the 3D target obstacle in the 3D inspection result to obtain a label verification result; The target label of the 3D target obstacle in the 3D inspection result is adjusted based on the label verification result, so that the target label of the same 3D target obstacle in the trajectory information is consistent.

16. The method according to any one of claims 13 to 15, further comprising: Based on the position information of the same 3D target obstacle in the trajectory information of the 3D target obstacle in different target frame point cloud data, the global position information of the 3D target obstacle is obtained; The global position information of the 3D target obstacle on the target frame point cloud data is used to at least adjust the 3D feature information of the 3D target obstacle in the 3D detection result.

17. The method according to claim 16, wherein: The obtaining of the global position information of the 3D target obstacle based on the position information of the same 3D target obstacle in different target frame point cloud data in the trajectory information of the 3D target obstacle includes: The 3D target obstacle on each target frame point cloud data in the trajectory information of the 3D target obstacle is converted into a preset coordinate system to obtain the position information of the 3D target obstacle on the target frame point cloud data in the preset coordinate system; Based on the position information of the same 3D target obstacle in a preset coordinate system, the global position information of the 3D target obstacle on the target frame point cloud data is obtained.

18. The method according to claim 16, further comprising: Obtaining a projection position of global position information of a 3D target obstacle on each of the multiple video frames; Based on the projection position of the global position information of the 3D target obstacle on each video frame, it is determined whether the video frame has a 2D target obstacle corresponding to the 3D target obstacle, so as to adjust the 2D detection result.

19. The method according to claim 18, wherein: The obtaining of the projection position of the global position information of the 3D target obstacle on each of the multiple video frames includes: Converting the global position information of the 3D target obstacle to the coordinate system of the image acquisition device that acquires the video frame to obtain the global position information of the 3D target obstacle in the coordinate system of the image acquisition device; The global position information of the 3D target obstacle in the coordinate system of the image acquisition device is projected onto the video frame to obtain the projection position of the global position information of the 3D target obstacle on the video frame.

20. The method according to claim 18, wherein: The determining whether there is a 2D target obstacle corresponding to the 3D target obstacle in the video frame based on the projection position of the global position information of the 3D target obstacle on each video frame to adjust the 2D detection result includes: When it is determined that the projection position of the global position information of the 3D target obstacle on the video frame falls within a preset range in the video frame, determining whether there is a 2D target obstacle corresponding to the 3D target obstacle whose projection position falls within the preset range in the 2D inspection result; If it is determined that the obstacle does not exist, the 2D target obstacle corresponding to the 3D target obstacle whose projection position in the video frame falls within the preset range is recalled, and the 2D detection result is updated based on the recalled 2D target obstacle.

21. The method according to any one of claims 13 to 20, further comprising: Obtain target confidence of trajectory information of each 3D target obstacle; Based on the target confidence of the trajectory information, the trajectory information is verified to obtain a trajectory verification result, so as to adjust the obtained trajectory information of each 3D target obstacle based on the trajectory verification result.

22. The method according to claim 21, wherein: The target confidence level based on the trajectory information verifies the trajectory information to obtain a trajectory verification result, and adjusts the obtained trajectory information of each 3D target obstacle based on the trajectory verification result, including: When the target confidence of the trajectory information is greater than or equal to a first preset threshold, retaining the trajectory whose target confidence is greater than or equal to the first preset threshold; or, When the target confidence of the trajectory information is less than the first preset threshold, the trajectory information with the target confidence less than the first preset threshold is determined as a false detection trajectory.

23. The method according to claim 21, further comprising: Determine each video frame corresponding to the 3D target obstacle in the false detection trajectory; The single-frame confidence of each video frame corresponding to the 3D target obstacle in the false detection trajectory is obtained; the single-frame confidence represents the confidence that there is a 2D target obstacle corresponding to the 3D target obstacle in the video frame; Based on the single frame confidence of each video frame corresponding to the 3D target obstacle, determine whether to delete the misdetected trajectory.

24. An obstacle marking device, comprising: A result determination unit is used to obtain a 2D detection result, wherein the 2D detection result is obtained based on a plurality of video frames collected by the target vehicle, and at least includes 2D feature information of a 2D target obstacle in a target environment where the target vehicle is located and a preset label of the 2D target obstacle; and obtain a 3D detection result, wherein the 3D detection result is obtained based on a plurality of frames of point cloud data collected by the target vehicle, and includes 3D feature information of a 3D target obstacle in the target environment; The labeling unit is used to obtain a target label of each 3D target obstacle based on the 2D feature information and the preset label of the 2D target obstacle included in the 2D detection result, and the 3D feature information included in the 3D detection result.

25. The device according to claim 24, wherein: The result determination unit is specifically used for: Based on the 2D detection result, determining a target point corresponding to the 2D target obstacle in the 2D detection result from the multi-frame point cloud data collected by the target vehicle; Based on all the determined target points, a target global map of the target environment where the target vehicle is located is obtained; Determine the 3D target obstacles in the target global map to obtain the 3D detection result.

26. The device according to claim 25, wherein The result determination unit is specifically used for: Projecting points in each frame of point cloud data in the multiple frames of point cloud data into each video frame of the multiple video frames to obtain initial projection points of each point in each frame of point cloud data; The point in each frame of point cloud data where the initial projection point falls within the range of the 2D target obstacle in the video frame is taken as the target point.

27. The device according to claim 26, wherein: The result determination unit is specifically used for: Determine the video frame corresponding to each frame of point cloud data; The points in each frame of point cloud data are projected onto the video frame corresponding to each frame of point cloud data to obtain the initial projection points of each point in each frame of point cloud data.

28. The device according to claim 26, wherein The result determination unit is further used for: Determining static obstacles from all detected 2D obstacles in the target environment where the target vehicle is located; From all the determined static obstacles, static obstacles whose preset tags meet the tag requirements are selected to determine the 2D target obstacle.

29. The device according to claim 25, wherein: The result determination unit is specifically used for: Projecting each target point in the target global map into a video frame of the 2D target obstacle to obtain a target projection point of each target point; Based on the target projection points falling within the range of the 2D target obstacle, a target point group of the 3D target obstacle in the target global map is determined to obtain the 3D detection result.

30. The device according to claim 29, wherein: The result determination unit is specifically used for: The target global map is converted into a target coordinate system corresponding to the video frame, and each target point in the target global map after the coordinate system conversion is projected into the video frame in the 2D target obstacle to obtain a target projection point of each target point.

31. The device according to claim 29, wherein: The result determination unit is specifically used for: The target projection points falling within the range of the 2D target obstacle are used as the initial point group of the 3D target obstacle corresponding to the 2D target obstacle; The initial point group of the 3D target obstacle corresponding to the 2D target obstacle is subjected to at least one of noise filtering and preset point cloud computing to determine the target point group of the 3D target obstacle in the target global map and obtain the 3D detection result.

32. The apparatus of claim 24, wherein: The marking unit is further used for: Obtaining target association information based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, wherein the target association information at least represents a mapping relationship between the 2D target obstacle and the 3D target obstacle; When it is determined based on the target association information that there is a 2D target obstacle associated with the 3D target obstacle, a preset label of the 2D target obstacle is assigned to the 3D target obstacle associated with the 2D target obstacle to obtain target labels of at least part of the 3D target obstacles included in the 3D detection result.

33. The device according to claim 32, wherein: The marking unit is specifically used for: Based on the 2D feature information of the 2D target obstacle included in the 2D detection result and the 3D feature information of the 3D target obstacle included in the 3D detection result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained; The target association information is obtained based on a feature value representing a similarity between the 2D target obstacle and the 3D target obstacle.

34. The device according to claim 33, wherein The marking unit is specifically used for: Based on the 2D feature information of the 2D target obstacle contained in the 2D detection result and the 3D feature information of the 3D target obstacle contained in the 3D detection result, at least one of the following is obtained: intersection over union ratio, shape similarity, area difference value, and effective distance; Based on at least one of the calculated intersection-over-union ratio, the shape similarity, and the area difference value, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

35. The device according to claim 34, wherein The marking unit is specifically used for: Performing weighted processing on at least two of the intersection-over-union ratio, the shape similarity, the area difference value, and the effective distance to obtain a weighted result; Based on the weighted result, a feature value representing the similarity between the 2D target obstacle and the 3D target obstacle is obtained.

36. The device according to any one of claims 32 to 35, wherein: The marking unit is further used for: Performing frame-by-frame tracking and association on the 3D target obstacles included in the target global map to obtain trajectory information of each 3D target obstacle included in the target global map; The trajectory information of the 3D target obstacle includes at least one of the following: the position of the 3D target obstacle in the target frame point cloud data, where the target frame point cloud data is any frame in the multi-frame point cloud data.

37. The device according to claim 36, wherein The marking unit is further used for: In a case where a 2D target obstacle associated with the 3D target obstacle is not determined based on the target association information, a label prediction is performed on the 3D target obstacle not associated with the 2D target obstacle based at least on the trajectory information of the 3D target obstacle to obtain a prediction result of the 3D target obstacle not associated with the 2D target obstacle, wherein the prediction result includes a prediction label and a probability value of the prediction label; Based on the predicted labels whose probability values ​​are greater than or equal to the target values, the target labels of the 3D target obstacles that are not associated with the 2D target obstacles are obtained.

38. The device according to claim 37, wherein The marking unit is further used for: Based on the label consistency principle of the same 3D target obstacle in the trajectory information, verify the target label of the 3D target obstacle in the 3D inspection result to obtain a label verification result; The target label of the 3D target obstacle in the 3D inspection result is adjusted based on the label verification result, so that the target label of the same 3D target obstacle in the trajectory information is consistent.

39. The device according to any one of claims 36 to 38, wherein: The marking unit is further used for: Based on the position information of the same 3D target obstacle in the trajectory information of the 3D target obstacle in different target frame point cloud data, the global position information of the 3D target obstacle is obtained; The global position information of the 3D target obstacle on the target frame point cloud data is used to at least adjust the 3D feature information of the 3D target obstacle in the 3D detection result.

40. The device according to claim 39, wherein The marking unit is specifically used for: The 3D target obstacle on each target frame point cloud data in the trajectory information of the 3D target obstacle is converted into a preset coordinate system to obtain the position information of the 3D target obstacle on the target frame point cloud data in the preset coordinate system; Based on the position information of the same 3D target obstacle in a preset coordinate system, the global position information of the 3D target obstacle on the target frame point cloud data is obtained.

41. The apparatus of claim 39, wherein: The marking unit is further used for: Obtaining a projection position of global position information of a 3D target obstacle on each of the multiple video frames; Based on the projection position of the global position information of the 3D target obstacle on each video frame, it is determined whether the video frame has a 2D target obstacle corresponding to the 3D target obstacle, so as to adjust the 2D detection result.

42. The device according to claim 41, wherein The marking unit is specifically used for: Converting the global position information of the 3D target obstacle to the coordinate system of the image acquisition device that acquires the video frame to obtain the global position information of the 3D target obstacle in the coordinate system of the image acquisition device; The global position information of the 3D target obstacle in the coordinate system of the image acquisition device is projected onto the video frame to obtain the projection position of the global position information of the 3D target obstacle on the video frame.

43. The device according to claim 41, wherein The marking unit is specifically used for: When it is determined that the projection position of the global position information of the 3D target obstacle on the video frame falls within a preset range in the video frame, determining whether there is a 2D target obstacle corresponding to the 3D target obstacle whose projection position falls within the preset range in the 2D inspection result; If it is determined that the obstacle does not exist, the 2D target obstacle corresponding to the 3D target obstacle whose projection position in the video frame falls within the preset range is recalled, and the 2D detection result is updated based on the recalled 2D target obstacle.

44. The device according to any one of claims 36 to 43, wherein: The marking unit is further used for: Obtain target confidence of trajectory information of each 3D target obstacle; Based on the target confidence of the trajectory information, the trajectory information is verified to obtain a trajectory verification result, so as to adjust the obtained trajectory information of each 3D target obstacle based on the trajectory verification result.

45. The device according to claim 44, wherein The marking unit is specifically used for: When the target confidence of the trajectory information is greater than or equal to a first preset threshold, retaining the trajectory whose target confidence is greater than or equal to the first preset threshold; or, When the target confidence of the trajectory information is less than the first preset threshold, the trajectory information with the target confidence less than the first preset threshold is determined as a false detection trajectory.

46. ​​The device according to claim 44, wherein The marking unit is further used for: Determine each video frame corresponding to the 3D target obstacle in the false detection trajectory; The single-frame confidence of each video frame corresponding to the 3D target obstacle in the false detection trajectory is obtained; the single-frame confidence represents the confidence that there is a 2D target obstacle corresponding to the 3D target obstacle in the video frame; Based on the single frame confidence of each video frame corresponding to the 3D target obstacle, determine whether to delete the misdetected trajectory.

47. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 23.

48. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-23.

49. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 23.

Images