Background obstacle association method and device, vehicle machine, medium and vehicle
By constructing a global semantic map and calculating the interchange score, the problems of low accuracy and poor stability of autonomous vehicles when detecting background obstacles are solved, and high-precision tracking association of background obstacles is achieved.
Patent Information
- Application Number
- CN202311245192.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-03-25
AI Technical Summary
When detecting background obstacles, existing autonomous vehicles have low accuracy and poor stability, mainly because adjacent background obstacles in physical space are not necessarily the same obstacle.
By constructing a global semantic map, the global semantic raster of background obstacles in the current and historical frames is obtained, and the interleaving scores are calculated to determine the correlation of obstacles. When the interleaving score is greater than or equal to the threshold, the background obstacle in the current frame is associated with the background obstacle in the historical frame.
Improve the detection accuracy and stability of background obstacles, ensuring the correct tracking and correlation of the same obstacle.
Smart Images
Figure CN119693909A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of assisted driving, and in particular to a method, device, vehicle computer, medium and vehicle for associating background obstacles. Background Art
[0002] Unmanned driving technology relies on the detection of surrounding obstacles by autonomous vehicles. In existing detection technologies, multi-frame point cloud data is usually required to detect obstacles, including foreground obstacles and background obstacles. Background obstacles are generally immovable obstacles, and their positions in physical space are generally stable. Currently, the nearest neighbor algorithm is mainly used to calculate the translation parameters and rotation parameters of background obstacles between adjacent frames, and determine the background obstacles with similar distances based on the translation parameters and rotation parameters, so as to achieve tracking and association of background obstacles. However, the adjacent background obstacles in physical space are not necessarily the same background obstacles, which leads to low detection accuracy and poor stability of background obstacles. Summary of the invention
[0003] In order to solve the above technical problem or at least partially solve the above technical problem, the present disclosure provides a background obstacle association method, which can improve the detection accuracy and stability of background obstacles.
[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:
[0005] In a first aspect, the present disclosure provides a method for associating background obstacles, comprising:
[0006] Building a global semantic map based on odometer information, wherein the global semantic map includes background obstacles;
[0007] Based on the global semantic map, obtaining a first global semantic grid of background obstacles in a current frame and a second global semantic grid of background obstacles in a historical frame;
[0008] Calculating an intersection-and-union score between the second global semantic grid and the first global semantic grid according to a preset rule, wherein the intersection-and-union score is used to indicate a degree of overlap between the second global semantic grid and the first global semantic grid;
[0009] When the intersection-over-combination ratio value is greater than or equal to a threshold, the background obstacle in the current frame is associated with the background obstacle in the historical frame.
[0010] As an optional implementation provided by an embodiment of the present disclosure, calculating the intersection-and-union ratio of the second global semantic grid and the first global semantic grid according to a preset rule includes:
[0011] Calculating a first score of a preset element in the first global semantic grid, and calculating a second score of a preset element in the second global semantic grid, wherein the preset elements include a laser point, a neighboring point, a minimum enclosing polygon, a bounding box, and a neighboring point of the bounding box corresponding to a background obstacle;
[0012] determining an intersection grid of the first global semantic grid and the second global semantic grid;
[0013] Calculating a third score of the intersection of the preset elements in the intersecting grids;
[0014] Substitute the first score, the second score, and the third score into the intersection-and-union ratio formula to calculate the intersection-and-union ratio score.
[0015] As an optional implementation provided by an embodiment of the present disclosure, calculating a first score of a preset element in the first global semantic grid, and calculating a second score of a preset element in the second global semantic grid, include:
[0016] Determine the score of the preset laser point, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box;
[0017] The first score is obtained by adding the score of the laser point corresponding to the first global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box;
[0018] The second score is obtained by adding the score of the laser point corresponding to the second global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box.
[0019] As an optional implementation provided by an embodiment of the present disclosure, calculating a third score of the intersection of the preset elements in the intersecting grids includes:
[0020] Determine the score of the intersection of a preset laser point with a preset element, the score of the intersection of an adjacent point with the preset element, the score of the intersection of a minimum enclosing polygon with the preset element, the score of the intersection of a bounding box with the preset element, and the score of the intersection of an adjacent point of the bounding box with the preset element;
[0021] The third score is obtained by adding the score of the intersection between the laser point corresponding to the intersecting grid and the preset element, the score of the intersection between the adjacent point and the preset element, the score of the intersection between the minimum enclosing polygon and the preset element, the score of the intersection between the bounding box and the preset element, and the score of the intersection between the adjacent point of the bounding box and the preset element.
[0022] As an optional implementation provided by the embodiment of the present disclosure, a global semantic map is constructed according to odometer information. Before that, the method further includes:
[0023] Acquire a point cloud sequence collected by a sensor, wherein the sensor includes an image sensor and / or a laser sensor, the point cloud sequence is a set of point data on the appearance surface of the measured object collected by the sensor, and the point cloud sequence includes point cloud information of background obstacles;
[0024] Performing semantic segmentation on the point cloud sequence to obtain semantic features, wherein the semantic features include background obstacle features;
[0025] The step of constructing a global semantic map based on odometer information includes:
[0026] Obtain odometer information and determine the posture information corresponding to the odometer information;
[0027] The semantic features are projected into a global coordinate system based on the posture information to obtain the global semantic map.
[0028] As an optional implementation provided by an embodiment of the present disclosure, after projecting the semantic features to a global coordinate system based on the posture information to obtain the global semantic map, the method further includes:
[0029] The background obstacles in the current frame and the historical frames of the global semantic map are annotated, and the annotated elements include laser points, neighboring points, minimum enclosing polygons, bounding boxes, and neighboring points of the bounding boxes corresponding to the background obstacles.
[0030] As an optional implementation provided by an embodiment of the present disclosure, based on the global semantic map, obtaining a first global semantic grid of background obstacles in a current frame and a second global semantic grid of background obstacles in a historical frame includes:
[0031] rasterizing the current frame according to the background obstacle features included in the current frame in the global semantic map to obtain a first global semantic grid corresponding to the background obstacle in the current frame;
[0032] According to the background obstacle features included in the historical frames in the global semantic map, the historical frames are rasterized to obtain a second global semantic grid corresponding to the background obstacles in the historical frames.
[0033] In a second aspect, the present disclosure provides a background obstacle association device, comprising:
[0034] A map construction module, used to: construct a global semantic map according to odometer information, wherein the global semantic map includes background obstacles;
[0035] A processing module, configured to: obtain, based on the global semantic map, a first global semantic grid of background obstacles in a current frame and a second global semantic grid of background obstacles in a historical frame;
[0036] A calculation module, used to calculate the intersection-and-union ratio of the second global semantic grid and the first global semantic grid according to a preset rule;
[0037] The association module is used to: when the intersection-and-combination ratio value is greater than or equal to a threshold, associate the background obstacle in the current frame with the background obstacle in the historical frame.
[0038] As an optional implementation provided by an embodiment of the present disclosure, a calculation module is specifically used to: calculate a first score of a preset element in the first global semantic grid, and calculate a second score of a preset element in the second global semantic grid, wherein the preset elements include a laser point, a neighboring point, a minimum enclosing polygon, a bounding box, and a neighboring point of the bounding box corresponding to a background obstacle;
[0039] determining an intersection grid of the first global semantic grid and the second global semantic grid;
[0040] Calculating a third score of the intersection of the preset elements in the intersecting grids;
[0041] Substitute the first score, the second score, and the third score into the intersection-and-union ratio formula to calculate the intersection-and-union ratio score.
[0042] As an optional implementation provided by an embodiment of the present disclosure, the calculation module is specifically used to: determine a score of a preset laser point, a score of a neighboring point, a score of a minimum enclosing polygon, a score of a bounding box, and a score of a neighboring point of the bounding box;
[0043] The first score is obtained by adding the score of the laser point corresponding to the first global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box;
[0044] The second score is obtained by adding the score of the laser point corresponding to the second global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box.
[0045] As an optional implementation provided by an embodiment of the present disclosure, the calculation module is specifically used to: determine the score of the intersection between a preset laser point and a preset element, the score of the intersection between a neighboring point and the preset element, the score of the intersection between a minimum enclosing polygon and the preset element, the score of the intersection between a bounding box and the preset element, and the score of the intersection between a neighboring point of the bounding box and the preset element;
[0046] The third score is obtained by adding the score of the intersection between the laser point corresponding to the intersecting grid and the preset element, the score of the intersection between the adjacent point and the preset element, the score of the intersection between the minimum enclosing polygon and the preset element, the score of the intersection between the bounding box and the preset element, and the score of the intersection between the adjacent point of the bounding box and the preset element.
[0047] As an optional implementation provided by an embodiment of the present disclosure, the map construction module is further used to: before constructing the global semantic map according to the odometer information, obtain a point cloud sequence collected by a sensor, wherein the sensor includes an image sensor and a laser sensor, and the point cloud sequence is a set of point cloud data of the appearance surface of the measured object collected by the sensor, and the point cloud sequence includes point cloud information of background obstacles;
[0048] Performing semantic segmentation on the point cloud sequence to obtain semantic features, wherein the semantic features include background obstacle features;
[0049] The map construction module is specifically used to: obtain odometer information and determine the posture information corresponding to the odometer information;
[0050] The semantic features are projected into a global coordinate system based on the posture information to obtain the global semantic map.
[0051] As an optional implementation provided by an embodiment of the present disclosure, the map construction module is also used to: mark the background obstacles in the current frame and the historical frames of the global semantic map, and the marked elements include laser points, neighboring points, minimum enclosing polygons, bounding boxes and neighboring points of the bounding box corresponding to the background obstacles.
[0052] As an optional implementation provided by an embodiment of the present disclosure, the processing module is specifically used to: rasterize the current frame according to the background obstacle features included in the current frame in the global semantic map to obtain a first global semantic grid corresponding to the background obstacle in the current frame;
[0053] According to the background obstacle features included in the historical frames in the global semantic map, the historical frames are rasterized to obtain a second global semantic grid corresponding to the background obstacles in the historical frames.
[0054] In a third aspect, the present disclosure provides a vehicle computer, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for associating background obstacles as described in the first aspect or any one of its optional embodiments is implemented.
[0055] In a fourth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method for associating background obstacles as described in the first aspect or any one of its optional embodiments is implemented.
[0056] In a fifth aspect, a computer program product is provided, comprising: when the computer program product is run on a computer, the computer is enabled to implement the method for determining automotive field speech keywords as described in the first aspect or any one of its optional embodiments.
[0057] The existing assisted driving navigation route uses the traditional navigation system, which is not very accurate and difficult to match user habits.
[0058] The disclosed embodiment first constructs a global semantic map according to the odometer information, and the global semantic map includes background obstacles. Then, based on the global semantic map, the first global semantic grid of the background obstacles in the current frame and the second global semantic grid of the background obstacles in the historical frame are obtained to rasterize the background obstacles in the current frame and the historical frame, and then the intersection-and-union ratio of the second global semantic grid and the first global semantic grid is calculated according to the preset rule. The intersection-and-union ratio is used to indicate the overlap between the second global semantic grid and the first global semantic grid, and also describes the relative size of the overlap between the current obstacle and the background obstacle. The higher the overlap, the greater the possibility that the current background obstacle and the historical obstacle are the same obstacle; if the intersection-and-union ratio is greater than or equal to the threshold, it means that the current background obstacle is likely to be the same obstacle as the historical obstacle, and then the two are associated. The disclosed embodiment associates and tracks the background obstacles through the intersection-and-union ratio of the background obstacles in the current frame and the background obstacles in the historical frame on the global semantic map, thereby improving the accuracy and stability of background obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0061] Figure 1 is a flowchart of a method for associating background obstacles;
[0062] Figure 2A schematic diagram of the structure of a background obstacle association device provided in an embodiment of the present disclosure;
[0063] Figure 3 A structural schematic diagram of a vehicle computer is provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0064] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0066] The method for associating background obstacles provided in the embodiments of the present disclosure may be implemented by an associating device or electronic device of the background obstacles, wherein the electronic device may include but is not limited to: a car machine, a server, a personal computer, a laptop, a tablet computer, a smart phone, etc. Car machine. The operating system of the car machine may include Android (Android), a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiments of the present disclosure are not limited to this. The server includes but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud composed of a large number of computers or network servers in cloud computing, wherein cloud computing is a kind of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets. Among them, the electronic device can be operated alone to implement the present disclosure, or it can be connected to the network and implement the present disclosure through interactive operations with other electronic devices in the network. Among them, the network where the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN) network, etc.
[0067] It should be noted that the protection scope of the background obstacle association method described in the embodiment of the present disclosure is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.
[0068] like Figure 1 As shown, Figure 1 FIG. 1 is a flow chart of a method for associating background obstacles, the method comprising the following steps S101 to S104:
[0069] S101, constructing a global semantic map according to odometer information.
[0070] Among them, the odometer is a device for measuring distance and speed. In the embodiment of the present disclosure, the odometer can be a device installed on the vehicle to measure the driving distance. It can obtain data collected by mobile sensors to estimate the change of vehicle position over time. Common odometers include: lidar odometer, combined inertial navigation odometer, wheel odometer, visual odometer and visual inertial odometer, etc.
[0071] In some embodiments, the odometer information includes: laser radar odometer information and combined inertial navigation odometer information, wherein the laser radar odometer information is obtained from the laser point cloud data through a specific mileage calculation method, and can be displayed in the figure as a trajectory for estimating the posture transformation of the vehicle when it moves. Among them, the specific mileage calculation method can be an iterative closest point (Iterative Closest Point, ICP) algorithm, or a normal distribution change (Normal Distributions Transform, NDT) algorithm. Combined inertial navigation odometer information refers to the data such as the position, direction, height and linear speed of a moving object relative to a given starting point determined by the measurement value of an inertial measurement unit (Inertial Measurement Unit, IMU) sensor. All combined inertial navigation data are processed by the inertial navigation system and output as the combined inertial navigation odometer information, which can be displayed in the figure as a trajectory for estimating the posture information of the vehicle.
[0072] The global semantic map provides rich annotation information, such as obstacle type, vectorized representation of the road, type (lane, bike lane or bus lane), speed limit, lane line type (dashed line, solid line, double solid line), etc.
[0073] In some embodiments, before constructing a global semantic map based on odometer information, the following steps are also included: first, a point cloud sequence collected by a sensor is obtained, the sensor includes an image sensor and / or a laser sensor, the point cloud sequence is a point cloud data set of the surface of the measured object collected by the sensor, and the point cloud sequence includes point cloud information of background obstacles; then, semantic segmentation is performed on the point cloud sequence to extract semantic features, the semantic features include background obstacle features. Among them, the image sensor can be a fisheye camera, a depth camera, etc., and the laser sensor can be a lidar.
[0074] Furthermore, in the process of constructing a global semantic map based on the odometer information, the odometer information is first obtained to determine the pose information corresponding to the odometer information, and then the semantic features extracted from the point cloud sequence are projected to the global coordinate system based on the pose information to obtain the global semantic map. In this process, the background obstacle features are projected to the global coordinate system so that the global semantic map includes the background obstacles.
[0075] S102: Based on the global semantic map, obtain a first global semantic grid of background obstacles in the current frame and a second global semantic grid of background obstacles in the historical frame.
[0076] The current frame is the point cloud frame collected by the sensor at the current moment, and the historical frame is the point cloud frame collected by the sensor at the historical moment before the current moment. Optionally, the historical frame can be the point cloud frame collected by the sensor at the previous moment before the current moment.
[0077] In some embodiments, after executing step S101 and before executing step S102, the process also includes: marking background obstacles in the global semantic map, optionally marking background obstacles in the current frame of the global semantic map, and marking background obstacles in historical frames, and the marked elements include laser points (point), neighboring points (point_nearby), minimum enclosing polygons (polygon, box), bounding boxes (box), and bounding box neighboring points (box_nearby) corresponding to the background obstacles.
[0078] Among them, the annotation methods include but are not limited to manual annotation, automatic annotation, semi-supervised annotation, etc., and the present disclosure does not specifically limit this; for example, manual annotation is that the annotator observes each frame of point cloud visually through the software to determine the position of the background obstacle; then, around the background obstacle, manually click and drag a minimum bounding rectangle with the mouse, and enter other information such as the obstacle type; finally, the software saves all the laser points of the background obstacles contained in the rectangular box, as well as the corresponding position, size, direction, type and other information into a file as the annotation result. Automatic annotation is based on a deep learning algorithm, which automatically learns and calculates the probability of background obstacles appearing in the global semantic map, so as to generate the corresponding minimum bounding rectangle for the background obstacles in the current frame and the background obstacles in the historical frame when the probability is greater than a preset threshold, and obtain the location of the background obstacles and other information.
[0079] Taking laser radar as an example, the laser point corresponding to the background obstacle is the point corresponding to the laser signal reflected back by the background obstacle after the laser radar emits the laser signal. The collection of these laser points can concretely display the position, size, shape and other information of the background obstacle; the neighboring point is the laser point whose distance from the above laser point is less than or equal to the preset distance, which can represent the outline of the background obstacle; the minimum enclosing polygon can be obtained by connecting the convex polygon corner points of the background obstacle through the convex hull algorithm; the bounding box is a rectangular box obtained by connecting the laser points with the largest distance from the center of the background obstacle in all directions; the neighboring point of the bounding box is the laser point whose distance from the above bounding box is less than or equal to the preset distance. It can be understood that the above elements can correctly indicate different degrees of background obstacles. The laser point corresponding to the background obstacle can best correctly represent the background obstacle itself, and the corresponding degrees of the neighboring point, the minimum enclosing polygon, the bounding box and the neighboring point of the bounding box decrease in turn.
[0080] In some embodiments, the current frame is rasterized according to the background obstacle features included in the current frame in the global semantic map to obtain a first global semantic grid corresponding to the background obstacle in the current frame, and the historical frame is rasterized according to the background obstacle features included in the historical frame in the global semantic map to obtain a second global semantic grid corresponding to the background obstacle in the historical frame. The first global semantic grid includes the first point cloud data of the background obstacle in the current frame, and the second global semantic grid includes the second point cloud data of the background obstacle in the historical frame.
[0081] Specifically, the global semantic map is grid-divided, and a first global semantic grid including first point cloud data is determined in the grid to identify background obstacles in the current frame on the global semantic grid of the global semantic map; and a second global semantic grid including second point cloud data is determined in the grid to identify background obstacles in historical frames on the global semantic grid of the global semantic map.
[0082] S103: Calculate the intersection-and-union ratio of the second global semantic grid and the first global semantic grid according to a preset rule.
[0083] The preset rules are preset scoring rules, including scores of preset elements in the global semantic grid and scores of intersections of preset elements in intersecting global semantic grids.
[0084] The intersection-over-union ratio value is used to represent the overlap between the second global semantic grid and the first global semantic grid, so as to describe the overlap between the background obstacle of the current frame and the background obstacle of the historical frame, thereby measuring the possibility that the background obstacle of the current frame and the background obstacle of the historical frame are the same obstacle.
[0085] In some embodiments, firstly, a first score of a preset element in a first global semantic grid is calculated, and a second score of a preset element in a second global semantic grid is calculated, wherein the preset elements include a laser point (point), a neighboring point (point_nearby), a minimum enclosing polygon (polygon), a bounding box (box), and a neighboring point of a bounding box (box_nearby). Then, the intersection grid of the first global semantic grid and the second global semantic grid is determined, and then a third score of the intersection of the preset elements in the intersection grid is calculated. Furthermore, the first score, the second score, and the third score are substituted into the intersection-union ratio formula to calculate the intersection-union ratio score.
[0086] In the above embodiment, the scores corresponding to the preset elements can be preset, and the scores of the preset elements include: the score of the laser point, the score of the neighboring points, the score of the minimum outer polygon, the score of the bounding box, and the score of the neighboring points of the bounding box. The scores corresponding to the preset elements can be the same or different, and the present disclosure does not specifically limit this.
[0087] Exemplarily, as shown in Table 1, Table 1 is an exemplary score of preset elements in an embodiment of the present disclosure.
[0088] Table 1
[0089] Preset Elements Points Laser Point a1 Nearby Points a2 Minimum enclosing polygon a3 Bounding Box a4 Bounding Box Neighbors a5
[0090] The scores of the preset elements in the embodiment of the present disclosure support customization and updating, as shown in Table 2, the exemplary setting of the score of the laser point is 1.0, the score of the neighboring point is 1.0, the score of the minimum enclosing polygon is 0.9, the score of the bounding box is 0.7, and the score of the neighboring point of the bounding box is 0.4.
[0091] Table 2
[0092] Preset Elements Points Laser Point 1.0 Nearby Points 1.0 Minimum enclosing polygon 0.9 Bounding Box 0.7 Bounding Box Neighbors 0.4
[0093] As shown in Table 1 or Table 2, in the embodiment of the present disclosure, if the laser point appears once in the global semantic grid, 1.0 is added, and the more times it appears, the higher the score. If other similar preset elements appear once, the corresponding score is added.
[0094] In some embodiments, when calculating a first score for a preset element in a first global semantic grid and calculating a second score for a preset element in a second global semantic grid, the score of a preset laser point, the score of a neighboring point, the score of a minimum enclosing polygon, the score of a bounding box, and the score of neighboring points of a bounding box can be first determined; then, the scores of the laser point, the neighboring point, the minimum enclosing polygon, the bounding box, and the neighboring points of the bounding box in the first global semantic grid are counted to obtain a first score; and the scores of the laser point, the neighboring point, the minimum enclosing polygon, the bounding box, and the neighboring points of the bounding box in the second global semantic grid are counted to obtain a second score.
[0095] In the process of calculating the first score and the second score, the first score of the preset element in the first global semantic grid is the sum of the scores corresponding to all the preset elements contained in the first global semantic grid; the second score of the preset element in the second global semantic grid is the sum of the scores corresponding to all the preset elements contained in the second global semantic grid.
[0096] Specifically, the number n1 of laser points, the number n2 of neighboring points, the number n3 of minimum enclosing polygons, the number n4 of bounding boxes and the number n5 of neighboring points of the bounding boxes in the first global semantic grid are counted; the number m1 of laser points, the number m2 of neighboring points, the number m3 of minimum enclosing polygons, the number m4 of bounding boxes and the number m5 of neighboring points of the bounding boxes in the second global semantic grid are counted; then according to the preset score a1 of the laser point, the score a2 of the neighboring point, the score a3 of the minimum enclosing polygon, the score a4 of the bounding box and the score a5 of the neighboring points of the bounding box, the corresponding multiplication is performed to obtain a first score T1=n1*a1+n2*a2+n3*a3+n4*a4+n5*a5; the second score T2=m1*a1+m2*a2+m3*a3+m4*a4+m5*a5.
[0097] In the above embodiment, the preset elements in the intersecting grids of the first global semantic grid and the second global semantic grid intersect, including the following situations: the laser point intersects with the preset element, the adjacent point intersects with the preset element, the minimum enclosing polygon intersects with the preset element, the bounding box intersects with the preset element, and the adjacent point of the bounding box intersects with the preset element.
[0098] Specifically, the laser points are added, the laser points intersect with the adjacent points, the laser points intersect with the minimum enclosing polygon, the laser points intersect with the bounding box, and the laser points intersect with the adjacent points of the bounding box; the adjacent points intersect with the adjacent points, the adjacent points intersect with the minimum enclosing polygon, the adjacent points intersect with the bounding box, and the adjacent points intersect with the adjacent points of the bounding box; the minimum enclosing polygon intersects with the minimum enclosing polygon, the minimum enclosing polygon intersects with the bounding box, and the minimum enclosing polygon intersects with the adjacent points of the bounding box; the bounding box intersects with the bounding box, and the bounding box intersects with the adjacent points of the bounding box; the bounding box intersects with the bounding box, and the bounding box intersects with the adjacent points of the bounding box; the adjacent points of the bounding box intersect with the adjacent points of the bounding box.
[0099] When calculating the third score of the intersection of preset elements in the intersecting grid, the score of the intersection of the preset elements is first determined, including: the score of the intersection of the laser point with the preset element, the score of the intersection of the adjacent point with the preset element, the score of the intersection of the minimum enclosing polygon with the preset element, the score of the intersection of the bounding box with the preset element, and the score of the intersection of the bounding box adjacent point with the preset element.
[0100] The embodiment of the present disclosure supports customizing the score of intersection of preset elements in the intersection grid. For example, as shown in Table 3, Table 3 is an exemplary setting of the score of intersection of preset elements in the intersection grid of the present disclosure.
[0101] Table 3
[0102]
[0103] As shown in Table 3, the score of the intersection of preset elements in the intersection grid can be set as follows: the score of the intersection between laser points is 1.0, the score of the intersection between laser points and adjacent points is 1.0, the score of the intersection between laser points and the minimum enclosing polygon is 0.9, the score of the intersection between laser points and the bounding box is 0.7, and the score of the intersection between laser points and adjacent points of the bounding box is 0.4; the score of the intersection between adjacent points and adjacent points is 1.0, the score of the intersection between adjacent points and the minimum enclosing polygon is 0.9, and the score of the intersection between adjacent points and the edge is 0. The score for bounding box intersection is 0.7, and the score for intersection between neighboring points and neighboring points of the bounding box is 0.4; the score for intersection between minimum enclosing polygon and minimum enclosing polygon is 0.8, the score for intersection between minimum enclosing polygon and bounding box is 0.6, and the score for intersection between minimum enclosing polygon and neighboring points of the bounding box is 0.3; the score for intersection between bounding boxes is 0.5, and the score for intersection between bounding boxes and neighboring points of the bounding box is 0.2; the score for intersection between neighboring points of the bounding box and neighboring points of the bounding box is 0.1.
[0104] As shown in Table 3, if the laser points in the intersecting grid intersect once, the third score is increased by 1.0, and the score calculation corresponding to the intersection of other preset elements is the same.
[0105] When calculating the third score of the intersection of preset elements in the intersecting grids, after determining the score of the preset intersection of the preset elements, the scores of the intersection of the preset elements in the intersecting grids of the first global semantic grid and the second global semantic grid are added. Specifically, the scores of the intersection of the laser point with the preset element, the scores of the intersection of the neighboring points with the preset element, the scores of the intersection of the minimum enclosing polygon with the preset element, the scores of the intersection of the bounding box with the preset element, and the scores of the intersection of the neighboring points of the bounding box with the preset element in the intersecting grids are counted. In this process, the number of intersections of the preset elements is first counted, including: the number of intersections between the laser point and the preset element, the number of intersections between the neighboring points and the preset element, the number of intersections between the minimum enclosing polygon and the preset element, the number of intersections between the bounding box and the preset element, and the number of intersections between the neighboring points of the bounding box and the preset element.
[0106] Exemplarily, assuming that in the intersecting grid, the number of intersections between laser points and laser points is s1, the number of intersections between laser points and neighboring points is s2, the number of intersections between laser points and the minimum enclosing polygon is s3, the number of intersections between laser points and the bounding box is s4, and the number of intersections between laser points and neighboring points of the bounding box is s5; the number of intersections between neighboring points is s6, the number of intersections between neighboring points and the minimum enclosing polygon is s7, the number of intersections between neighboring points and the bounding box is s8, and the number of intersections between neighboring points and neighboring points of the bounding box is s9; the number of intersections between the minimum enclosing polygon and the minimum enclosing polygon is s10, the number of intersections between the minimum enclosing polygon and the bounding box is s11, and the number of intersections between the minimum enclosing polygon and neighboring points of the bounding box is s12; the number of intersections between the bounding box and the bounding box is s13, and the number of intersections between the bounding box and neighboring points of the bounding box is s14; the number of intersections between neighboring points of the bounding box and neighboring points of the bounding box is s15.
[0107] Combined with the scores of the intersection of the preset elements shown in Table 3, the scores and quantities are multiplied accordingly to obtain the third score T3=1.0*s1+1.0*s2+0.9*s3+0.7*s4+0.4*s5+1.0*s6+0.9*s7+0.7*s8+0.4*s9+0.8*s10+0.6*s11+0.3*s12+0.5*s13+0.2*s14+0.1*s15.
[0108] Based on the above embodiment, after the first score, the second score and the third score are calculated, they are substituted into the intersection-and-union ratio formula, such as formula (1):
[0109] IoU = (A∩B) / (A∪B) (1)
[0110] Among them, (A∩B) is the third score, such as T3 mentioned above; (A∪B) is the sum of the first score and the second score, such as (A∪B)=T1+T2.
[0111] The calculated intersection-and-union ratio value is used to indicate the overlap between the second global semantic grid and the first global semantic grid, and also indicates the overlap between the background obstacle in the current frame and the background obstacle in the historical frame, so as to describe the possibility that the background obstacle in the current frame and the background obstacle in the historical frame are the same obstacle. After obtaining the intersection-and-union ratio value, the intersection-and-union ratio value is compared with the threshold value to determine the correlation between the background obstacle in the current frame and the background obstacle in the historical frame. Among them, the threshold value is preset and indicates the minimum value of the overlap between the two global semantic grids.
[0112] In the above embodiment, the background obstacles in the current frame and the background obstacles in the historical frames are placed in the layer of the global semantic map, and after rasterization, the corresponding global semantic grid intersection score is determined to indicate the degree of overlap between the background obstacles in the current frame and the background obstacles in the historical frames.
[0113] S104: When the intersection-and-union ratio value is greater than or equal to the threshold, the background obstacles in the current frame are associated with the background obstacles in the historical frames.
[0114] When the intersection-and-union ratio value is greater than or equal to the threshold, it means that the overlap degree between the second global semantic grid and the first global semantic grid is high, and it is highly likely that the background obstacle in the current frame corresponding to the second global semantic grid and the background obstacle in the historical frame corresponding to the first global semantic grid are the same obstacle, so the background obstacle in the current frame is associated with the background obstacle in the historical frame.
[0115] When the intersection-and-combination ratio value is less than the threshold, it is determined that the background obstacle in the current frame is independent of the background obstacle in the historical frame and is not the same background obstacle.
[0116] In summary, the embodiment of the present disclosure provides a method for associating background obstacles. First, a global semantic map including background obstacles is constructed according to odometer information. Then, based on the global semantic map, a first global semantic grid of the background obstacle in the current frame and a second global semantic grid of the background obstacle in the historical frame are obtained. Then, according to a preset rule, the intersection-and-union ratio value of the second global semantic grid and the first global semantic grid is calculated. The intersection-and-union ratio value is used to indicate the overlap between the second global semantic grid and the first global semantic grid, and also describes the relative size of the overlap between the current obstacle and the background obstacle. The higher the overlap, the greater the possibility that the current background obstacle and the historical obstacle are the same obstacle. If the intersection-and-union ratio value is greater than or equal to the threshold, it means that the current background obstacle is likely to be the same obstacle as the historical obstacle, and then the two are associated. The present disclosure associates and tracks the background obstacles through the intersection-and-union ratio value of the global semantic grid of the background obstacle in the current frame and the background obstacle in the historical frame on the global semantic map, thereby improving the accuracy and stability of background obstacle detection.
[0117] like Figure 2 As shown, Figure 2 A schematic diagram of a background obstacle association device provided in an embodiment of the present disclosure, the device comprising:
[0118] A map construction module 201 is used to: construct a global semantic map according to odometer information, wherein the global semantic map includes background obstacles;
[0119] The processing module 202 is used to: obtain a first global semantic grid of background obstacles in a current frame and a second global semantic grid of background obstacles in a historical frame based on the global semantic map;
[0120] The calculation module 203 is used to calculate the intersection-and-union ratio of the second global semantic grid and the first global semantic grid according to a preset rule;
[0121] The association module 204 is configured to: when the intersection-and-combination ratio value is greater than or equal to a threshold, associate the background obstacle in the current frame with the background obstacle in the historical frame.
[0122] As an optional implementation provided by an embodiment of the present disclosure, the calculation module 203 is specifically used to: calculate a first score of a preset element in the first global semantic grid, and calculate a second score of a preset element in the second global semantic grid, wherein the preset elements include a laser point, a neighboring point, a minimum enclosing polygon, a bounding box, and a neighboring point of the bounding box corresponding to a background obstacle;
[0123] determining an intersection grid of the first global semantic grid and the second global semantic grid;
[0124] Calculating a third score of the intersection of the preset elements in the intersecting grids;
[0125] Substitute the first score, the second score, and the third score into the intersection-and-union ratio formula to calculate the intersection-and-union ratio score.
[0126] As an optional implementation provided by the embodiment of the present disclosure, the calculation module 203 is specifically used to: determine the score of a preset laser point, the score of a neighboring point, the score of a minimum enclosing polygon, the score of a bounding box, and the score of a neighboring point of the bounding box;
[0127] The first score is obtained by adding the score of the laser point corresponding to the first global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box;
[0128] The second score is obtained by adding the score of the laser point corresponding to the second global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box.
[0129] As an optional implementation provided by an embodiment of the present disclosure, the calculation module 203 is specifically used to: determine the score of the intersection between a preset laser point and a preset element, the score of the intersection between a neighboring point and the preset element, the score of the intersection between a minimum enclosing polygon and the preset element, the score of the intersection between a bounding box and the preset element, and the score of the intersection between a neighboring point of the bounding box and the preset element;
[0130] The third score is obtained by adding the score of the intersection between the laser point corresponding to the intersecting grid and the preset element, the score of the intersection between the adjacent point and the preset element, the score of the intersection between the minimum enclosing polygon and the preset element, the score of the intersection between the bounding box and the preset element, and the score of the intersection between the adjacent point of the bounding box and the preset element.
[0131] As an optional implementation provided by an embodiment of the present disclosure, the map construction module 201 is further used to: before constructing the global semantic map according to the odometer information, obtain a point cloud sequence collected by a sensor, wherein the sensor includes an image sensor and / or a laser sensor, the point cloud series is a set of point data on the appearance surface of the measured object collected by the laser sensor, and the point cloud sequence includes point cloud information of background obstacles;
[0132] Performing semantic segmentation on the point cloud sequence to obtain semantic features, wherein the semantic features include background obstacle features;
[0133] The map construction module 201 is specifically used to: obtain odometer information and determine the position and posture information corresponding to the odometer information;
[0134] The semantic features are projected into a global coordinate system based on the posture information to obtain the global semantic map.
[0135] As an optional implementation provided by an embodiment of the present disclosure, the map construction module 201 is also used to: mark the background obstacles in the current frame and the historical frame of the global semantic map, and the marked elements include laser points, neighboring points, minimum enclosing polygons, bounding boxes and bounding box neighboring points corresponding to the background obstacles.
[0136] As an optional implementation provided by an embodiment of the present disclosure, the processing module 202 is specifically used to: rasterize the current frame according to the background obstacle features included in the current frame in the global semantic map to obtain a first global semantic grid corresponding to the background obstacle in the current frame;
[0137] According to the background obstacle features included in the historical frames in the global semantic map, the historical frames are rasterized to obtain a second global semantic grid corresponding to the background obstacles in the historical frames.
[0138] In summary, the embodiment of the present disclosure provides a device for associating background obstacles. First, the map construction module constructs a global semantic map including background obstacles according to the odometer information. Then, the processing module obtains the first global semantic grid of the background obstacle in the current frame and the second global semantic grid of the background obstacle in the historical frame based on the global semantic map. Then, the calculation module calculates the intersection-and-union score of the second global semantic grid and the first global semantic grid according to the preset rule. The intersection-and-union score is used to indicate the overlap between the second global semantic grid and the first global semantic grid, and also describes the relative size of the overlap between the current obstacle and the background obstacle. The higher the overlap, the greater the possibility that the current background obstacle and the historical obstacle are the same obstacle. If the intersection-and-union score is greater than or equal to the threshold, it means that the current background obstacle is likely to be the same obstacle as the historical obstacle, and then the two are associated through the association module. The present disclosure associates and tracks the background obstacles through the intersection-and-union score of the global semantic grid of the background obstacle in the current frame and the background obstacle in the historical frame on the global semantic map, thereby improving the accuracy and stability of background obstacle detection.
[0139] like Figure 3 As shown, Figure 3 The present disclosure provides a schematic diagram of the structure of a vehicle computer, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, each process of the method for associating background obstacles in the above method embodiment is implemented. The same technical effect can be achieved, and it will not be repeated here to avoid repetition.
[0140] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the method for associating background obstacles in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0141] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0142] An embodiment of the present invention provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the method for associating background obstacles in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0143] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.
[0144] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0145] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0146] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0147] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0148] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for associating background obstacles, characterized in that: include: Building a global semantic map based on odometer information, wherein the global semantic map includes background obstacles; Based on the global semantic map, obtaining a first global semantic grid of background obstacles in a current frame and a second global semantic grid of background obstacles in a historical frame; Calculating an intersection-and-union score between the second global semantic grid and the first global semantic grid according to a preset rule, wherein the intersection-and-union score is used to indicate a degree of overlap between the second global semantic grid and the first global semantic grid; When the intersection-over-combination ratio value is greater than or equal to a threshold, the background obstacle in the current frame is associated with the background obstacle in the historical frame.
2. The method according to claim 1, characterized in that The calculating the intersection-and-combination score of the second global semantic grid and the first global semantic grid according to a preset rule includes: Calculating a first score of a preset element in the first global semantic grid, and calculating a second score of a preset element in the second global semantic grid, wherein the preset elements include a laser point, a neighboring point, a minimum enclosing polygon, a bounding box, and a neighboring point of the bounding box corresponding to a background obstacle; determining an intersection grid of the first global semantic grid and the second global semantic grid; Calculating a third score of the intersection of the preset elements in the intersecting grids; Substitute the first score, the second score, and the third score into the intersection-and-union ratio formula to calculate the intersection-and-union ratio score.
3. The method according to claim 2, characterized in that The calculating a first score of a preset element in the first global semantic grid and the calculating a second score of a preset element in the second global semantic grid include: Determine the score of the preset laser point, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box; The first score is obtained by adding the score of the laser point, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box and the score of the neighboring points of the bounding box corresponding to the first global semantic grid; The second score is obtained by adding the score of the laser point corresponding to the second global semantic grid, the score of the neighboring points, the score of the minimum enclosing polygon, the score of the bounding box, and the score of the neighboring points of the bounding box.
4. The method according to claim 2, characterized in that: The calculating a third score of the intersection of the preset elements in the intersecting grids includes: Determine the score of the intersection of a preset laser point with a preset element, the score of the intersection of an adjacent point with the preset element, the score of the intersection of a minimum enclosing polygon with the preset element, the score of the intersection of a bounding box with the preset element, and the score of the intersection of an adjacent point of the bounding box with the preset element; The third score is obtained by adding the score of the intersection between the laser point corresponding to the intersecting grid and the preset element, the score of the intersection between the adjacent point and the preset element, the score of the intersection between the minimum enclosing polygon and the preset element, the score of the intersection between the bounding box and the preset element, and the score of the intersection between the adjacent point of the bounding box and the preset element.
5. The method according to claim 1, characterized in that Before constructing the global semantic map according to the odometer information, the method further includes: Acquire a point cloud sequence collected by a sensor, wherein the sensor includes an image sensor and / or a laser sensor, the point cloud sequence is a set of point data of the appearance surface of the measured object collected by the sensor, and the point cloud sequence includes point cloud information of background obstacles; Performing semantic segmentation on the point cloud sequence to obtain semantic features, wherein the semantic features include background obstacle features; The step of constructing a global semantic map based on odometer information includes: Obtain odometer information and determine the posture information corresponding to the odometer information; The semantic features are projected into a global coordinate system based on the posture information to obtain the global semantic map.
6. The method according to claim 5, characterized in that After projecting the semantic features to a global coordinate system based on the posture information to obtain the global semantic map, the method further includes: The background obstacles in the current frame and the historical frames of the global semantic map are annotated, and the annotated elements include laser points, neighboring points, minimum enclosing polygons, bounding boxes, and neighboring points of the bounding boxes corresponding to the background obstacles.
7. The method according to claim 6, characterized in that The acquiring, based on the global semantic map, a first global semantic grid of background obstacles in the current frame and a second global semantic grid of background obstacles in the historical frame comprises: rasterizing the current frame according to the background obstacle features included in the current frame in the global semantic map to obtain a first global semantic grid corresponding to the background obstacle in the current frame; According to the background obstacle features included in the historical frames in the global semantic map, the historical frames are rasterized to obtain a second global semantic grid corresponding to the background obstacles in the historical frames.
8. A device for associating background obstacles, characterized in that: include: A map construction module, used to: construct a global semantic map according to odometer information, wherein the global semantic map includes background obstacles; A processing module, configured to: obtain, based on the global semantic map, a first global semantic grid of background obstacles in a current frame and a second global semantic grid of background obstacles in a historical frame; A calculation module, used to calculate the intersection-and-union ratio of the second global semantic grid and the first global semantic grid according to a preset rule; The association module is used to: when the intersection-and-combination ratio value is greater than or equal to a threshold, associate the background obstacle in the current frame with the background obstacle in the historical frame.
9. A vehicle computer, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the background obstacle association method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for associating background obstacles according to any one of claims 1 to 7 is implemented.
11. A vehicle, characterized in that: include: The background obstacle association device as claimed in claim 8, or the vehicle computer as claimed in claim 9.