Method, apparatus and medium for multi-frame fusion of static obstacle grounding contour lines

Abstract

This application discloses a multi-frame fusion method, apparatus, and medium for static obstacle ground contour lines, belonging to the field of autonomous driving technology. The method includes: a preprocessing process; a single-frame point acquisition process; a single-frame point clustering process; a single-frame point cluster matching process; and a multi-frame fusion process. In the single-frame point cluster matching process, all single-frame points matched into any point cluster are transformed to a global Cartesian coordinate system and rasterized around them to obtain a temporary map. If the total number of occurrences of a corresponding single-frame point in a grid of the temporary map from multiple consecutive environmental images exceeds a single-frame point occurrence threshold, the information of that grid is stored in a stable map. The multi-frame fusion method for static obstacle ground contour lines in this application has a near-linear computational complexity, resulting in low computational load and fast processing speed, meeting the requirements of autonomous driving for the speed, accuracy, and stability of static obstacle recognition.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus and medium for multi-frame fusion of static obstacle grounding contour lines. Background Technology

[0002] In autonomous driving, ensuring driving safety requires real-time route planning to avoid collisions with potential obstacles. This necessitates the dynamic and real-time identification and precise location of obstacles during autonomous driving. Current technologies primarily rely on installing LiDAR, ultrasonic sensors, and other detection devices on vehicles to directly measure the distance between the obstacle and the autonomous vehicle. However, LiDAR and ultrasonic sensors are costly, and directly using the measured distance for obstacle avoidance planning requires enormous computational resources, resulting in significant data processing and transmission delays, making them difficult to apply directly to the real-time demands of autonomous driving. Furthermore, the data's instability and presence of noise can severely interfere with the algorithm's judgment when directly applied to autonomous driving. Summary of the Invention

[0003] In view of the technical problem that the amount of computation required for vehicle obstacle avoidance planning in the prior art is too large, resulting in excessive delays in data processing and transmission, this application proposes a method, device and medium for multi-frame fusion of static obstacle grounding contour lines.

[0004] In one technical solution of this application, a multi-frame fusion method for static obstacle ground contour lines is provided, which includes: a preprocessing process, in which obstacle contour points are extracted from each frame of environmental image captured by cameras deployed around the vehicle, and points located outside a predetermined geometric range are filtered out to obtain filtered obstacle contour points; a single-frame point acquisition process, in which the filtered obstacle contour points corresponding to each frame of environmental image are transformed into a polar coordinate system with the center of the vehicle as the origin, and among all filtered obstacle contour points with the same angular coordinates, the point closest to the origin of the polar coordinate system is retained as a single-frame point; a single-frame point clustering process, in which all single-frame points are sorted in order of angular coordinate values, and for each single-frame point, its nearest other single-frame point is taken, and the angular coordinate difference and relative distance between the two single-frame points are calculated. If the angular coordinate difference is less than the angular coordinate difference, the clustering process is performed. If the standard deviation threshold is met and the relative distance is less than the relative distance threshold, then the two single-frame points are clustered into the same single-frame point cluster. In the single-frame point cluster matching process, the current frame point cluster belonging to the current frame environment image and the historical frame point cluster belonging to the adjacent historical frame environment image are traversed. If the distance between the current frame point cluster and the historical frame point cluster is less than the first distance threshold, and the degree of overlap between the projection of the current frame point cluster and the historical frame point cluster is greater than the overlap degree threshold, then the current frame point cluster and the historical frame point cluster are matched into the same point cluster class. In the multi-frame fusion process, all single-frame points matched into any point cluster class in the single-frame point cluster matching process are transformed into the global Cartesian coordinate system and rasterized around them to obtain a temporary map. If the total number of times the corresponding single-frame point of the consecutive multi-frame environment image has appeared in a grid of the temporary map is greater than the single-frame point occurrence frequency threshold, then the information of that grid is stored in the stable map.

[0005] In another technical solution of this application, a multi-frame fusion device for static obstacle ground contour lines is provided, comprising: a preprocessing module, which extracts obstacle contour points for each frame of environmental image captured by cameras deployed around the vehicle, and filters out points located outside a predetermined geometric range among the obstacle contour points to obtain filtered obstacle contour points; a single-frame point acquisition module, which transforms the filtered obstacle contour points corresponding to each frame of environmental image into a polar coordinate system with the center of the vehicle as the origin, and retains the point closest to the origin of the polar coordinate system among all filtered obstacle contour points with the same angular coordinates as a single-frame point; and a single-frame point clustering module, which sorts all single-frame points in order of angular coordinate values, and for each single-frame point, takes its nearest other single-frame point, calculates the angular coordinate difference and relative distance between the two single-frame points, and if the angular coordinate difference is less than the angle... If the coordinate difference threshold is met and the relative distance is less than the relative distance threshold, then the two single-frame points are clustered into the same single-frame point cluster. The single-frame point cluster matching module traverses the current frame point cluster belonging to the current frame environment image and the historical frame point cluster belonging to the adjacent historical frame environment image. If the distance between the current frame point cluster and the historical frame point cluster is less than the first distance threshold, and the degree of overlap between the projection of the current frame point cluster and the historical frame point cluster is greater than the overlap degree threshold, then the current frame point cluster and the historical frame point cluster are matched into the same point cluster class. The multi-frame fusion module transforms all single-frame points matched into any point cluster class during the single-frame point cluster matching process into the global Cartesian coordinate system and rasterizes them around them to obtain a temporary map. If the total number of times the corresponding single-frame point of the consecutive multi-frame environment image has appeared in a grid of the temporary map is greater than the single-frame point occurrence frequency threshold, then the information of that grid is stored in the stable map.

[0006] In another technical solution of this application, a computer-readable storage medium is provided, which stores computer instructions that are operated to execute the static obstacle grounding contour multi-frame fusion method in the above technical solution.

[0007] In another technical solution of this application, a computer device is provided, comprising: a memory storing computer instructions; and a processor that operates the computer instructions to execute the static obstacle grounding contour multi-frame fusion method of the above technical solution.

[0008] By adopting the above technical solution, this application can achieve the following technical effects: the complexity of the calculation process is close to linear, thus the amount of calculation is small and the processing speed is fast, which can meet the requirements of autonomous driving for the recognition speed, recognition accuracy and recognition stability of static obstacles. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the accompanying drawings used in the description of the specific embodiments of this application or the prior art will be briefly described below. Obviously, the accompanying drawings described below pertain to some specific embodiments of this application. For those skilled in the art, without any creative effort, they can directly and unambiguously determine other drawings corresponding to equivalent substitutions or modifications of the specific embodiments based on these drawings.

[0010] Figure 1 A flowchart illustrating a specific implementation of the static obstacle grounding contour multi-frame fusion method of this application is shown.

[0011] Figure 2 This diagram illustrates an example of a single-frame point cluster matching process in a specific embodiment of the multi-frame fusion method for static obstacle grounding contour lines of this application.

[0012] Figure 3 This diagram illustrates an example of the multi-frame fusion process in a specific embodiment of the static obstacle grounding contour multi-frame fusion method of this application;

[0013] Figure 4 The diagram shows a schematic representation of the constituent modules of a specific embodiment of the static obstacle grounding contour multi-frame fusion device of this application.

[0014] The accompanying drawings clearly illustrate some specific embodiments of this application, which will be described in more detail below. The purpose of these drawings and descriptions is not to limit the scope of protection of the inventive concept in any way, but rather to enable those skilled in the art to more easily understand the inventive idea of ​​this application through specific embodiments. Detailed Implementation

[0015] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, some specific embodiments of the technical solutions of this application will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described specific embodiments are only a part of the specific embodiments of the technical solutions of this application, and not all of them. Based on the specific embodiments already described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0016] It should be noted that the terms "first," "second," "third," "fourth," etc. (if any) used in this application are used to distinguish similar objects and are not necessarily construed as describing a specific order or sequence. It should be understood that these numbers can be interchanged where appropriate so that the technical solutions of this application described herein can be implemented, for example, in an order other than that shown in the figures or descriptions. Furthermore, the terms "comprising" and "having," and any variations thereof, are not intended to cover only exclusive inclusion. For example, a method comprising a series of steps, or a product or device comprising several units or modules, is not necessarily limited to the listed steps, units, or modules, but may also include other steps, units, or modules not listed or inherent to these methods, products, or devices themselves.

[0017] Furthermore, the various specific embodiments, examples, and instances described below for different parts of the technical solution of this invention can be combined with each other to form a complete technical solution of this invention without mutual exclusion. Concepts and processes that are the same or similar as those described in a particular embodiment, example, or instance may not be repeated in other embodiments, examples, or instances.

[0018] The following, with reference to the accompanying drawings, provides a detailed description of some specific embodiments, examples, and instances of this application.

[0019] Figure 1 This is a flowchart illustrating a specific implementation of the static obstacle grounding contour multi-frame fusion method of this application.

[0020] exist Figure 1 In the embodiment shown, the static obstacle grounding contour multi-frame fusion method of this application includes: S101, preprocessing.

[0021] In the preprocessing process represented by S101, obstacle contour points are extracted for each frame of environmental image captured by cameras deployed around the vehicle, and points outside a predetermined geometric range are filtered out to obtain filtered obstacle contour points.

[0022] In one embodiment of this implementation, one or more cameras may be deployed around the vehicle. When there are multiple vehicle cameras, they may be installed at different locations on the vehicle. Each frame of the environmental image captured by the cameras may be the environmental image obtained by fusing the corresponding environmental images captured by the multiple vehicle cameras.

[0023] In one embodiment of this implementation, the process of extracting obstacle contour points for each frame of environmental image captured by cameras deployed around the vehicle can be performed using a dedicated deep learning-based neural network algorithm, or by using other existing technologies.

[0024] In one embodiment of this implementation, the process of filtering out points located outside a predetermined geometric range from the obstacle contour points to obtain filtered obstacle contour points may include: removing points from the obstacle contour points that exceed the field of view angle threshold of the camera, and removing points whose distance from the camera is less than a second distance threshold or greater than a third distance threshold, wherein the second distance threshold is less than the third distance threshold.

[0025] In one example of this embodiment, the aforementioned field of view angle threshold can be any value between 90 and 150 degrees, the second distance threshold can be any value between 3 and 7 meters, and the third distance threshold can be any value between 8 and 12 meters. Optionally, the included angle of the field of view is 120 degrees, the second distance threshold is 6 meters, and the third distance threshold is 10 meters. These thresholds are related to the shooting distance range where the vehicle camera has good shooting accuracy. When using different vehicle cameras, these thresholds can be adjusted accordingly, or even adjusted beyond the above range.

[0026] In one embodiment of this implementation, for each frame of environmental image captured by cameras deployed around the vehicle, obstacle contour points exceeding the field of view threshold of the vehicle's cameras are removed, as are pixels that are too close or too far from the cameras. This allows for processing only obstacle contour points that meet the threshold requirements during the subsequent single-frame point acquisition process (S101), increasing the targeting of the processing, reducing computational load, minimizing unnecessary noise, and improving the processing speed of the technical solution. This makes the technical solution of this application better suited for autonomous driving applications with high real-time requirements.

[0027] exist Figure 1 In the embodiment shown, the multi-frame fusion method for static obstacle grounding contour lines of this application includes: S102, single-frame point acquisition process.

[0028] In the single-frame point acquisition process represented by S102, the filtered obstacle contour points corresponding to each frame of the environment image are transformed into a polar coordinate system with the center of the vehicle as the origin. Among all the filtered obstacle contour points with the same angular coordinates, the point closest to the origin of the polar coordinate system is retained as the single-frame point.

[0029] In one embodiment of this implementation, the center of the vehicle, and the origin of the polar coordinate system, can be the intersection of the diagonals of the quadrilateral formed by the four corners of the vehicle.

[0030] In one embodiment of this implementation, the number of coordinate values ​​of the aforementioned angular coordinates in the polar coordinate system can be an integer multiple or fraction of 360, such as 360, 720, 180, etc. The following explanation uses an example where the number of coordinate values ​​of the multiple angular coordinates in the polar coordinate system is 360.

[0031] When the number of angular coordinate values ​​in a polar coordinate system is 360, that is, with the vehicle center as the origin of the polar coordinate system, it can rotate 360 ​​degrees. In this way, with a resolution of 1 degree, 360 rays originating from the origin are formed in the polar coordinate system. Let the origin be represented by P0, and the angular coordinate values ​​corresponding to the 360 ​​rays be represented by θ. Thus, the angular coordinate value θ of the polar coordinate system has 360 possible values.

[0032] As shown above, each value of the angular coordinate θ in the polar coordinate system corresponds to a ray originating from the origin of the polar coordinate system. This ray is projected onto each frame of the environmental image captured by the vehicle camera. If it projects onto one or more pixels, it is considered to have been projected onto the corresponding point on the obstacle contour. The pixel with the smallest distance coordinate value in the polar coordinate system, i.e., the obstacle contour point closest to the vehicle center among the multiple projected pixels, is taken as a single-frame point. This single-frame point is considered to be the corresponding point on the ground contour line of the obstacle facing the vehicle camera. In this case, the single-frame point is recorded and retained. Optionally, the single-frame point can be recorded by recording its coordinates.

[0033] Similarly, specific implementation examples can be obtained when the number of coordinate values ​​for multiple angle coordinates in the polar coordinate system is other than a certain number.

[0034] In one embodiment of this implementation, the single-frame point acquisition process represented by S102 may further include a distance smoothing process, that is, dividing the single-frame points into multiple adjacent groups at equal intervals according to the corresponding angular coordinates, and using the Gaussian algorithm to perform a weighted average of the distance coordinates of multiple single-frame points in each group of single-frame points.

[0035] In the distance smoothing process of this embodiment, taking the case where the number of coordinate values ​​of multiple angle coordinates in the polar coordinate system in the above embodiment is 360 as an example, the single frame point can be divided into multiple groups according to the angle coordinate interval of 5 degrees.

[0036] Optionally, within each group, for any single frame point whose distance coordinate value falls outside a predetermined distance range, its distance coordinate value can be reset to zero. This predetermined distance range is related to the effective shooting distance range of the vehicle camera; for example, it can be 0.5-6.5 meters from the coordinate center of the vehicle camera. When using different vehicle cameras, this predetermined distance range can be adjusted accordingly, thus not being limited to the above-mentioned numerical range.

[0037] Alternatively, a weighted average of the distance coordinates of a single frame point can be calculated using a Gaussian algorithm with weights {1 / 9, 2 / 9, 3 / 9, 2 / 9, 1 / 9}.

[0038] By using the Gaussian algorithm to perform a weighted average of the distance coordinates of each group of single-frame points, the distance coordinates can be smoothed, and missing gaps can be filled in when connecting the points of each group of single frames. In this way, the smoothness of the static obstacle grounding contour line obtained by the technical solution of this application can be improved.

[0039] In this embodiment, the single-frame point acquisition process represented by S102 may further include a boundary gradient sharpening process. That is, the distance coordinates of multiple single-frame points within each group of single-frame points processed by the above-mentioned distance smoothing process are further weighted and averaged using the Sobel operator. This can sharpen the boundary gradient. Consequently, the static obstacle grounding contour lines obtained by the technical solution of this application can be made clearer.

[0040] Optionally, taking the specific example of the above distance smoothing process, the distance coordinates of a single frame point can be weighted and averaged using the Gaussian algorithm with weights {1 / 9, 2 / 9, 3 / 9, 2 / 9, 1 / 9}, and then further sharpened using the Sobel operator with weights {-1, -2, 0, 2, 1}.

[0041] exist Figure 1 In the embodiments shown, the multi-frame fusion method for static obstacle grounding contour lines of this application further includes: S103, single-frame point clustering process.

[0042] In the single-frame point clustering process represented by S103, after sorting all single-frame points in order of their angular coordinate values, for each single-frame point, take its nearest other single-frame point, calculate the angular coordinate difference and relative distance between the two single-frame points. If the angular coordinate difference is less than the angular coordinate difference threshold and the relative distance is less than the relative distance threshold, then the two single-frame points are clustered into the same single-frame point cluster.

[0043] In the single-frame point clustering process, all single-frame points are first sorted in order of their angular coordinate values. Then, the nearest neighboring single-frame point of each single-frame point is taken, and the difference in their angular coordinates and their relative distance are calculated. This is done instead of traversing any two single-frame points in an unordered manner. This greatly reduces the computational complexity, improves the processing speed of the technical solution, and makes the technical solution of this application more suitable for autonomous driving with high real-time requirements.

[0044] In one embodiment of this application, the relative distance can be the distance between two corresponding single-frame points. This distance can be calculated using the polar coordinate system angular coordinates and distance coordinates of the two single-frame points, or it can be calculated using the coordinates of the global Cartesian coordinate system of the two single-frame points.

[0045] In this embodiment, by grouping all pairs of single-frame points that are less than the angular coordinate difference threshold and the relative distance threshold into the same single-frame point cluster, it can be ensured that single-frame points in the same single-frame point cluster often belong to points on the grounding contour line of the same static obstacle.

[0046] In one embodiment of this implementation, the angular coordinate difference threshold between two single-frame points in the aforementioned polar coordinate system with the vehicle center as the origin can be any angle value within the range of 3-6 degrees, optionally, the angular coordinate difference threshold is 5 degrees. Furthermore, the relative distance threshold between the two single-frame points can be any value within the range of 0.3-0.7 meters, optionally, the relative distance threshold is 0.5 meters.

[0047] In another embodiment of this implementation, the single-frame point clustering process represented by S103 may also include a single-frame point cluster removal process, that is, removing single-frame point clusters that contain fewer single-frame points than the single-frame point number threshold.

[0048] Optionally, the threshold for the number of points per frame can be a natural number between 5 and 10.

[0049] Clusters of single-frame points with a number of points less than the threshold number of single-frame points are removed to avoid misidentifying temporarily appearing dynamic obstacle images or images that are not obstacles as static obstacles, thus ensuring the accuracy of obstacle identification and, consequently, the accuracy of the ground contour lines of static obstacles.

[0050] exist Figure 1 In the embodiments shown, the multi-frame fusion method for static obstacle grounding contour lines of this application further includes: S104, single-frame point cluster matching process.

[0051] In the single-frame point cluster matching process represented by S103, the current frame point cluster belonging to the current frame environment image and the historical frame point cluster belonging to the historical frame environment image are traversed. If the distance between the current frame point cluster and the historical frame point cluster is less than the first distance threshold, and the degree of overlap between the projection of the current frame point cluster and the historical frame point cluster is greater than the overlap threshold, then the current frame point cluster and the historical frame point cluster are matched to the same point cluster class.

[0052] In one embodiment of this implementation, the distance between the current frame point cluster and the historical frame point cluster can be the horizontal coordinate distance between the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system. The degree of overlap between the projections of the current frame point cluster and the historical frame point cluster can be the degree of overlap between the horizontal coordinate ranges covered by the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system.

[0053] Optionally, the horizontal coordinate distance between the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system can be the distance between the projection points in the horizontal coordinate plane between the feature points that represent the current frame point cluster and the feature points that represent the historical frame point cluster.

[0054] Optionally, the feature points of the current frame point cluster can be the centroids of all or some single frame points in the current frame point cluster, and the feature points of the historical frame point cluster can be the centroids of all or some single frame points in the historical frame point cluster.

[0055] In particular, such as Figure 2 As shown, the feature point of the current frame point cluster can be the midpoint C1 of the line connecting the two farthest endpoints P1 and P2 among the projection points of the single frame points in the current frame point cluster on the horizontal coordinate plane. The feature point of the historical frame point cluster can be the midpoint C2 of the line connecting the two farthest endpoints Q1 and Q2 among the projection points of the single frame points in the historical frame point cluster on the horizontal coordinate plane.

[0056] Correspondingly, the horizontal coordinate distance between the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system is the distance between C1, the midpoint of the line connecting the two farthest endpoints P1 and P2 of the single frame point in the current frame point cluster on the horizontal coordinate plane, and C2, the midpoint of the line connecting the two farthest endpoints Q1 and Q2 of the single frame point in the historical frame point cluster on the horizontal coordinate plane.

[0057] In this embodiment, the first distance threshold can be any value within the range of 0.3-0.7 meters, and optionally, the first distance threshold is 0.5 meters. Thus, provided that the distance between the current frame point cluster and the historical frame point cluster is less than the first distance threshold, it can be ensured that the current frame point cluster and the historical frame point cluster often belong to the grounding contour line of the same static obstacle.

[0058] In one embodiment of this implementation, the degree of overlap between the projection of the current frame point cluster and the historical frame point cluster can be the degree of overlap between the horizontal coordinate ranges covered by the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system.

[0059] In particular, such as Figure 2 As shown, the degree of overlap between the horizontal coordinate ranges covered by the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system can be expressed as the degree of overlap of the projections of the two furthest endpoints P1 and P2 of a single frame point in the current frame point cluster onto the horizontal coordinate plane, and the projections of the two furthest endpoints Q1 and Q2 of a single frame point in the historical frame point cluster onto the horizontal coordinate plane, onto the horizontal coordinate plane. Specifically, it is the ratio of the length of the projected line segment Q1'P2 to the length of line segment P1P2, |Q1'-P2| / |P1-P2|, and the ratio of the length of line segment Q1P2' to the length of line segment Q1Q2, |Q1-P2'| / |Q1-Q2|, where Q1' is the projection point of Q1 on line segment P1P2, and P2' is the projection point of P2 on line segment Q1Q2.

[0060] In this embodiment, the overlap threshold can be any value in the range of 0.3-0.7, and optionally, the overlap threshold is 0.5.

[0061] If the overlap between the current frame point cluster and the historical frame point cluster is greater than the overlap threshold, it can be ensured that the current frame point cluster and the historical frame point cluster often belong to the grounding contour line of the same static obstacle.

[0062] exist Figure 1 In the embodiments shown, the static obstacle grounding contour multi-frame fusion method of this application further includes: S105, multi-frame fusion process.

[0063] In the multi-frame fusion process represented by S105, all single-frame points matched to any point cluster class during the single-frame point cluster matching process are transformed into the global Cartesian coordinate system and rasterized around them to obtain a temporary map. If the total number of times the corresponding single-frame point of the environmental image in a grid of the temporary map has appeared is greater than the single-frame point appearance threshold, the information of the grid is stored in the stable map.

[0064] Here, the stable map is the final result of the multi-frame fusion process represented by S105.

[0065] In one embodiment of this implementation, the threshold for the number of occurrences of a single frame point can be any natural number from 3 to 6, and optionally, the threshold for the number of occurrences of a single frame point can be 4.

[0066] Only when the total number of times a corresponding single-frame point in a grid of a temporary map appears in multiple consecutive frames of environmental images exceeds the single-frame point occurrence threshold is the information of that grid stored in the stable map. This prevents temporary dynamic obstacle images or images that are not obstacles from being mistaken for static obstacles, ensuring the accuracy of obstacle identification and thus ensuring the accuracy of the ground contour lines of static obstacles.

[0067] By storing only the statistical average position of multiple single-frame points appearing in the raster, instead of storing the position information of multiple single-frame points, the statistical average position can be updated in real time without storing all single-frame points, thereby reducing the amount of computation and reducing excessive storage space requirements.

[0068] In one embodiment of this implementation, the grid information may include: the global Cartesian coordinates of the grid, the statistical average position of a single frame point appearing in the grid, and the total number of times a single frame point appears in the grid.

[0069] As an example of this embodiment, such as Figure 3 As shown, the statistical average position can be the average position of the geometric center of a single frame point within the corresponding grid.

[0070] exist Figure 3 In a grid, when a single frame point falls into a certain grid for N consecutive frames, that grid is designated as the active grid. For example, Figure 3 Solid dots in the middle grid represent points in a single frame, while hollow dots in the upper right and lower right grids represent the average position of the geometric center of multiple solid dots in that grid. That is, the position is represented by the coordinates obtained by averaging the geometric center coordinates of all solid dots in that grid.

[0071] In one embodiment of this implementation, the multi-frame fusion process represented by S105 may further include a map update process, that is, for the current frame environment image, it is determined whether the proportion of all corresponding single frame points acquired during the current frame acquisition process falling into any grid already stored in the stable map is greater than the hit rate threshold. If it is greater than the hit rate threshold, all corresponding single frame points of the current frame image are added to the stable map; otherwise, only the information of the corresponding grid in the temporary map of the corresponding single frame point is updated.

[0072] In this embodiment, "hitting" refers to using each new environmental image frame and the current frame environmental image, and all the corresponding single-frame points acquired during the current frame acquisition process shown in S102, to "hit" all the grids already stored in the stable map. Among all the single-frame points in the current frame environmental image, "hitting" refers to the proportion of points that fall into any grid already stored in the stable map, out of all the single-frame points in the current frame environmental image; this is the "hit rate." The hit rate threshold can be any value within the range of 65% to 100%, optionally, it can be 70%, 75%, 80%, or 85%.

[0073] In this embodiment, when the hit rate is greater than the hit rate threshold, it is determined that the obstacle represented by the single frame point in the current frame has been stably present in the environmental images of multiple frames before this time.

[0074] Otherwise, if the hit rate is not greater than the hit rate threshold, it is determined that the obstacle represented by the single frame point in the current frame did not exist stably in the environmental images of the previous multiple frames. Therefore, only the information of the corresponding grid of these single frame points in the temporary map is updated.

[0075] In one embodiment of this implementation, each grid cell can use a hash table to calculate a key value, thereby representing temporary and stable maps using a hash table.

[0076] Furthermore, as an example, the Cartesian coordinates of the active raster can be used as the key value of the hash table. For instance, the key value could be the sum of the x-coordinate and y-coordinate of the corresponding active raster. In this way, the existence of Cartesian coordinates for a specific active raster can be queried among several key values ​​in the hash table. If not found, it means that the relevant information for that active raster was not previously stored in the temporary map and / or stable map. In this case, the Cartesian coordinates of the active raster, the statistical average position associated with each active raster's Cartesian coordinates, and the number of occurrences of a single-frame point associated with each active raster's Cartesian coordinates are added to the hash table. If found, it means that the relevant information for that active raster was previously stored in the stable map. In this case, only the corresponding statistical average position and the corresponding number of occurrences of a single-frame point are updated in the hash table.

[0077] In this embodiment, by utilizing temporary and stable maps, such as hash tables, and by activating the coordinate information of the grid, a rapid query of the temporary and stable maps can be achieved to determine whether it is necessary to add new information about multi-frame fusion points or update information about historical multi-frame fusion points. Here, multi-frame fusion points refer to the points corresponding to the statistically averaged positions obtained through the aforementioned static obstacle grounding contour multi-frame fusion scheme of this application. This significantly reduces the algorithm's time complexity, making it nearly linear. Therefore, the computation speed of this application's scheme is fast, making it highly suitable for autonomous driving applications with high real-time requirements.

[0078] Figure 4 The diagram shows a schematic representation of the constituent modules of a specific embodiment of the static obstacle grounding contour multi-frame fusion device of this application.

[0079] In this specific embodiment, the static obstacle grounding contour line multi-frame fusion device includes: a preprocessing module 401, which extracts obstacle contour points for each frame of environmental image captured by cameras deployed around the vehicle, and filters out points outside a predetermined geometric range to obtain filtered obstacle contour points; a single-frame point acquisition module 402, which transforms the filtered obstacle contour points corresponding to each frame of environmental image into a polar coordinate system with the center of the vehicle as the origin, and retains the point closest to the origin of the polar coordinate system among all filtered obstacle contour points with the same angular coordinates as a single-frame point; and a single-frame point clustering module 403, which sorts all single-frame points in order of angular coordinate values, and for each single-frame point, takes its nearest other single-frame point, calculates the angular coordinate difference and relative distance between the two single-frame points, and if the angular coordinate difference is less than the angular coordinate difference threshold... If the distance between two single-frame points is less than the relative distance threshold, then the two single-frame points are clustered into the same single-frame point cluster. The single-frame point cluster matching module 404 traverses the current frame point cluster belonging to the current frame environment image and the historical frame point cluster belonging to the adjacent historical frame environment image. If the distance between the current frame point cluster and the historical frame point cluster is less than the first distance threshold, and the degree of overlap between the projection of the current frame point cluster and the historical frame point cluster is greater than the overlap degree threshold, then the current frame point cluster and the historical frame point cluster are matched into the same point cluster class. The multi-frame fusion module 405 converts all single-frame points matched into any point cluster class during the single-frame point cluster matching process into the global Cartesian coordinate system and rasterizes them around them to obtain a temporary map. If the total number of times the corresponding single-frame point of the consecutive multi-frame environment image has appeared in a grid of the temporary map is greater than the single-frame point occurrence frequency threshold, then the information of the grid is stored in the stable map.

[0080] In this specific embodiment, the preprocessing module 401, the single-frame point acquisition module 402, the single-frame point clustering module 403, the single-frame point cluster matching module 404, and the multi-frame fusion module 405 can be specifically executed. Figure 1 The preprocessing process S101, single-frame point acquisition process S102, single-frame point clustering process S103, single-frame point cluster matching process S104, and multi-frame fusion process S105 of the multi-frame fusion method for static obstacle grounding contour lines shown above, as described in the specific implementations, embodiments, and examples, can achieve the following: Figure 1 The technical effects achieved by the corresponding processes described in the above-described specific implementation methods, examples, and instances of the static obstacle grounding contour multi-frame fusion method are illustrated.

[0081] In one specific embodiment of this application, a computer-readable storage medium is provided, which stores computer instructions, wherein the computer instructions are operated to perform the static obstacle grounding contour multi-frame fusion method described in any of the above embodiments, examples, and instances. The computer instructions stored in the storage medium may be stored directly in hardware, stored in a software module executed by a processor, or stored in a combination of both.

[0082] Software modules may reside in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium that can be used to store computer instructions. Typically, the storage medium is under the control of the processor, allowing the processor to read information from and write information to the storage medium.

[0083] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, but alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, the storage medium and processor can be integrated. The processor and storage medium can reside in an ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium can reside as discrete components in the user terminal.

[0084] In one specific embodiment of this application, a computer device is provided, which includes a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the static obstacle grounding contour multi-frame fusion method described in any of the above embodiments, examples, and instances.

[0085] In the embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the technical objectives of this application, depending on actual needs.

[0087] The above are merely embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A static obstacle ground wheel profile multi-frame fusion method, characterized in that, Comprising: a pre-processing procedure, for each frame of environment image captured by a camera deployed around a vehicle, extracting obstacle contour points, and filtering out points that are out of a predetermined geometric range, to obtain filtered obstacle contour points; a single-frame point obtaining procedure, transforming the filtered obstacle contour points corresponding to each frame of environment image into a polar coordinate system with the center of the vehicle as the origin, and keeping the point closest to the origin among all the filtered obstacle contour points with the same angle coordinate as the single-frame point; a single-frame point clustering procedure, after sorting all the single-frame points in order of angle coordinate value, for each single-frame point, taking its nearest single-frame point, calculating the angle coordinate difference and the relative distance between the two single-frame points, if the angle coordinate difference is less than an angle coordinate difference threshold and the relative distance is less than a relative distance threshold, clustering the two single-frame points into the same single-frame point cluster; a single-frame point cluster matching procedure, traversing the current-frame point cluster belonging to the current frame of environment image and the historical-frame point cluster belonging to the adjacent historical frame of environment image, if the distance between the current-frame point cluster and the historical-frame point cluster is less than a first distance threshold, and the degree of overlap of the projection of the current-frame point cluster and the historical-frame point cluster onto each other is greater than an overlap degree threshold, matching the current-frame point cluster and the historical-frame point cluster into the same point cluster class; and a multi-frame fusion procedure, transforming all the single-frame points matched into any point cluster class in the single-frame point cluster matching procedure into a global Cartesian coordinate system, and rasterizing around it to obtain a temporary map, if the total number of times that the corresponding single-frame points of consecutive multi-frames of environment image have appeared in one grid of the temporary map is greater than a single-frame point appearance number threshold, storing the information of the grid to a stable map.

2. The static obstacle ground wheel profile multi-frame fusion method according to claim 1, characterized in that, The information of the grid includes: the global Cartesian coordinate information of the grid, the statistical average position of the single-frame points appearing in the grid, and the total number of times that the single-frame points appear in the grid.

3. The static obstacle ground contour line multi-frame fusion method according to claim 2, wherein: the statistical average position is the geometric center average position of the single-frame points in the corresponding grid.

4. The static obstacle ground wheel profile multi-frame fusion method according to claim 1, characterized in that, The multi-frame fusion procedure further comprises: a map updating procedure, for the current frame of environment image, judging whether the proportion of all the single-frame points obtained in the current frame obtaining procedure falling into any grid already stored in the stable map is greater than a hit rate threshold, if it is greater than the hit rate threshold, adding all the single-frame points corresponding to the current frame of image to the stable map, otherwise, only updating the information of the corresponding single-frame points appearing in the corresponding grid of the temporary map.

5. The static obstacle ground contour line multi-frame fusion method according to claim 1, wherein: the distance between the current-frame point cluster and the historical-frame point cluster is the horizontal coordinate distance of the current-frame point cluster and the historical-frame point cluster in the Cartesian coordinate system; and / or The degree of coincidence of the mutual projection of the current frame point cluster and the historical frame point cluster is the degree of coincidence between the horizontal coordinate ranges covered by the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system.

6. The static obstacle ground wheel profile multi-frame fusion method according to claim 5, characterized in that, The horizontal coordinate distance of the current frame point cluster and the historical frame point cluster in the Cartesian coordinate system is the distance between the midpoints of the lines connecting the two endpoints of the projection points of the single frame points in the current frame point cluster on the horizontal coordinate plane and the projection points of the single frame points in the historical frame point cluster on the horizontal coordinate plane.

7. The static obstacle ground wheel profile multi-frame fusion method according to claim 1, characterized in that, The single frame point clustering process further comprises: A single frame point cluster removal process removes the single frame point cluster containing a number of single frame points less than a single frame point number threshold.

8. The static obstacle ground wheel profile multi-frame fusion method according to claim 1, characterized in that, The single frame point acquisition process further comprises: A distance smoothing process divides the single frame points into a plurality of adjacent groups at equal intervals according to the corresponding angle coordinates, and performs weighted averaging on the distance coordinates of the plurality of single frame points in each group of single frame points by using a Gaussian algorithm.

9. The static obstacle ground wheel profile multi-frame fusion method according to claim 8, characterized in that, The single frame point acquisition process further comprises: A boundary gradient sharpening process further performs weighted averaging on the distance coordinates of the plurality of single frame points in each group of single frame points processed by the distance smoothing process by using a Sobel operator.

10. The static obstacle ground wheel profile multi-frame fusion method according to claim 1, characterized in that, In the preprocessing process, the process of filtering out the points of the obstacle profile points located outside a predetermined geometric range comprises: Among the obstacle profile points, removing the points exceeding the angle threshold of the field of view of the camera, and removing the points having a distance from the camera less than a second distance threshold or greater than a third distance threshold, wherein the second distance threshold is less than the third distance threshold.

11. A static obstacle ground wheel profile multi-frame fusion device, characterized in that, Comprise: A preprocessing module extracts obstacle profile points from each frame of environment image captured by the camera arranged around the vehicle, and filters out the points of the obstacle profile points located outside a predetermined geometric range to obtain filtered obstacle profile points; A single frame point acquisition module converts the filtered obstacle profile points corresponding to each frame of environment image into a polar coordinate system with the center of the vehicle as the origin of the polar coordinate system, and retains the point closest to the origin of the polar coordinate system among all the filtered obstacle profile points having the same angle coordinate as the single frame point; A single frame point clustering module sorts all the single frame points in order of the size of the angle coordinate value, and for each single frame point, takes another single frame point closest to it, calculates the angle coordinate difference and the relative distance difference between the two single frame points, and if the angle coordinate difference is less than an angle coordinate difference threshold and the relative distance difference is less than a relative distance difference threshold, the two single frame points are clustered into the same single frame point cluster. a single-frame point cluster matching module, which traverses the current-frame point clusters belonging to the current-frame environment image and the historical-frame point clusters belonging to the adjacent historical-frame environment image in the single-frame point clusters, and matches the current-frame point cluster and the historical-frame point cluster to the same point cluster class if the distance between the current-frame point cluster and the historical-frame point cluster is less than a first distance threshold, and the coincidence degree of the mutual projection of the current-frame point cluster and the historical-frame point cluster is greater than a coincidence degree threshold; and a multi-frame fusion module, which converts all the single-frame points matched into any of the point cluster classes in the single-frame point cluster matching process to a global Cartesian coordinate system, and rasterizes around the same to obtain a temporary map, and stores the information of a grid in the temporary map to a stable map if the total appearance times of the corresponding single-frame points of the continuous multi-frame environment images in the grid are greater than a single-frame point appearance times threshold.

12. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions, and the computer instructions are operated to perform the static obstacle grounding contour multi-frame fusion method in any one of claims 1-10.

13. A computer device, comprising: comprise: a memory that stores computer instructions; and a processor that operates the computer instructions to perform the static obstacle grounding contour multi-frame fusion method in any one of claims 1-10.

Images