Method, device and electronic equipment for locating a distant target

Abstract

The application provides a positioning method and device for a long-distance target and electronic equipment, and belongs to the technical field of target identification and positioning. In the positioning method, a sparse point cloud that cannot be identified at a long distance is utilized, the sparse point cloud is screened, a screened sparse point cloud is obtained, the longitude and latitude coordinates of the screened sparse point cloud and the real-world longitude and latitude coordinates of a to-be-determined region are determined, target sparse point clouds belonging to the to-be-determined region in the screened sparse point cloud are determined, the target sparse point clouds are part of the point cloud of the long-distance target, the longitude and latitude coordinates of the frontmost point cloud in the target sparse point cloud are taken as the outer contour position of the long-distance target, and the outer contour of the long-distance target can be accurately positioned.

Description

Technical Field

[0001] This invention relates to the technical field of target identification and positioning, and in particular to a method, apparatus and electronic device for locating distant targets. Background Technology

[0002] With the development of technology, laser-visual fusion algorithms for intersection vehicle perception are increasingly being applied in traffic intersection scenarios. Perceiving targets on the road can help traffic light control systems more accurately control signal changes based on information such as the position, number, and speed of vehicles on the road; it can also analyze the traffic efficiency and safety of intersections, enhancing traffic rationality; and it can help traffic management departments promptly detect and handle traffic accidents and congestion, thereby ensuring road traffic safety.

[0003] While the target perception capability of lidar-based radar-visual fusion is richer and more accurate than that of millimeter-wave radar-based radar-visual fusion, its perception range is limited and the distance is shorter. This is because lidar-visual fusion requires matching the target perceived by the radar with the target perceived by the camera. The distance depends on the field of view of the lidar and the camera. The farther away from the lidar, the sparser the scanned point cloud becomes. Therefore, the lidar target perception algorithm cannot perceive targets at a distance.

[0004] Adding radar-visual fusion sensing devices at different distances along the roadside can compensate for the lack of long-range sensing, but it increases costs, requires higher synchronization and consistency between devices, and communication between devices is prone to delays and interruptions.

[0005] In summary, how to achieve the identification and positioning of distant targets without adding radar-visual fusion sensing devices has become an urgent technical problem to be solved. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for locating distant targets, so as to alleviate the technical problem that the existing radar-visual fusion technology cannot identify and locate distant targets.

[0007] In a first aspect, embodiments of the present invention provide a method for locating a distant target, comprising:

[0008] The image to be identified is obtained by the image acquisition device from the road image acquisition, and the target detection is performed on the image to be identified to obtain the detection box of each target in the image to be identified;

[0009] The preset proportion area above the bottom border of the detection box of the target is taken as the undetermined area of ​​the corresponding target, and the pixel coordinates of the undetermined area are subjected to perspective transformation to obtain the real-world latitude and longitude coordinates of the undetermined area.

[0010] The system acquires point clouds obtained by a lidar that is synchronized with the image acquisition device to detect road areas, and identifies the point clouds to obtain the positioning information of nearby targets and sparse point clouds that cannot be identified at long distances.

[0011] The sparse point cloud is filtered to obtain a filtered sparse point cloud, and the latitude and longitude coordinates of the filtered sparse point cloud are calculated.

[0012] The target sparse point cloud belonging to the undetermined region is determined based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the undetermined region.

[0013] The latitude and longitude coordinates of the foremost point cloud in the sparse point cloud of the target are used as the outer contour position of the distant target.

[0014] Furthermore, the preset proportion area is the proportion of the area from the bottom border of the target's detection box to the vehicle body, calculated based on the height and angle information of the image acquisition device.

[0015] Furthermore, a perspective transformation is performed on the pixel coordinates of the region to be determined, including:

[0016] Calculate the mapping matrix of the pixel coordinates of the road image projected onto the real-world latitude and longitude coordinate system;

[0017] The pixel coordinates of the region to be determined are transformed by perspective transformation based on the mapping matrix to obtain the real-world latitude and longitude coordinates of the region to be determined.

[0018] Furthermore, the sparse point cloud is filtered, including:

[0019] The sparse point cloud is selected from the sparse point cloud to obtain a sparse point cloud within a preset height range from the ground, and then the selected sparse point cloud is obtained.

[0020] Further, calculating the latitude and longitude coordinates of the filtered sparse point cloud includes:

[0021] The latitude and longitude coordinates of the filtered sparse point cloud are calculated based on the latitude and longitude of the installation location of the lidar and the included northward angle.

[0022] Furthermore, the method also includes:

[0023] A perspective transformation is performed on the pixel coordinates within the detection box of the target to obtain the detection box region in the real-world latitude and longitude coordinate system.

[0024] Semantic segmentation is performed on the detection box region to obtain the outline of the target;

[0025] Multiple simulated point clouds are generated based on the described contour;

[0026] The type of the target detected by the target is obtained, and then the size of the target is obtained. The simulated point cloud is scaled according to the size to obtain a scaled simulated point cloud.

[0027] The scaled simulated point cloud is stitched together with the target sparse point cloud according to the specified size to obtain the stitched point cloud.

[0028] The stitched point cloud is identified to obtain the corresponding target location information, wherein the target location information includes: the target's bounding box, the target's center point position, and the target's heading angle.

[0029] Furthermore, multiple simulated point clouds are generated based on the contour, including:

[0030] Multiple simulated point clouds are generated within the outline using a generative network or random sampling method.

[0031] Secondly, embodiments of the present invention also provide a positioning device for a long-distance target, comprising:

[0032] The target detection unit is used to acquire the image to be identified obtained by the image acquisition device from the road image acquisition, and to perform target detection on the image to be identified to obtain the detection box of each target in the image to be identified;

[0033] The perspective transformation unit is used to take the preset proportion area above the bottom border of the detection box of the target as the undetermined area of ​​the corresponding target, and to perform perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area.

[0034] The point cloud recognition unit is used to acquire point clouds obtained by the lidar that is synchronized with the image acquisition device to detect the road area, and to recognize the point clouds to obtain the positioning information of near targets and sparse point clouds that cannot be recognized at long distances.

[0035] The filtering and calculation unit is used to filter the sparse point cloud to obtain the filtered sparse point cloud, and calculate the latitude and longitude coordinates of the filtered sparse point cloud.

[0036] The determining unit is used to determine the target sparse point cloud belonging to the region to be determined in the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the region to be determined.

[0037] The setting unit is used to take the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud as the outer contour position of the distant target.

[0038] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0039] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0040] In this embodiment of the invention, a method for locating distant targets is provided, comprising: acquiring an image to be identified obtained by an image acquisition device capturing images of a road, and performing target detection on the image to be identified to obtain detection boxes for each target in the image to be identified; taking a predetermined proportion area above the lower border of the target's detection box as the undetermined area of ​​the corresponding target, and performing perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area; acquiring a point cloud obtained by a LiDAR synchronized with the image acquisition device detecting the road area, and identifying the point cloud to obtain the location information of near-distance targets and sparse point clouds that cannot be identified at a distance; filtering the sparse point cloud to obtain filtered sparse point cloud, and calculating the latitude and longitude coordinates of the filtered sparse point cloud; determining the target sparse point cloud belonging to the undetermined area in the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the undetermined area; and taking the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud as the outer contour position of the distant target. As described above, the long-distance target localization method of the present invention can utilize sparse point clouds that are unrecognizable at long distances, filter these sparse point clouds to obtain filtered sparse point clouds, and then determine the target sparse point clouds belonging to the undetermined area based on the latitude and longitude coordinates of the filtered sparse point clouds and the real-world latitude and longitude coordinates of the undetermined area. This part of the target sparse point cloud is the accurate part of the point cloud of the long-distance target. Furthermore, the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud are used as the outer contour position of the long-distance target, thereby achieving accurate localization of the outer contour of the long-distance target and alleviating the technical problem that existing radar-visual fusion technology cannot identify and locate long-distance targets. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a method for locating a distant target according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of an image to be identified carrying a detection frame, provided in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram showing the relative positions of the region to be determined and the detection frame provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of another undetermined region and a detection frame region provided in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the effective identification area and the remote sensing area provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the fusion of the region to be determined and the filtered sparse point cloud, provided as an embodiment of the present invention.

[0048] Figure 7 A schematic diagram of the perception reconstruction and recognition process provided in an embodiment of the present invention;

[0049] Figure 8 A schematic diagram illustrating the process of a method for locating a distant target provided in an embodiment of the present invention;

[0050] Figure 9 A schematic diagram of a long-distance target positioning device provided in an embodiment of the present invention;

[0051] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Existing radar-visual fusion technology cannot identify and locate distant targets.

[0054] Based on this, the method for locating distant targets in this invention can utilize sparse point clouds that are not easily identifiable at a distance, filter these sparse point clouds to obtain filtered sparse point clouds, and then determine the target sparse point clouds belonging to the undetermined area based on the latitude and longitude coordinates of the filtered sparse point clouds and the real-world latitude and longitude coordinates of the undetermined area. This part of the target sparse point cloud is the accurate part of the point cloud of the distant target. Then, the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud are used as the outer contour position of the distant target, thus achieving accurate positioning of the outer contour of the distant target.

[0055] To facilitate understanding of this embodiment, a method for locating a distant target disclosed in this embodiment of the invention will first be described in detail.

[0056] Example 1:

[0057] According to an embodiment of the present invention, an embodiment of a method for locating a distant target is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0058] Figure 1 This is a flowchart of a method for locating a distant target according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0059] Step S102: Obtain the image to be identified obtained by the image acquisition device from the road image acquisition, and perform target detection on the image to be identified to obtain the detection box of each target in the image to be identified;

[0060] In this embodiment of the invention, target detection of the image to be recognized specifically involves using a target detection network model to perform target detection on the image to be recognized.

[0061] The target detection network model is pre-trained. Specifically, road image training samples are obtained by acquiring road images from an intersection image acquisition device (such as a camera). Distant vehicle targets in the road image training samples are labeled. Then, the original target detection network model is trained using the labeled road image training samples to obtain a target detection network model (such as YOLOv4, YOLOv5, etc.). This target detection network model can then identify distant vehicle targets and vehicle types.

[0062] The target in the above target detection can specifically be a distant vehicle target in the image to be identified, thereby obtaining the detection box of the distant vehicle target and the type of the distant vehicle target.

[0063] Step S104: Take the preset proportion area above the bottom border of the target's detection box as the target's undetermined area, and perform perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area.

[0064] Specifically, based on the fundamental conditions for homography and the distance relationship between objects, objects of the same length appear smaller at a distance than at a closer distance. Therefore, perspective transformation only applies to the ground plane; targets not on the ground plane, such as buildings on either side or the sky, cannot be mapped. The mapping of distant vehicles in a bird's-eye view is elongated, becoming more elongated the further away they are. The recognition area of ​​a vehicle of the same size will be larger in the distant mapping. Traditional image mapping localization uses the latitude and longitude of the pixel at the bottom center of the target's detection box as the vehicle's latitude and longitude. Since the positions of adjacent pixels at a distance differ greatly, even a slight shrinking or enlarging of the detection box by one pixel will cause drastic fluctuations, resulting in inaccurate or even impossible vehicle localization due to jumping and hopping. Figure 2 As shown, distant vehicles appear elongated after mapping, due to the increased distance represented by a single pixel and the fact that the vehicle is perpendicular to the road.

[0065] Since perspective transformation is planar, the coordinates of road pixels within the target's detection box are accurate to the coordinates of pixels on the vehicle body. Therefore, this invention proposes to use a preset proportion area above the lower boundary (i.e., the lower border) [c,d] of the detection box (specifically, the area between the lower border of the detection box and the vehicle body, which is actually a portion of the pixels on the road) as the vehicle's front position area, i.e., the undetermined area of ​​the target (which is the undetermined area of ​​the vehicle's accurate position).

[0066] Considering that the image between the lower boundary and the vehicle body is on a planar road, which meets the condition that inverse perspective transformation only applies to the ground plane, according to the perspective of the image acquisition device, this invention defines the detection box area one-fifth of the area above the lower boundary (i.e., the preset proportion area; the value of this one-fifth is not limited here but calculated based on relevant information) as the area to be determined for the vehicle's outer contour. Figure 3 The pixel coordinates [[x1, y1], [x2, y2], [x3, y3], [x4, y4]] of the region to be determined (the specific process of perspective transformation will be described in detail below, and will not be repeated here) are subjected to perspective transformation to obtain the real-world latitude and longitude coordinates of the region to be determined.

[0067] Figure 4To map the pixel coordinates of the region to be determined onto a real-world latitude and longitude coordinate system image through perspective transformation, it can be seen that since the vehicle is three-dimensional and the ground is two-dimensional, the ground can be projected normally, while the vehicle is elongated and its size is distorted. Since the region to be determined is the area of ​​the ground between the bottom edge of the vehicle's detection frame and the vehicle body, the vehicle's outer contour falls within this region. Compared to mapping the detection frame area using the vehicle body's detection frame, the region to be determined narrows down the possible location of the vehicle, improving accuracy.

[0068] Step S106: Obtain the point cloud obtained by the lidar that is synchronized with the image acquisition device to detect the road area, and identify the point cloud to obtain the positioning information of the near target and the sparse point cloud that cannot be identified at a distance.

[0069] Specifically, in order to ensure that the information sensed by the image acquisition device and the lidar is at the same time and to guarantee the consistency and synchronization of the data, it is necessary to synchronize the roadside lidar and the corresponding intersection camera (i.e., the image acquisition device) in time.

[0070] The roadside lidar is divided into road areas, and useless point clouds in non-road areas are filtered out. Only the point clouds in the road areas are acquired (to reduce the amount of computation). LiDAR point cloud recognition algorithms (such as PVRCNN, CenterPoint, etc.) are used to identify the point clouds, and the effective recognition range (to identify the positioning information of nearby targets) and the effective illumination range are calculated. In this way, the sparse point cloud areas that cannot be identified (i.e., sparse point clouds that cannot be identified at long distances) are delineated.

[0071] The following is a detailed explanation:

[0072] First, the distance at which the roadside lidar can identify vehicles on the road is measured to determine the effective identification area. Figure 5 1) Effectively identifies rectangular areas on roads outside the designated area (such as...). Figure 5 2) is the edge region of the radar's sensing range. The point clouds in this region are sparse and far apart. When a vehicle passes through this region, it can only be illuminated by sporadic laser point clouds. The laser radar point cloud recognition algorithm (i.e., the laser radar target perception algorithm) and even the human eye have difficulty distinguishing the vehicle target. Therefore, this region is a remote sensing region where the target cannot be identified but sparse unknown point clouds can still be obtained.

[0073] from Figure 5As can be seen, when the vehicle target is close to or at a medium distance from the radar, the vehicle is covered by a large number of laser lines (the arcs in the figure), and the vehicle body is illuminated by a large number of laser points. The point cloud recognition algorithm can effectively identify the point cloud cluster as a vehicle. When the vehicle target is far away from the radar, that is, in the edge area of ​​the radar's sensing range, the vehicle can only intersect with one laser beam, and only the front of the vehicle can be illuminated by a portion of the point cloud. The point cloud is too sparse, so the point cloud recognition algorithm cannot classify this point cloud cluster.

[0074] The above process can be simply described as follows: a remote sensing area is defined on the roadside where the lidar is located. The remote sensing area is a distance area where the point cloud is so sparse that the point cloud recognition algorithm cannot recognize it or has a low recognition accuracy. The sparse point cloud of the remote sensing area (i.e., the sparse point cloud area) is obtained.

[0075] Step S108: Filter the sparse point cloud to obtain the filtered sparse point cloud, and calculate the latitude and longitude coordinates of the filtered sparse point cloud.

[0076] Specifically, during the screening process, laser point cloud data located above the ground are selected from the sparse point cloud. The screening process will be described in detail below, and will not be repeated here.

[0077] After obtaining the filtered sparse point cloud, the latitude and longitude coordinates of the filtered sparse point cloud are calculated. The calculation process will be described in detail below.

[0078] Step S110: Determine the target sparse point cloud belonging to the region to be determined from the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the region to be determined.

[0079] Specifically, the filtered sparse point cloud is mapped to the undetermined area, and the target sparse point cloud falling within the undetermined area is selected. These target sparse point clouds can then be regarded as point clouds illuminated by the roadside lidar on the vehicle target. The accurate latitude and longitude information of these point clouds is the information of the location of the vehicle target's outer contour.

[0080] This involves fusing the undetermined region with the filtered sparse point cloud (i.e., unknown point cloud coordinates—which cannot be identified by LiDAR): projecting the filtered sparse point cloud mapped to world coordinates and the undetermined region of the vehicle's position in the image to the world coordinate range, such as... Figure 6As shown, the undetermined area of ​​the vehicle's position in the image uses the mapping data of ground pixels under the vehicle body. In the case of long distance, the distance covered by each pixel is much greater than the distance covered at close distance, so the range of world latitude and longitude is large and cannot provide an accurate position. The sparse point cloud of the target falling in the undetermined area is the point cloud of the outer contour of the vehicle body hit by the lidar. Taking the latitude and longitude coordinates [a,b] of the foremost point cloud is the accurate position of the outer contour of the distant target that was not detected in reality.

[0081] Step S112: Use the latitude and longitude coordinates of the foremost point cloud in the sparse point cloud of the target as the outer contour position of the distant target.

[0082] In this embodiment of the invention, a method for locating distant targets is provided, comprising: acquiring an image to be identified obtained by an image acquisition device capturing images of a road, and performing target detection on the image to be identified to obtain detection boxes for each target in the image to be identified; taking a predetermined proportion area above the lower border of the target's detection box as the undetermined area of ​​the corresponding target, and performing perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area; acquiring a point cloud obtained by a LiDAR synchronized with the image acquisition device detecting the road area, and identifying the point cloud to obtain the location information of near-distance targets and sparse point clouds that cannot be identified at a distance; filtering the sparse point cloud to obtain filtered sparse point cloud, and calculating the latitude and longitude coordinates of the filtered sparse point cloud; determining the target sparse point cloud belonging to the undetermined area in the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the undetermined area; and taking the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud as the outer contour position of the distant target. As described above, the long-distance target localization method of the present invention can utilize sparse point clouds that are unrecognizable at long distances, filter these sparse point clouds to obtain filtered sparse point clouds, and then determine the target sparse point clouds belonging to the undetermined area based on the latitude and longitude coordinates of the filtered sparse point clouds and the real-world latitude and longitude coordinates of the undetermined area. This part of the target sparse point cloud is the accurate part of the point cloud of the long-distance target. Furthermore, the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud are used as the outer contour position of the long-distance target, thereby achieving accurate localization of the outer contour of the long-distance target and alleviating the technical problem that existing radar-visual fusion technology cannot identify and locate long-distance targets.

[0083] The above provides a brief overview of the method for locating distant targets according to the present invention. The specific details involved are described in detail below.

[0084] In an optional embodiment of the present invention, the preset proportion area is the proportion of the area from the lower border of the target detection box to the vehicle body, calculated based on the height and angle information of the image acquisition device.

[0085] Specifically, the area from the bottom border of the detection frame to the vehicle body is calculated based on information such as the height and angle of the image acquisition device, thereby obtaining the target's undetermined area.

[0086] In an optional embodiment of the present invention, performing perspective transformation on the pixel coordinates of the region to be determined specifically includes the following steps:

[0087] (1) Calculate the mapping matrix of the pixel coordinates of the road image to the real-world latitude and longitude coordinate system;

[0088] Specifically, the road area is selected based on the road information in the camera image, the ground pixel positions of the four endpoints of the road area are obtained, and the latitude and longitude coordinates of the real world corresponding to these four pixels are obtained through RTK, high-precision map positioning, on-site measurement and other means, and the mapping matrix is ​​calculated.

[0089] Find the mapping matrix of the road perceived by the camera:

[0090]

[0091] The specific operation is as follows: Obtain the latitude and longitude coordinates in the real world of four pixels (the ground pixel positions at the four endpoints of the road area above) [[x1, y1], [x2, y2], [x3, y3], [x4, y4]] on the road image, both near and far (x1, y2, y3, y4, y4). r y1 r ], [x2 r y2 r ],[x3 r y3 r ],[x4 r y4 r The mapping matrix can be obtained according to the formula. This allows us to calculate the projection of each pixel in the camera image onto the real world:

[0092] (2) Perform perspective transformation on the pixel coordinates of the region to be determined based on the mapping matrix to obtain the real-world latitude and longitude coordinates of the region to be determined.

[0093] Specifically, will Figure 3 The pixel coordinates [[x1, y1], [x2, y2], [x3, y3], [x4, y4]] and the mapping matrix of the region to be determined (small rectangle) Multiplying them together yields the real-world latitude and longitude coordinates of the region to be determined.

[0094] In an optional embodiment of the present invention, filtering of sparse point clouds includes:

[0095] The sparse point cloud is filtered to obtain a sparse point cloud within a preset height range above the ground, and then the filtered sparse point cloud is obtained. The preset height range is determined according to the height of the target.

[0096] Specifically, laser point cloud data located above the ground are selected within the remote sensing area (i.e., sparse point cloud). In order to exclude the influence of roadside facilities such as road signs or power lines, only sparse point clouds with a height of less than 3 meters and more than 0.1 meters are selected as the sparse point cloud.

[0097] In other words, the point cloud on the ground (with a height less than a certain value) is filtered out, and the remaining point cloud is re-analyzed. Considering that the height of road targets is generally less than 3 meters, the point cloud above 3 meters is filtered out. After excluding the influence of factors such as signs and power lines, the remaining point cloud (i.e., the sparse point cloud after filtering) can be regarded as the point cloud illuminating the outer surface of road objects and other useless point clouds. The latitude and longitude information corresponding to the location of these point clouds is calculated.

[0098] In an optional embodiment of the present invention, calculating the latitude and longitude coordinates of the filtered sparse point cloud specifically includes the following steps:

[0099] The latitude and longitude coordinates of the sparse point cloud were calculated and filtered based on the latitude and longitude of the lidar installation location and the northward angle.

[0100] Specifically, the unit of the lidar point cloud is meters, and the set of the acquired unknown point cloud (i.e., the sparse point cloud after filtering) is [[x1, y1, z1], [x2, y2, z2]...[x n y n , z n Based on the latitude and longitude [lon1, lat1] of the radar installation location and the northward angle α, the latitude and longitude of the filtered sparse point cloud are calculated and mapped to latitude and longitude coordinates [[x1] loc y1 loc ], [x2 loc y2 loc ...[x n loc y n loc ]]superior:

[0101]

[0102] x n loc =0.00001141*x′ n +lon1

[0103] y n loc =0.00000895*y′ n +lat1

[0104] z n loc =0

[0105] In an alternative embodiment of the present invention, reference is made to... Figure 7 The method also includes:

[0106] (1) Perform perspective transformation on the pixel coordinates within the target detection box to obtain the detection box region in the real-world latitude and longitude coordinate system.

[0107] (2) Perform semantic segmentation on the detection box region to obtain the outline of the target;

[0108] (3) Generate multiple simulated point clouds based on the contour;

[0109] Specifically, multiple simulated point clouds are generated within the contour using a generative network or random sampling method. The purpose of these simulated point clouds is to simulate and predict the distribution of the real point cloud at the vehicle's location, reconstruct and enhance the morphology of useless point clouds (i.e., target sparse point clouds—point clouds that radar cannot recognize), and supplement the missing information of distant point clouds.

[0110] (4) Obtain the type of the target detected by the target, then obtain the size of the target, and scale the simulated point cloud according to the size to obtain the scaled simulated point cloud;

[0111] Specifically, the above types include: cars, SUVs, buses, etc. Based on the type, the size of the target can be obtained, and the approximate dimensions of the vehicle target [l, w, h] can be obtained.

[0112] Since the simulated points are generated in the semantic segmentation region of the image, their positions are inaccurate, just like in image mapping. Therefore, the simulated point cloud needs to be scaled according to the vehicle size to be within the range of the real size.

[0113] (5) Based on the size, the scaled simulated point cloud is stitched together with the target sparse point cloud to obtain the stitched point cloud;

[0114] Specifically, the true unknowable point cloud (i.e., the target sparse point cloud) consists of the laser points on the closest side of the target's outline to the radar. Therefore, the true unknowable point cloud is located outside the target. During stitching, the true point cloud (i.e., the target sparse point cloud) should be near the radar, and the simulated point cloud should not be closer to the sensing device than the true point cloud. After stitching, adjustments are made to ensure that the position of the center point of the simulated point cloud cluster relative to the true point cloud is roughly equivalent to the center point of the approximate size obtained based on the vehicle type, thus completing the information completion of the sparse point cloud.

[0115] (6) Identify the stitched point cloud to obtain the corresponding target location information, wherein the target location information includes: the target's bounding box, the target's center point position and the target's heading angle.

[0116] Specifically, the stitched point cloud information is rich and indistinguishable from near-range point clouds. Applying this information to a lidar target perception algorithm yields a bounding box for the target object, with its center point representing the target's center position and its orientation representing the target's heading angle. This enables the perception of targets at the far edge of the radar range. Combined with vehicle information identified at other ranges (medium and near range), it can support tracking, safety warnings, and driving analysis.

[0117] The above process can be summarized as follows: Based on the lane where the target sparse point cloud is located, the vehicle type and shape (front and side) detected in the image, and the vehicle size and external contour point cloud position information obtained based on the vehicle type, the target sparse point cloud is stitched together, reconstructed, and identified. The center position and heading angle of the vehicle can be calculated. This completes the function of roadside perception equipment in identifying and locating vehicles at a distance.

[0118] To ensure a unified global coordinate system and facilitate the collaborative fusion of sensing devices, this invention maps all targets to the same standard latitude and longitude coordinate system. The lidar utilizes sparse, unidentifiable laser points (unknown point clouds) at the edge of its sensing range to calculate target positions. The camera identifies distant targets in the image and performs coarse mapping to determine whether the unknown point clouds at the radar's sensing edge represent reflections of laser light from the target object. If the unknown point clouds are identified as traffic target point clouds, then the point clouds are matched, mapped, and reconstructed to obtain the accurate target position.

[0119] The process of the long-distance target positioning method of the present invention is as follows: Figure 8 As shown.

[0120] Traditional laser-visual fusion technology is ineffective in locating and identifying distant targets. Therefore, this invention proposes a novel method for long-distance target perception, aiming to overcome the shortcomings of existing technologies. This method includes utilizing sparse, useless point clouds from lidar to provide location information, and employing laser-visual fusion and image perspective transformation to complete the identification and localization of distant targets.

[0121] Furthermore, this invention proposes a series of data information improvement methods, such as filtering the image localization area using the planar region between the lower border of the detection frame and the vehicle body, combining distant point cloud with height information, and regional information. Compared to traditional radar-visual fusion technology, this invention solves the problems of roadside lidar's inability to identify vehicles at a distance due to sparse point clouds and large deviations in the position of distant targets during visual perspective transformation localization, thus enabling roadside sensing devices to identify and locate blurred targets at a distance. This method has higher and longer-range localization and recognition accuracy and is applicable to various application scenarios.

[0122] The present invention has the following advantages and effects:

[0123] (1) It can utilize the unusable point cloud information of the perception edge that cannot be detected by the point cloud recognition algorithm. By combining it with the recognition of distant targets in the image, it can filter the point cloud that hits the target in the useless point cloud and turn it into a useful point cloud.

[0124] (2) By combining the location information of distant useless point clouds with images, the shortcomings of the inability to locate distant parts of the image can be compensated, and the long-distance radar target perception function can be realized, thus solving the problem of small radar fusion perception range.

[0125] (3) By mapping two sensing devices to the same coordinate system and reconstructing the missing information, fusion is achieved based on the predicted and simulated position matching. This method is low-cost, effective, and has a high information utilization rate.

[0126] (4) Using the ground pixels from the bottom border of the target detection box in the image to the target vehicle body as the target location information in the visual image is more accurate;

[0127] (5) The approximate area of ​​the vehicle (i.e. the undetermined area of ​​the target) is determined based on the vehicle body to ground data within the target detection box, and then fused with the laser point cloud. The accuracy of the vehicle outline information obtained by using the useless point cloud is high.

[0128] (6) Useless point clouds are generated and reconstructed into virtual vehicle point clouds, and the accurate position and heading angle of the vehicle are identified based on the reconstructed point clouds, resulting in high information utilization.

[0129] Example 2:

[0130] This invention also provides a long-distance target positioning device, which is mainly used to execute the long-distance target positioning method provided in Embodiment 1 of this invention. The long-distance target positioning device provided in this invention will be described in detail below.

[0131] Figure 9 This is a schematic diagram of a long-distance target positioning device according to an embodiment of the present invention, as shown below. Figure 9As shown, the device mainly includes: a target detection unit 10, a perspective transformation unit 20, a point cloud recognition unit 30, a filtering and calculation unit 40, a determination unit 50, and a setting unit 60, wherein:

[0132] The target detection unit is used to acquire the image to be identified obtained by the image acquisition device from the road image acquisition, and to perform target detection on the image to be identified to obtain the detection box of each target in the image to be identified;

[0133] The perspective transformation unit is used to take the preset proportion area above the bottom border of the target's detection box as the target's undetermined area, and perform perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area.

[0134] The point cloud recognition unit is used to acquire point clouds obtained by the lidar that is synchronized with the image acquisition device to detect the road area, and to recognize the point clouds to obtain the positioning information of nearby targets and sparse point clouds that cannot be recognized at long distances.

[0135] The filtering and calculation unit is used to filter the sparse point cloud, obtain the filtered sparse point cloud, and calculate the latitude and longitude coordinates of the filtered sparse point cloud.

[0136] The determination unit is used to determine the target sparse point cloud belonging to the region to be determined in the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the region to be determined.

[0137] The setting unit is used to take the latitude and longitude coordinates of the foremost point cloud in the sparse point cloud of the target as the outer contour position of the distant target.

[0138] In this embodiment of the invention, a device for locating distant targets is provided, comprising: acquiring an image to be identified obtained by an image acquisition device from a road, and performing target detection on the image to be identified to obtain detection boxes for each target in the image to be identified; taking a predetermined proportion area above the lower border of the target's detection box as the undetermined area of ​​the corresponding target, and performing perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area; acquiring a point cloud obtained by a lidar synchronized with the image acquisition device from a road area, and identifying the point cloud to obtain the location information of near-distance targets and sparse point clouds that cannot be identified at long distances; filtering the sparse point cloud to obtain filtered sparse point clouds, and calculating the latitude and longitude coordinates of the filtered sparse point clouds; determining the target sparse point cloud belonging to the undetermined area in the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the undetermined area; and taking the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud as the outer contour position of the distant target. As described above, the long-distance target positioning device of the present invention can utilize sparse point clouds that are unrecognizable at long distances, filter these sparse point clouds to obtain filtered sparse point clouds, and then determine the target sparse point clouds belonging to the undetermined area based on the latitude and longitude coordinates of the filtered sparse point clouds and the real-world latitude and longitude coordinates of the undetermined area. This part of the target sparse point cloud is the accurate part of the point cloud of the long-distance target. Furthermore, the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud are used as the outer contour position of the long-distance target, thereby achieving accurate positioning of the outer contour of the long-distance target and alleviating the technical problem that existing radar-visual fusion technology cannot identify and locate long-distance targets.

[0139] Optionally, the preset proportion area is the proportion of the area from the bottom border of the target's detection box to the vehicle body, calculated based on the height and angle information of the image acquisition device.

[0140] Optionally, the perspective transformation unit is also used to: calculate the mapping matrix of the pixel coordinates of the road image projected onto the real-world latitude and longitude coordinate system; and perform perspective transformation on the pixel coordinates of the region to be determined based on the mapping matrix to obtain the real-world latitude and longitude coordinates of the region to be determined.

[0141] Optionally, the filtering and calculation unit is also used to: filter out sparse point clouds within a preset height range from the ground from the sparse point cloud, and then obtain the filtered sparse point cloud.

[0142] Optionally, the filtering and calculation unit is also used to: calculate the latitude and longitude coordinates of the filtered sparse point cloud based on the latitude and longitude and the northward angle of the lidar installation location.

[0143] Optionally, the device is further configured to: perform perspective transformation on the pixel coordinates within the target's detection box to obtain the detection box region in the real-world latitude and longitude coordinate system; perform semantic segmentation on the detection box region to obtain the target's outline; generate multiple simulated point clouds based on the outline; obtain the type of the target detected, thereby obtaining the target's size, and scale the simulated point cloud according to the size to obtain a scaled simulated point cloud; stitch the scaled simulated point cloud with the target's sparse point cloud according to the size to obtain a stitched point cloud; and identify the stitched point cloud to obtain the corresponding target's location information, wherein the target's location information includes: the target's identification box, the target's center point position, and the target's heading angle.

[0144] Optionally, the device is also used to generate multiple simulated point clouds within the contour using a generative network or random sampling method.

[0145] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0146] like Figure 10 As shown in the figure, an electronic device 600 provided in this application includes: a processor 601, a memory 602 and a bus. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus. The processor 601 executes the machine-readable instructions to perform the steps of the above-described method for locating distant targets.

[0147] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned method for locating distant targets.

[0148] The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 602, and processor 601 reads the information from memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0149] Corresponding to the above-described method for locating distant targets, this application also provides a computer-readable storage medium storing machine-executable instructions. When these machine-executable instructions are invoked and executed by a processor, they cause the processor to perform the steps of the above-described method for locating distant targets.

[0150] The long-range target positioning device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0151] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0152] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0155] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the vehicle marking method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0157] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for locating a distant target, characterized in that, include: The image to be identified is obtained by the image acquisition device from the road image acquisition, and the target detection is performed on the image to be identified to obtain the detection box of each target in the image to be identified; The preset proportion area above the bottom border of the detection box of the target is taken as the undetermined area of ​​the corresponding target, and the pixel coordinates of the undetermined area are subjected to perspective transformation to obtain the real-world latitude and longitude coordinates of the undetermined area. The system acquires point clouds obtained by a lidar that is synchronized with the image acquisition device to detect road areas, and identifies the point clouds to obtain the positioning information of nearby targets and sparse point clouds that cannot be identified at long distances. The sparse point cloud is filtered to obtain a filtered sparse point cloud, and the latitude and longitude coordinates of the filtered sparse point cloud are calculated. The target sparse point cloud belonging to the undetermined region is determined based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the undetermined region. The latitude and longitude coordinates of the foremost point cloud in the sparse point cloud of the target are used as the outer contour position of the distant target; The preset proportion area is the proportion of the area from the bottom border of the target's detection box to the vehicle body, calculated based on the height and angle information of the image acquisition device. The perspective transformation of the pixel coordinates of the region to be determined includes: Calculate the mapping matrix of the pixel coordinates of the road image projected onto the real-world latitude and longitude coordinate system; Based on the mapping matrix, the pixel coordinates of the region to be determined are subjected to perspective transformation to obtain the real-world latitude and longitude coordinates of the region to be determined. The filtering of the sparse point cloud includes: From the sparse point cloud, sparse point clouds within a preset height range above the ground are selected, and then the selected sparse point cloud is obtained. The calculation of the latitude and longitude coordinates of the filtered sparse point cloud includes: The latitude and longitude coordinates of the filtered sparse point cloud are calculated based on the latitude and longitude of the installation location of the lidar and the included northward angle.

2. The method according to claim 1, characterized in that, The method further includes: A perspective transformation is performed on the pixel coordinates within the detection box of the target to obtain the detection box region in the real-world latitude and longitude coordinate system. Semantic segmentation is performed on the detection box region to obtain the outline of the target; Multiple simulated point clouds are generated based on the described contour; The type of the target detected by the target is obtained, and then the size of the target is obtained. The simulated point cloud is scaled according to the size to obtain a scaled simulated point cloud. The scaled simulated point cloud is stitched together with the target sparse point cloud according to the specified size to obtain the stitched point cloud. The stitched point cloud is identified to obtain the corresponding target location information, wherein the target location information includes: the target's bounding box, the target's center point position, and the target's heading angle.

3. The method according to claim 2, characterized in that, Multiple simulated point clouds are generated based on the contour, including: Multiple simulated point clouds are generated within the outline using a generative network or random sampling method.

4. A positioning device for a long-distance target, characterized in that, include: The target detection unit is used to acquire the image to be identified obtained by the image acquisition device from the road image acquisition, and to perform target detection on the image to be identified to obtain the detection box of each target in the image to be identified; The perspective transformation unit is used to take the preset proportion area above the bottom border of the detection box of the target as the undetermined area of ​​the corresponding target, and to perform perspective transformation on the pixel coordinates of the undetermined area to obtain the real-world latitude and longitude coordinates of the undetermined area. The point cloud recognition unit is used to acquire point clouds obtained by the lidar that is synchronized with the image acquisition device to detect the road area, and to recognize the point clouds to obtain the positioning information of near targets and sparse point clouds that cannot be recognized at long distances. The filtering and calculation unit is used to filter the sparse point cloud to obtain the filtered sparse point cloud, and calculate the latitude and longitude coordinates of the filtered sparse point cloud. The determining unit is used to determine the target sparse point cloud belonging to the region to be determined in the filtered sparse point cloud based on the latitude and longitude coordinates of the filtered sparse point cloud and the real-world latitude and longitude coordinates of the region to be determined. The setting unit is used to take the latitude and longitude coordinates of the foremost point cloud in the target sparse point cloud as the outer contour position of the distant target; The preset proportion area is the proportion of the area from the bottom border of the target's detection box to the vehicle body, calculated based on the height and angle information of the image acquisition device. The perspective transformation unit is further configured to: calculate the mapping matrix of the pixel coordinates of the road image projected onto the real-world latitude and longitude coordinate system; and perform perspective transformation on the pixel coordinates of the region to be determined according to the mapping matrix to obtain the real-world latitude and longitude coordinates of the region to be determined. The filtering and calculation unit is further configured to: filter out sparse point clouds within a preset height range from the ground from the sparse point cloud, and then obtain the filtered sparse point cloud. The filtering and calculation unit is further used to calculate the latitude and longitude coordinates of the filtered sparse point cloud based on the latitude and longitude of the installation location of the lidar and the northward angle.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 3.

Images