Multi-sensor target trajectory fusion method based on holographic intersection scene

Abstract

The invention discloses a multi-sensor target trajectory fusion method based on a holographic intersection scene, and relates to the technical field of image fusion, and the method comprises the steps: setting the number of cameras and the number of radars based on the intersection property corresponding to a to-be-detected region, and carrying out the detection image configuration and latitude and longitude coordinate calibration of each camera and each radar; and configuring an intersection fusion region corresponding to the to-be-detected region, and fusing any detection target in the to-be-detected region in combination with the fusion rule under the view angle of any camera or any radar. The number of the cameras and the number of the radars are set in combination with the intersection property, and the preset fusion rule is combined, so that target fusion processing in various intersection scenes can be compatible, and the continuity of the target vehicle in different advancing directions in the intersection scenes is ensured.

Description

Background Art

[0002] Traffic scenes at intersections are relatively complex, especially those in large cities, which are one of the key management objects. As autonomous driving applications are gradually realized in regions across the country, the algorithm functions of intersection sensors are increasingly required. In recent years, urban traffic intelligent technology has developed rapidly, and the realization of holographic digital display technology has greatly reduced the difficulty of intersection management. Usually, one or more hardware sensors, such as cameras or radar equipment, are installed in one direction of a driving road to dynamically perceive the location and attribute information of the target in real time.

[0003] The existing technology usually uses a drone to take an image of the entire intersection scene, maps the targets and detection areas detected by each sensor into the aerial image, and uses the Hungarian matching algorithm to match the radar targets in the radar-vision fusion area with the targets from the machine vision to establish a corresponding relationship. It usually only considers straight-ahead situations, and has less algorithm processing for target vehicles turning left or right in the intersection scene, which makes it easy for targets to be discontinuous when displayed on the twin map. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a multi-sensor target trajectory fusion method based on a holographic intersection scene, which is as follows:

[0005] 1) In the first aspect, the present invention provides a multi-sensor target trajectory fusion method based on a holographic intersection scene, and the specific technical solution is as follows:

[0006] The number of cameras and radars is set based on the nature of the intersection corresponding to the area to be detected, and the detection screen configuration and longitude and latitude coordinates calibration are performed for each camera and each radar;

[0007] Configure the intersection fusion area corresponding to the area to be detected, and fuse any detection target in the area to be detected in combination with the fusion rules from the perspective of any camera or any radar.

[0008] The beneficial effects of the multi-sensor target trajectory fusion method based on the holographic intersection scene provided by the present invention are as follows:

[0009] This solution sets the number of cameras and radars based on the nature of the intersection, and combined with preset fusion rules, it can be compatible with target fusion processing in various intersection scenarios, ensuring the continuity of target vehicles in different travel directions in intersection scenarios.

[0010] Based on the above solution, the present invention can also be improved as follows.

[0011] Furthermore, the detection screen configuration includes: labeling each lane and the corresponding lane range.

[0012] Furthermore, it also includes: configuring points at intersections corresponding to the area to be detected in a clockwise or counterclockwise order.

[0013] Furthermore, the fusion rules include:

[0014] The latitude and longitude coordinates of the target to be fused and the current lane information are collected through any camera or any radar, and the collection result is associated with the unique ID corresponding to the target to be fused, to generate detection information, and in combination with the current point corresponding to the target to be fused and the lane information where the target to be fused is currently located, the associated image data corresponding to the target to be fused in the point associated with the current point is determined, and the image data collected by the current camera or radar is tracked and fused with the associated image data.

[0015] 2) In the second aspect, the present invention also provides a multi-sensor target trajectory fusion system based on a holographic intersection scene, and the specific technical solution is as follows:

[0016] The first configuration module is used to: set the number of cameras and the number of radars based on the nature of the intersection corresponding to the area to be detected, and perform detection screen configuration and latitude and longitude coordinate calibration for each camera and each radar;

[0017] Configure the intersection fusion area corresponding to the area to be detected, and fuse any detection target in the area to be detected in combination with the fusion rules from the perspective of any camera or any radar.

[0018] Based on the above solution, the present invention can also be improved as follows.

[0019] Furthermore, the detection screen configuration includes: labeling each lane and the corresponding lane range.

[0020] Furthermore, it also includes: a second configuration module for: configuring points of the intersections corresponding to the area to be detected in a clockwise or counterclockwise order.

[0021] Furthermore, the fusion rules include:

[0022] The latitude and longitude coordinates of the target to be fused and the current lane information are collected through any camera or any radar, and the collection result is associated with the unique ID corresponding to the target to be fused, to generate detection information, and in combination with the current point corresponding to the target to be fused and the lane information where the target to be fused is currently located, the associated image data corresponding to the target to be fused in the point associated with the current point is determined, and the image data collected by the current camera or radar is tracked and fused with the associated image data.

[0023] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0024] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0025] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 This is one of the flow charts of a multi-sensor target trajectory fusion method based on a holographic intersection scene according to an embodiment of the present invention;

[0028] Figure 2 This is a second flow chart of a multi-sensor target trajectory fusion method based on a holographic intersection scene according to an embodiment of the present invention;

[0029] Figure 3 The figure is a structural frame diagram of an electronic device. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0031] like Figure 1 as well as Figure 2 As shown, a multi-sensor target trajectory fusion method based on a holographic intersection scene according to an embodiment of the present invention has the following specific technical solutions:

[0032] S1, setting the number of cameras and radars based on the nature of the intersection corresponding to the area to be detected, and configuring the detection screen and calibrating the longitude and latitude coordinates for each camera and each radar;

[0033] S2, configuring the intersection fusion area corresponding to the area to be detected, and fusing any detection target in the area to be detected in combination with the fusion rules from the perspective of any camera or any radar.

[0034] The beneficial effects of the multi-sensor target trajectory fusion method based on the holographic intersection scene provided by the present invention are as follows:

[0035] This solution sets the number of cameras and radars based on the nature of the intersection, and combined with preset fusion rules, it can be compatible with target fusion processing in various intersection scenarios, ensuring the continuity of target vehicles in different travel directions in intersection scenarios.

[0036] The intersection properties corresponding to the area to be detected refer to: crossroads, T-junctions, one-way streets and two-way streets.

[0037] The number of cameras and radars is determined according to the nature of the intersection. For example, a camera and a radar are set in each of the four directions of the intersection, facing the direction in which the vehicle is moving.

[0038] The configuration of the detection screen includes configuring the lanes and lane ranges, and marking the lanes, that is, configuring lane 1, lane 2, etc. from left to right.

[0039] In this solution, taking the crossroads as an example, four intersections are set as A, B, C, and D in a clockwise direction. Taking point A as an example, the intersection fusion area includes 8 areas, specifically:

[0040] Area 1: The entrance area of ​​the current camera detection target, that is, after the target vehicle enters or passes through the entrance area, it is set to type 1; the entrance area refers to the lane range in the monitoring direction and the position extending from the lane range to the crosswalk. Entering the entrance area means that the left-turning vehicle at the right intersection (point D) passes through the entrance area while turning left from point A. Passing the entrance area means that the vehicle drives within the lane range and passes through the above-mentioned entrance area normally, for example, a straight-moving vehicle passes through the entrance area in lane 2.

[0041] Area 2: The right-turn area of ​​the current camera detection target, that is, the target vehicle of type 1, is set to type 2 after entering or passing through the right-turn area from the right-turn lane; the size of the right-turn area is set according to the size of the intersection, but at least includes the right-turn lane and the turning area.

[0042] Area 3: Targets appearing from the right side of the field of vision, the area to enter by turning right, that is, when untyped targets enter or pass through this area, it is set to type 3; untyped targets can include pedestrians, non-motor vehicles, animals, etc.

[0043] Area 4: The left-turn area of ​​the current camera detection target, that is, the target of type 1. After entering or passing through the left-turn area from the left-turn lane 1, it is set to type 4; the left-turn area is similar to the right-turn area, but the left-turn area is larger than the right-turn area, including the left-turn lane, traffic post and turning area.

[0044] Area 5: Targets that appear from the left field of view and enter the area by turning left, that is, targets without type, are set to type 5 after entering or passing through the left-turn area;

[0045] Area 6: Targets that appear from the opposite field of view and enter the area by turning around, that is, targets without a type, are set to type 6 after entering or passing through this left-turn area;

[0046] Area 7: The U-turn area of ​​the current camera detection target, that is, the target of type 1, is set to type 7 after entering or passing through the U-turn area from the left turn lane 1;

[0047] Area 8: The exit area of ​​the current straight lane after passing the traffic post, that is, the driving area after passing the intersection, is set to type 8.

[0048] The area configuration of points B, C, and D is basically the same as that of point A, with 8 areas configured.

[0049] The specific fusion rules are:

[0050] Register the vehicle information under each type and generate a list of targets to be fused under each type. The list of targets to be fused includes the unique ID of the vehicle, the lane number where the vehicle is located, and the type of vehicle.

[0051] Left turn target fusion: From the left side B point camera detection, find the target list of target area type 4 to be fused; from the A point camera detection, find the target list of target area type 5 to be fused. The targets in these two target lists are matched in longitude and latitude coordinates. Usually, when the minimum distance is less than 3 meters, they are the same vehicle target. The attribute information of the vehicle target at B point, such as license plate, unique ID, etc., is assigned to the vehicle target at A point until the vehicle disappears at A point. At the same time, the target area type of the vehicle at B point is set to -1, and the target area type at A point is set to 0;

[0052] U-turn target fusion: From the camera detection at point C on the opposite side, find a list of targets to be fused with target area type 7; from the camera detection at point A, find a list of targets to be fused with area type 6. Match the longitude and latitude coordinates of the targets in these two target lists. Usually, when the minimum distance is less than 3 meters, they are the same vehicle target. Assign the attribute information of the vehicle target at point C, such as license plate, unique ID, etc., to the vehicle target at point A until the vehicle disappears at point A. At the same time, set the target area type of the vehicle at point C to -1, and the target area type at point A to 0.

[0053] Right turn target fusion: From the camera detection at point D on the right, find the list of targets to be fused with target area type 2; from the camera detection at point A, find the list of targets to be fused with area type 3. The targets in these two target lists are matched in longitude and latitude coordinates. Usually, when the minimum distance is less than 3 meters, they are the same vehicle target. The attribute information of the vehicle target at point D, such as license plate and unique ID, is assigned to the vehicle target at point A until the vehicle disappears at point A. At the same time, the target area type of the vehicle at point D is set to -1, and the target area type at point A is set to 0.

[0054] If the target in the current point A is a fusion target of area type 0, when the target disappears, it is deleted from the target list of point A and the vehicle target is also deleted from the target list of other points, ensuring the uniqueness of the attributes and ID after fusion;

[0055] Referring to the above three-point fusion rules, the vehicle targets in points B, C, and D are fused, that is, when the vehicle target at point B is fused, point C searches for the left-turn target source, point D searches for the U-turn target source, and point A searches for the right-turn target source.

[0056] According to a fixed period, the target with area type 1 or 0 is sent to the service terminal to ensure the uniqueness of the display on the front end. After receiving the above information, the service terminal displays the above information, that is, the same car is in the field of view of different cameras.

[0057] This description proposes a fusion algorithm for detecting targets with multiple devices at intersections. First, GPS coordinate calibration and lane configuration are performed on the detection algorithm side. The fusion area required for each point is configured on the twin map. The area information and lane information are sent to the fusion service for the fusion needs of vehicle targets in different directions. The targets are fused according to the area types at different points. Finally, the targets are distinguished by the area type attributes and sent to the platform service. When displayed on the twin map, the continuity of the attributes and ID of the same vehicle target before and after fusion is guaranteed. This fusion algorithm is also applicable to the fusion of intersection radar data or radar-vision integrated device data, which greatly improves the holographic display effect of intersection scene targets, facilitates traffic control personnel to manage vehicles at intersections in a timely manner, and makes vehicle travel smoother during peak hours in the morning and evening.

[0058] Based on the above solution, the present invention can also be improved as follows.

[0059] Furthermore, the detection screen configuration includes: labeling each lane and the corresponding lane range.

[0060] Furthermore, it also includes: configuring points at intersections corresponding to the area to be detected in a clockwise or counterclockwise order.

[0061] Furthermore, the fusion rules include:

[0062] The latitude and longitude coordinates of the target to be fused and the current lane information are collected through any camera or any radar, and the collection result is associated with the unique ID corresponding to the target to be fused, to generate detection information, and in combination with the current point corresponding to the target to be fused and the lane information where the target to be fused is currently located, the associated image data corresponding to the target to be fused in the point associated with the current point is determined, and the image data collected by the current camera or radar is tracked and fused with the associated image data.

[0063] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0064] A multi-sensor target trajectory fusion system based on holographic intersection scenes, the specific technical solution is as follows:

[0065] The first configuration module is used to: set the number of cameras and the number of radars based on the nature of the intersection corresponding to the area to be detected, and perform detection screen configuration and latitude and longitude coordinate calibration for each camera and each radar;

[0066] Configure the intersection fusion area corresponding to the area to be detected, and fuse any detection target in the area to be detected in combination with the fusion rules from the perspective of any camera or any radar.

[0067] Based on the above solution, the present invention can also be improved as follows.

[0068] Furthermore, the detection screen configuration includes: labeling each lane and the corresponding lane range.

[0069] Furthermore, it also includes: a second configuration module for: configuring points of the intersections corresponding to the area to be detected in a clockwise or counterclockwise order.

[0070] Furthermore, the fusion rules include:

[0071] The latitude and longitude coordinates of the target to be fused and the current lane information are collected through any camera or any radar, and the collection result is associated with the unique ID corresponding to the target to be fused, to generate detection information, and in combination with the current point corresponding to the target to be fused and the lane information where the target to be fused is currently located, the associated image data corresponding to the target to be fused in the point associated with the current point is determined, and the image data collected by the current camera or radar is tracked and fused with the associated image data.

[0072] It should be noted that the beneficial effects of the multi-sensor target trajectory fusion system based on the holographic intersection scene provided by the above embodiment are the same as the beneficial effects of the multi-sensor target trajectory fusion method based on the holographic intersection scene, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only takes the division of the above-mentioned functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0073] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0074] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a multi-sensor target trajectory fusion method based on a holographic intersection scene provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be repeated here.

[0075] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0076] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0077] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0079] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0080] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0081] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-sensor target trajectory fusion method based on holographic intersection scene, characterized in that: include: The number of cameras and radars is set based on the nature of the intersection corresponding to the area to be detected, and the detection screen configuration and longitude and latitude coordinates calibration are performed for each camera and each radar; Configure the intersection fusion area corresponding to the area to be detected, and fuse any detection target in the area to be detected in combination with the fusion rules from the perspective of any camera or any radar.

2. According to the multi-sensor target trajectory fusion method based on holographic intersection scene according to claim 1, it is characterized in that: The detection screen configuration includes: labeling each lane and the corresponding lane range.

3. According to the multi-sensor target trajectory fusion method based on holographic intersection scene in claim 2, it is characterized in that: Also includes: The intersections corresponding to the area to be detected are configured in a clockwise or counterclockwise order.

4. The multi-sensor target trajectory fusion method based on holographic intersection scene according to claim 3 is characterized in that: The fusion rules include: The latitude and longitude coordinates of the target to be fused and the current lane information are collected through any camera or any radar, and the collection result is associated with the unique ID corresponding to the target to be fused, to generate detection information, and in combination with the current point corresponding to the target to be fused and the lane information where the target to be fused is currently located, the associated image data corresponding to the target to be fused in the point associated with the current point is determined, and the image data collected by the current camera or radar is tracked and fused with the associated image data.

5. A multi-sensor target trajectory fusion system based on holographic intersection scenes, characterized in that: include: The first configuration module is used to: set the number of cameras and the number of radars based on the nature of the intersection corresponding to the area to be detected, and perform detection screen configuration and latitude and longitude coordinate calibration for each camera and each radar; Configure the intersection fusion area corresponding to the area to be detected, and fuse any detection target in the area to be detected in combination with the fusion rules from the perspective of any camera or any radar.

6. The multi-sensor target trajectory fusion system based on holographic intersection scene according to claim 5 is characterized in that: The detection screen configuration includes: labeling each lane and the corresponding lane range.

7. The multi-sensor target trajectory fusion system based on holographic intersection scene according to claim 6 is characterized in that: Also includes: The second configuration module is used to configure the intersections corresponding to the area to be detected in a clockwise or counterclockwise order.

8. The multi-sensor target trajectory fusion system based on holographic intersection scene according to claim 7 is characterized in that: The fusion rules include: The latitude and longitude coordinates of the target to be fused and the current lane information are collected through any camera or any radar, and the collection result is associated with the unique ID corresponding to the target to be fused, to generate detection information, and in combination with the current point corresponding to the target to be fused and the lane information where the target to be fused is currently located, the associated image data corresponding to the target to be fused in the point associated with the current point is determined, and the image data collected by the current camera or radar is tracked and fused with the associated image data.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images