Intersection twin trajectory fusion method based on aerial camera

Abstract

The invention discloses an intersection twin trajectory fusion method based on an aerial camera, and relates to the technical field of image processing, and the method comprises the steps: configuring an unmanned aerial vehicle and a bayonet camera to meet preset conditions, and shooting a to-be-detected region through the configured unmanned aerial vehicle and the configured bayonet camera, obtaining a first collected video under the view angle of the unmanned aerial vehicle and a second collected video under the view angle of the bayonet camera; analyzing the first collected video and the second collected video in real time to generate a first target queue and a second target queue; and performing fusion processing on any target vehicle based on the first target queue and the second target queue. According to the method, the target detected by aerial photography of the hovering unmanned aerial vehicle and the target detected by the bayonet camera are fused, it is ensured that each vehicle target track is smoothly displayed on the map, meanwhile, the track disappearance phenomenon caused by shielding is avoided, the twin track fusion complexity between the cameras at the intersection is reduced, and the display effect is improved.

Description

Background Art

[0002] As people's living standards improve, the number of motor vehicles on the road is increasing. Traffic safety and congestion at urban intersections have always been one of the key issues of concern. Cameras installed at urban intersections are becoming more and more dense. Driven by the rapid development of intelligent traffic, target trajectory fusion between multiple cameras has been realized. Due to the limited installation angle of the equipment and the large number of vehicles at the intersection, occlusion is often unavoidable. The fused targets are usually lost when displayed on the twin map, lacking a certain degree of smoothness.

[0003] Existing technology usually installs multiple cameras and radars at intersections. The algorithm detects vehicle targets in each direction separately. The vehicle targets in a single direction are first fused through a fusion algorithm, and then the targets in multiple directions are fused and filtered. The fusion effect is usually better when the vehicle targets are moving straight. When fusing turning vehicles, abnormal heading angles often occur. Especially when there is occlusion, the fused target is difficult to match the actual trajectory. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a twin trajectory fusion method for intersections based on an aerial camera, which is as follows:

[0005] 1) In the first aspect, the present invention provides a twin trajectory fusion method for intersections based on an aerial camera, and the specific technical solution is as follows:

[0006] The drone and the bayonet camera are configured to meet preset conditions, and the area to be inspected is photographed by the configured drone and the configured bayonet camera to obtain a first collected video from the perspective of the drone and a second collected video from the perspective of the bayonet camera;

[0007] Analyze the first collected video and the second collected video in real time, determine at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generate a first target queue based on the at least one target vehicle, and generate a second target queue based on the at least one vehicle license plate information;

[0008] Based on the first target queue and the second target queue, any target vehicle is fused.

[0009] The beneficial effects of the twin trajectory fusion method for intersections based on aerial cameras provided by the present invention are as follows:

[0010] The targets detected by aerial photography of hovering drones and those detected by bayonet cameras are integrated to ensure that the trajectory of each vehicle target is displayed smoothly on the map. Each vehicle target also carries information such as the model and license plate attributes, which solves the problem of trajectory disappearance caused by occlusion. It can greatly reduce the complexity of twin trajectory fusion between cameras at intersections and improve the display effect.

[0011] Based on the above solution, the present invention can also be improved as follows.

[0012] Furthermore, the preset condition includes: a first lane range and a first lane number in the shooting picture of the drone correspond one to one with a second lane range and a second lane number in the shooting picture of the bayonet camera.

[0013] Furthermore, the configuration process of the drone also includes:

[0014] Determine at least one latitude and longitude coordinate base point and at least one aerial image pixel base point.

[0015] Further, based on the first target queue and the second target queue, the process of fusing any target vehicle is specifically as follows:

[0016] The vehicle license plate information that appears for the first time in the second target queue and the lane number corresponding to the vehicle license plate information are determined, and the target vehicle corresponding to the vehicle license plate information that appears for the first time is fused based on the image pixel position mapping relationship between the drone and the bayonet camera.

[0017] 2) In the second aspect, the present invention also provides a twin trajectory fusion system for intersections based on an aerial camera, and the specific technical solution is as follows:

[0018] The configuration module is used to: configure the drone and the bayonet camera to meet preset conditions, and shoot the area to be inspected through the configured drone and the configured bayonet camera to obtain a first collected video from the perspective of the drone and a second collected video from the perspective of the bayonet camera;

[0019] The determination module is used to: analyze the first collected video and the second collected video in real time, determine at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generate a first target queue based on the at least one target vehicle, and generate a second target queue based on the at least one vehicle license plate information;

[0020] The fusion module is used to: perform fusion processing on any target vehicle based on the first target queue and the second target queue.

[0021] Based on the above solution, the present invention can also be improved as follows.

[0022] Furthermore, the preset condition includes: a first lane range and a first lane number in the shooting picture of the drone correspond one to one with a second lane range and a second lane number in the shooting picture of the bayonet camera.

[0023] Furthermore, the configuration process of the drone also includes:

[0024] Determine at least one latitude and longitude coordinate base point and at least one aerial image pixel base point.

[0025] Further, based on the first target queue and the second target queue, the process of fusing any target vehicle is specifically as follows:

[0026] The vehicle license plate information that appears for the first time in the second target queue and the lane number corresponding to the vehicle license plate information are determined, and the target vehicle corresponding to the vehicle license plate information that appears for the first time is fused based on the image pixel position mapping relationship between the drone and the bayonet camera.

[0027] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0028] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0029] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0031] Figure 1 A schematic diagram of the flow of a twin trajectory fusion method for an intersection based on an aerial camera according to an embodiment of the present invention;

[0032] Figure 2 The figure is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, a twin trajectory fusion method for intersections based on an aerial camera according to an embodiment of the present invention has the following specific technical solutions:

[0035] S1, configuring the drone and the bayonet camera to meet preset conditions, and shooting the area to be inspected by the configured drone and the configured bayonet camera to obtain a first captured video from the perspective of the drone and a second captured video from the perspective of the bayonet camera;

[0036] S2, analyzing the first collected video and the second collected video in real time, determining at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generating a first target queue based on the at least one target vehicle, and generating a second target queue based on the at least one vehicle license plate information;

[0037] S3: Based on the first target queue and the second target queue, perform fusion processing on any target vehicle.

[0038] The beneficial effects of the twin trajectory fusion method for intersections based on aerial cameras provided by the present invention are as follows:

[0039] The targets detected by aerial photography of hovering drones and those detected by bayonet cameras are integrated to ensure that the trajectory of each vehicle target is displayed smoothly on the map. Each vehicle target also carries information such as the model and license plate attributes, which solves the problem of trajectory disappearance caused by occlusion. It can greatly reduce the complexity of twin trajectory fusion between cameras at intersections and improve the display effect.

[0040] S1, configure the drone and the bayonet camera to meet the preset conditions, and use the configured drone and the configured bayonet camera to shoot the area to be inspected, to obtain a first captured video from the perspective of the drone and a second captured video from the perspective of the bayonet camera.

[0041] The area to be detected is any traffic intersection that needs to be monitored in real time.

[0042] A drone refers to an unmanned aerial vehicle hovering in the air. It has the function of aerial photography and detection, and can send the captured image data to the ground receiving end through the transmission equipment mounted on the drone, so that the ground receiving end can perform subsequent processing on the video data collected by the drone. At the same time, the drone is also equipped with a device for receiving signals sent by the ground sending end, which is used to receive instructions sent by the ground sending end in real time.

[0043] The specific process of configuring the drone to meet the preset conditions is as follows:

[0044] The image field captured by the drone covers the entire area to be inspected. Each lane in the image is numbered to obtain the lane number. It should be noted that the intersection is defined as four intersections with two lanes going in opposite directions. In any intersection, the lanes going in the same direction are numbered in sequence, and the lanes going in opposite directions are renumbered, that is, the northbound 1 lane, northbound 2 lanes, southbound 1 lane, and southbound 2 lanes are generated at intersection A.

[0045] A certain number of latitude and longitude coordinate base points are selected in the image captured by the drone, and the pixel coordinates corresponding to the latitude and longitude coordinate base points are further determined. The conversion matrix is ​​determined by combining the latitude and longitude coordinate base points and the pixel coordinates, so that the latitude and longitude coordinates corresponding to the target vehicle can be determined by the pixel center point coordinates of the target vehicle detected in the field of view. Multiple aerial image pixel base points A are determined in the image captured by the drone.

[0046] In the picture captured by the camera, a pixel base point a of a certain data is selected, and the lane number corresponding to the picture captured by the camera is marked in the same way as the lane number in the picture captured by the drone.

[0047] Meeting the preset conditions means that the lane number and lane range in the image captured by the drone must be consistent with the lane number and lane range in the image captured by the camera. In addition, the position relationship between the pixel base point A of the aerial image and the pixel base point a must correspond one to one.

[0048] S2, analyzing the first collected video and the second collected video in real time, determining at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generating a first target queue based on the at least one target vehicle, and generating a second target queue based on the at least one vehicle license plate information. Wherein:

[0049] Analyze the first collected video in real time to determine the latitude and longitude coordinates, heading angle and lane number corresponding to any target vehicle;

[0050] The second collected video is analyzed in real time to determine the license plate, vehicle model and lane number corresponding to any target vehicle.

[0051] It should be noted that the analysis result of the first captured video is sent to the intersection fusion edge box through wireless transmission of the 5G module, and the intersection fusion edge box is a processing device for analyzing the second captured video captured by the bayonet camera. In addition, the second captured video captured by the bayonet camera is sent to the intersection fusion edge box corresponding to the bayonet camera through limited transmission on the intranet.

[0052] S3: Based on the first target queue and the second target queue, perform fusion processing on any target vehicle.

[0053] The intersection edge box fusion algorithm receives the target queues of the drone algorithm and the intersection edge box algorithm in real time, and performs fusion based on the mapping relationship between the lane number and the image pixel position where the target first appears. Since the lane numbers configured in the drone image and the camera image correspond one to one, it is more convenient to make fusion judgments; when the fusion is successful, the camera video target attributes are updated to the drone target (taking the vehicle target as an example, the target attributes are mainly information such as license plate, model, color, etc.; updated to the vehicle target attribute information detected by the drone). Since the duration of each frame of the image target detected by the drone edge end and the intersection edge box is inconsistent, the target fusion matching is performed by a certain time difference, such as 1 second. Usually, the fusion is performed at a fixed frame rate of 10 frames / second, and the fused and unfused drone targets are sent to the platform for display.

[0054] It should be further explained that the target queue includes a first target queue and a second target queue, wherein the first target queue is generated according to the first collected video, and the second target queue is generated according to the second collected video. The first target queue and the second target queue include but are not limited to: the unique identification (license plate number) of the vehicle, the vehicle type, the vehicle color, the lane number of the vehicle, and the location information of the vehicle.

[0055] Based on the above solution, the present invention can also be improved as follows.

[0056] Furthermore, the preset condition includes: a first lane range and a first lane number in the shooting picture of the drone correspond one to one with a second lane range and a second lane number in the shooting picture of the bayonet camera.

[0057] Furthermore, the configuration process of the drone also includes:

[0058] Determine at least one latitude and longitude coordinate base point and at least one aerial image pixel base point.

[0059] Further, based on the first target queue and the second target queue, the process of fusing any target vehicle is specifically as follows:

[0060] The vehicle license plate information that appears for the first time in the second target queue and the lane number corresponding to the vehicle license plate information are determined, and the target vehicle corresponding to the vehicle license plate information that appears for the first time is fused based on the image pixel position mapping relationship between the drone and the bayonet camera.

[0061] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0062] The present invention also provides a twin trajectory fusion system for intersections based on an aerial camera, and the specific technical solution is as follows:

[0063] The configuration module is used to: configure the drone and the bayonet camera to meet preset conditions, and shoot the area to be inspected through the configured drone and the configured bayonet camera to obtain a first collected video from the perspective of the drone and a second collected video from the perspective of the bayonet camera;

[0064] The determination module is used to: analyze the first collected video and the second collected video in real time, determine at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generate a first target queue based on the at least one target vehicle, and generate a second target queue based on the at least one vehicle license plate information;

[0065] The fusion module is used to: perform fusion processing on any target vehicle based on the first target queue and the second target queue.

[0066] Based on the above solution, the present invention can also be improved as follows.

[0067] Furthermore, the preset condition includes: a first lane range and a first lane number in the shooting picture of the drone correspond one to one with a second lane range and a second lane number in the shooting picture of the bayonet camera.

[0068] Furthermore, the configuration process of the drone also includes:

[0069] Determine at least one latitude and longitude coordinate base point and at least one aerial image pixel base point.

[0070] Further, based on the first target queue and the second target queue, the process of fusing any target vehicle is specifically as follows:

[0071] The vehicle license plate information that appears for the first time in the second target queue and the lane number corresponding to the vehicle license plate information are determined, and the target vehicle corresponding to the vehicle license plate information that appears for the first time is fused based on the image pixel position mapping relationship between the drone and the bayonet camera.

[0072] It should be noted that the beneficial effects of the twin trajectory fusion system for intersections based on aerial cameras provided in the above-mentioned embodiment are the same as the beneficial effects of the twin trajectory fusion method for intersections based on aerial cameras, which will not be repeated here. In addition, when the system provided in the above-mentioned embodiment realizes its functions, it only takes the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above-mentioned embodiments belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0073] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0074] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a twin trajectory fusion method for intersections based on aerial cameras provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0075] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0076] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0077] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0079] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0080] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0081] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A twin trajectory fusion method for intersections based on aerial cameras, characterized in that: include: The drone and the bayonet camera are configured to meet preset conditions, and the area to be inspected is photographed by the configured drone and the configured bayonet camera to obtain a first collected video from the perspective of the drone and a second collected video from the perspective of the bayonet camera; Analyze the first collected video and the second collected video in real time, determine at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generate a first target queue based on the at least one target vehicle, and generate a second target queue based on the at least one vehicle license plate information; Based on the first target queue and the second target queue, any target vehicle is fused.

2. The twin trajectory fusion method for intersections based on aerial cameras according to claim 1, characterized in that: The preset condition includes: a first lane range and a first lane number in the shooting picture of the drone correspond one to one with a second lane range and a second lane number in the shooting picture of the bayonet camera.

3. The twin trajectory fusion method for intersections based on aerial cameras according to claim 1 is characterized in that: The drone configuration process also includes: Determine at least one latitude and longitude coordinate base point and at least one aerial image pixel base point.

4. The twin trajectory fusion method for intersections based on aerial cameras according to claim 1 is characterized in that: Based on the first target queue and the second target queue, the process of fusing any target vehicle is specifically as follows: The vehicle license plate information that appears for the first time in the second target queue and the lane number corresponding to the vehicle license plate information are determined, and the target vehicle corresponding to the vehicle license plate information that appears for the first time is fused based on the image pixel position mapping relationship between the drone and the bayonet camera.

5. A twin trajectory fusion system for intersections based on aerial cameras, characterized in that: include: The configuration module is used to: configure the drone and the bayonet camera to meet preset conditions, and shoot the area to be inspected through the configured drone and the configured bayonet camera to obtain a first collected video from the perspective of the drone and a second collected video from the perspective of the bayonet camera; The determination module is used to: analyze the first collected video and the second collected video in real time, determine at least one target vehicle in the first collected video and at least one vehicle license plate information in the second collected video, generate a first target queue based on the at least one target vehicle, and generate a second target queue based on the at least one vehicle license plate information; The fusion module is used to: perform fusion processing on any target vehicle based on the first target queue and the second target queue.

6. The twin trajectory fusion system for intersections based on aerial cameras according to claim 5, characterized in that: The preset condition includes: a first lane range and a first lane number in the shooting picture of the drone correspond one to one with a second lane range and a second lane number in the shooting picture of the bayonet camera.

7. The twin trajectory fusion system for intersections based on aerial cameras according to claim 5, characterized in that: The drone configuration process also includes: Determine at least one latitude and longitude coordinate base point and at least one aerial image pixel base point.

8. The twin trajectory fusion system for intersections based on aerial cameras according to claim 5, characterized in that: Based on the first target queue and the second target queue, the process of fusing any target vehicle is specifically as follows: The vehicle license plate information that appears for the first time in the second target queue and the lane number corresponding to the vehicle license plate information are determined, and the target vehicle corresponding to the vehicle license plate information that appears for the first time is fused based on the image pixel position mapping relationship between the drone and the bayonet camera.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images