High-speed scene vehicle target fusion method based on binocular unmanned aerial vehicle

Abstract

The invention discloses a high-speed scene vehicle target fusion method based on a binocular unmanned aerial vehicle, and relates to the technical field of image fusion, and the method comprises the steps: carrying out the video data collection of a to-be-detected road segment based on two configured collection devices, and determining a target queue corresponding to each collection device in a preset period; aiming at any one of the two sets of acquisition equipment, in combination with a pixel coordinate calibration processing result, sequentially mapping the detection frames corresponding to the target vehicles in the target queue corresponding to the acquisition equipment to the view angle of the other one of the two sets of acquisition equipment; and determining the same target vehicles in the public area to carry out target fusion processing. According to the invention, the two acquisition devices are cooperatively used, so that the acquisition range can be expanded, and the definition and accuracy of the fused image can be maintained under the condition that the acquisition range is not changed by combining with the pixel coordinate calibration of the public area.

Description

Background Art

[0002] At present, vehicle target detection technology is widely used in high-speed scenarios. In order to cover one section of the highway, a large number of cameras are usually installed on both sides of the road or on the gantry. Due to the position and height of the cameras, it is often difficult to fully cover the entire highway section, and there is a lack of flexibility. During peak hours in the morning and evening or holidays, the safety of vehicle driving has become a key concern. Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a high-speed scene vehicle target fusion method based on a binocular UAV, which is as follows:

[0004] 1) In the first aspect, the present invention provides a high-speed scene vehicle target fusion method based on a binocular drone, and the specific technical solution is as follows:

[0005] The video collection screens of two collection devices carried on the hovering drone for collecting data on the road section to be detected are configured, and the pixel coordinates of the common areas of the collection screens of the two collection devices for the road section to be detected are calibrated;

[0006] Based on the two configured acquisition devices, video data is collected on the road section to be detected, and a target queue corresponding to each acquisition device is determined within a preset period, wherein the target queue includes basic information of at least one target vehicle extracted based on the video data and arranged in time sequence;

[0007] For any of the two acquisition devices, combined with the pixel coordinate calibration processing results, the detection frames corresponding to the target vehicles in the target queue corresponding to the acquisition device are sequentially mapped to the perspective of the other of the two acquisition devices, and the same target vehicles in the common area are determined for target fusion processing.

[0008] The beneficial effects of the high-speed scene vehicle target fusion method based on binocular drone provided by the present invention are as follows:

[0009] Pixel coordinate calibration of the common area facilitates rapid coordinate conversion of the perspectives of the two acquisition devices. In addition, this solution can improve the acquisition range by using two acquisition devices in conjunction. Combined with the pixel coordinate calibration of the common area, it can maintain the clarity and accuracy of the fused image while ensuring that the acquisition range remains unchanged.

[0010] Based on the above solution, the present invention can also be improved as follows.

[0011] Furthermore, it also includes:

[0012] For the video data collected by any of the two collection devices, the driving trajectory and speed corresponding to each target vehicle in the video data are determined in real time, and whether the target vehicle has violated traffic behavior is determined based on the driving trajectory and speed corresponding to any target vehicle. For the target vehicle that has violated traffic behavior, an approval image is generated, and an alarm message is generated based on the approval image.

[0013] Furthermore, it also includes:

[0014] For the video data collected by any of the two collection devices, the longitude and latitude coordinates of any target vehicle are uploaded in real time. For the target vehicle in the public area, when the front of the target vehicle enters the public area, the collection device corresponding to the area where the body part that has not entered the public area is located continues to provide video data for determining the longitude and latitude coordinates of the target vehicle.

[0015] Furthermore, it also includes:

[0016] The target fusion processing results are displayed in real time.

[0017] 2) In the second aspect, the present invention also provides a high-speed scene vehicle target fusion system based on a binocular drone, and the specific technical solution is as follows:

[0018] The configuration module is used to: configure the video collection screens of two collection devices carried on the hovering drone for collecting data on the road section to be detected, and perform pixel coordinate calibration processing on the common area of ​​the collection screens of the two collection devices for the road section to be detected;

[0019] The acquisition module is used to: acquire video data of the road section to be detected based on the two configured acquisition devices, and determine the target queue corresponding to each acquisition device within a preset period, wherein the target queue includes basic information of at least one target vehicle extracted based on the video data and arranged in time sequence;

[0020] The fusion module is used to: for any of the two acquisition devices, in combination with the pixel coordinate calibration processing results, sequentially map the detection frame corresponding to the target vehicle in the target queue corresponding to the acquisition device to the perspective of the other acquisition device of the two acquisition devices, and determine the same target vehicle in the common area for target fusion processing.

[0021] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0022] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0023] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0025] Figure 1 A schematic diagram of a flow chart of a high-speed scene vehicle target fusion method based on a binocular UAV according to an embodiment of the present invention;

[0026] Figure 2 The figure is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0028] like Figure 1 As shown, a high-speed scene vehicle target fusion method based on a binocular drone in an embodiment of the present invention includes the following steps:

[0029] S1, configuring the video collection screens of two collection devices carried on the hovering drone for collecting data on the road section to be detected, and performing pixel coordinate calibration processing on the common area of ​​the collection screens of the two collection devices for the road section to be detected;

[0030] S2, based on the configured two acquisition devices, collect video data of the road section to be detected, and determine the target queue corresponding to each acquisition device within a preset period, wherein the target queue includes basic information of at least one target vehicle extracted based on the video data and arranged in time sequence;

[0031] S3, for any of the two acquisition devices, combined with the pixel coordinate calibration processing results, sequentially map the detection frames corresponding to the target vehicles in the target queue corresponding to the acquisition device to the perspective of the other of the two acquisition devices, and determine the same target vehicles in the common area for target fusion processing.

[0032] The beneficial effects of the high-speed scene vehicle target fusion method based on binocular drone provided by the present invention are as follows:

[0033] Pixel coordinate calibration of the common area facilitates rapid coordinate conversion of the perspectives of the two acquisition devices. In addition, this solution can improve the acquisition range by using two acquisition devices in conjunction. Combined with the pixel coordinate calibration of the common area, it can maintain the clarity and accuracy of the fused image while ensuring that the acquisition range remains unchanged.

[0034] The road sections to be inspected usually refer to expressway sections, ring expressway sections, and viaduct sections.

[0035] The acquisition device is a monocular camera or a camera that can support long-distance shooting.

[0036] The basic information of the target vehicle includes the vehicle detection frame, vehicle type, license plate number, vehicle location, etc.

[0037] The process of configuring the video capture screen includes:

[0038] Aperture configuration, ISO configuration, frame rate configuration, resolution configuration, anti-shake configuration, white balance configuration and focal length configuration, etc.

[0039] Aperture configuration:

[0040] Aperture refers to the size of the aperture of the camera lens, which is used to control the amount of light entering the camera. The larger the aperture, the more light enters the camera, and vice versa. The aperture setting is very important for controlling the depth of field of the video. A large aperture can achieve a shallow depth of field, which is suitable for shooting scenes such as portraits that need to highlight the subject; a small aperture can achieve a deeper depth of field, which is suitable for shooting scenes such as landscapes that require more details.

[0041] Configuration of ISO:

[0042] ISO refers to the camera's sensitivity, which is used to control the camera's light sensitivity. The higher the ISO, the stronger the camera's light sensitivity, which can produce brighter images in darker environments, but it will also introduce more noise; the lower the ISO, the weaker the camera's light sensitivity, which requires more light to capture a clear image, but it will also reduce the appearance of noise.

[0043] The specific process of configuring the sensitivity is as follows:

[0044] For the experimental area, collect multiple image data under the current configuration, calculate the clarity of each image data, determine the proportion of all clarity values ​​that exceed the threshold clarity, and when the proportion value is higher than the preset proportion value, output the sensitivity configuration result in the current configuration as the final result.

[0045] If the ratio is lower than the preset ratio, the contour of the target object in each image data is extracted, where the target object can be a vehicle, a pedestrian, a building, etc. The extracted contour is compared with the manually marked contour to determine the target similarity between the two, determine the target difference between the target similarity and the preset similarity, and determine the target fine-tuning amount corresponding to the target difference in the corresponding relationship between the difference and the fine-tuning amount, and fine-tune the sensitivity under the current configuration based on the target fine-tuning amount. The target difference includes both positive and negative numbers.

[0046] During the contour extraction of the target object:

[0047] Binarize the image. Binarization converts the image into black and white, with the background being black and the foreground being white. This simplifies the image and makes contour extraction more accurate.

[0048] To configure the frame rate:

[0049] Frame rate refers to the number of frames captured per second. Common frame rates include 24 frames per second, 30 frames per second, and 60 frames per second. The higher the frame rate, the smoother the picture, but it will also increase the file size and processing difficulty.

[0050] Configuration of resolution and image stabilization:

[0051] The higher the resolution, the finer the picture. The anti-shake function can effectively reduce the shaking during shooting and improve the stability of the video.

[0052] White balance and focal length configuration:

[0053] White balance is used to adjust the color balance of the picture to avoid color cast. The choice of focal length will affect the framing range and picture effect. Proper adjustment of the focal length can achieve better picture effects.

[0054] For exposure compensation configuration:

[0055] In the Program Auto mode, Aperture Priority mode, and Shutter Priority mode, exposure compensation can adjust the brightness of the photo. When the photo is dark, increase the exposure compensation; when the photo is bright, reduce the exposure compensation.

[0056] The specific process of exposure compensation is:

[0057] Acquire multiple image data taken for the experimental area, and determine the brightness corresponding to each image data. Combined with the time period of image data collection and the current weather conditions, search for all data corresponding to multiple days corresponding to the current weather conditions in the historical database. According to the time period of image data collection, extract the historical exposure compensation value corresponding to the time period in each of the searched multiple days. Sort the historical exposure compensation values ​​according to the time sequence of the searched multiple days, and assign weights to each historical exposure compensation value, where the weight value of the historical exposure compensation value corresponding to the time farther from the current moment is smaller. According to the weight assignment result, all historical exposure compensation values ​​are weighted to obtain the final exposure compensation value, and exposure compensation configuration is performed according to the final exposure compensation value.

[0058] The common area refers to the overlapping part of the scenes captured by two acquisition devices.

[0059] The purpose of performing pixel coordinate calibration processing in the public area is to determine the pixel coordinate conversion matrix corresponding to the two acquisition devices. The pixel coordinate conversion matrix can be used to convert the pixel coordinates of a point in the perspective of any acquisition device to the pixel coordinates in the perspective of another acquisition device.

[0060] The preset cycle is: according to the processing capacity of the back-end processor, the maximum data flow corresponding to the processing capacity is determined, and the efficient range is determined in the maximum data flow (the efficient range refers to the data flow range corresponding to when all data transmission channels are occupied but there is no queuing), and the data processing cycle range (preset cycle range) is determined with the efficient range, that is, the current traffic flow is determined, and the preset cycle is determined within the preset cycle range according to the current traffic flow and the road conditions of the multiple sections before and after the section. For example: There are traffic accidents on the three sections ahead of lane A, then a larger value is determined in the preset cycle range as the preset cycle.

[0061] Furthermore, it also includes:

[0062] For the video data collected by any of the two collection devices, the driving trajectory and speed corresponding to each target vehicle in the video data are determined in real time, and whether the target vehicle has violated traffic behavior is determined based on the driving trajectory and speed corresponding to any target vehicle. For the target vehicle that has violated traffic behavior, an approval image is generated, and an alarm message is generated based on the approval image.

[0063] Furthermore, it also includes:

[0064] For the video data collected by any of the two collection devices, the longitude and latitude coordinates of any target vehicle are uploaded in real time. For the target vehicle in the public area, when the front of the target vehicle enters the public area, the collection device corresponding to the area where the body part that has not entered the public area is located continues to provide video data for determining the longitude and latitude coordinates of the target vehicle.

[0065] Furthermore, it also includes:

[0066] The target fusion processing results are displayed in real time.

[0067] In Example 1, the video detection algorithm is deployed in the ARM box onboard the drone hovering in the air. After the platform obtains the push stream of the algorithm box, it configures the video screen and configures two cameras to detect the highway area. The pixel coordinates are calibrated in the common area of ​​the two video areas to facilitate the fusion of the target detection frames of the two videos. The pixels and longitude and latitude coordinates are calibrated in the highway area of ​​the two video screens. When the vehicle target is in the corresponding area, its longitude and latitude coordinates are calculated. The platform service sends the above configuration to the drone algorithm and starts real-time detection of the two video streams. The algorithm performs target fusion according to the target detection frame. The fused detection frame enters the tracking algorithm module to ensure the uniqueness and continuity of the target. This front fusion method reduces the complexity of fusion compared to the back fusion. The specific algorithm processing flow is as follows:

[0068] The video detection algorithm detects targets in the detection area of ​​the two video streams respectively. The detected target queue enters the fusion module. According to the pixel calibration configured in the previous public area, all target pixel detection frames in video screen 1 are mapped to video screen 2, and judged with the target frame in video screen 2. The intersection and union ratio of the detection frames is used to determine whether the targets in the public area are the same target, and then target fusion is performed.

[0069] The fused vehicle target queue uses a tracking algorithm to ensure the uniqueness and continuity of the target ID; the event judgment module is used to judge traffic events such as parking, congestion, wrong-way driving, and occupying the emergency lane. When the event is met, alarm videos and pictures are generated in the corresponding video screen. For targets in public areas, two alarm videos and two alarm pictures are generated for user viewing.

[0070] The fused target enters the twin trajectory generation module. In order to reduce the error of the target's longitude and latitude coordinates, the pixels and longitude and latitude coordinates are calibrated in the highway area of ​​the two video screens when the configuration is sent down. When the vehicle target is in the corresponding area, its longitude and latitude coordinates are calculated in the area. For vehicle targets in the public area, the area that the front of the vehicle has just entered is used as the basis to reduce the twin trajectory fluctuations caused by the conversion of pixel coordinates and longitude and latitude coordinates.

[0071] The generated traffic events and target trajectory information are reported to the platform service in real time and displayed on the page, so that users can understand the events occurring in the two video images and the vehicle target trajectory status in real time.

[0072] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0073] The present invention also provides a high-speed scene vehicle target fusion system based on a binocular drone, and the specific technical solution is as follows:

[0074] The configuration module is used to: configure the video collection screens of two collection devices carried on the hovering drone for collecting data on the road section to be detected, and perform pixel coordinate calibration processing on the common area of ​​the collection screens of the two collection devices for the road section to be detected;

[0075] The acquisition module is used to: acquire video data of the road section to be detected based on the two configured acquisition devices, and determine the target queue corresponding to each acquisition device within a preset period, wherein the target queue includes basic information of at least one target vehicle extracted based on the video data and arranged in time sequence;

[0076] The fusion module is used to: for any of the two acquisition devices, in combination with the pixel coordinate calibration processing results, sequentially map the detection frame corresponding to the target vehicle in the target queue corresponding to the acquisition device to the perspective of the other acquisition device of the two acquisition devices, and determine the same target vehicle in the common area for target fusion processing.

[0077] It should be noted that the beneficial effects of the high-speed scene vehicle target fusion system based on binocular drones provided in the above embodiment are the same as the beneficial effects of the high-speed scene vehicle target fusion method based on binocular drones, which will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0078] like Figure 2As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0079] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a high-speed scene vehicle target fusion method based on a binocular drone provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for realizing device functions, which will not be repeated here.

[0080] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0081] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0082] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0083] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0084] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0085] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0086] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A high-speed scene vehicle target fusion method based on binocular drone, characterized in that: include: The video collection screens of two collection devices carried on the hovering drone for collecting data on the road section to be detected are configured, and the pixel coordinates of the common areas of the collection screens of the two collection devices for the road section to be detected are calibrated; Based on the two configured acquisition devices, video data is collected on the road section to be detected, and a target queue corresponding to each acquisition device is determined within a preset period, wherein the target queue includes basic information of at least one target vehicle extracted based on the video data and arranged in time sequence; For any of the two acquisition devices, combined with the pixel coordinate calibration processing results, the detection frames corresponding to the target vehicles in the target queue corresponding to the acquisition device are sequentially mapped to the perspective of the other of the two acquisition devices, and the same target vehicles in the common area are determined for target fusion processing.

2. According to the high-speed scene vehicle target fusion method based on binocular drone according to claim 1, it is characterized in that: Also includes: For the video data collected by any of the two collection devices, the driving trajectory and speed corresponding to each target vehicle in the video data are determined in real time, and whether the target vehicle has violated traffic behavior is determined based on the driving trajectory and speed corresponding to any target vehicle. For the target vehicle that has violated traffic behavior, an approval image is generated, and an alarm message is generated based on the approval image.

3. According to the high-speed scene vehicle target fusion method based on binocular drone of claim 1, it is characterized in that: Also includes: For the video data collected by any of the two collection devices, the longitude and latitude coordinates of any target vehicle are uploaded in real time. For the target vehicle in the public area, when the front of the target vehicle enters the public area, the collection device corresponding to the area where the body part that has not entered the public area is located continues to provide video data for determining the longitude and latitude coordinates of the target vehicle.

4. According to the high-speed scene vehicle target fusion method based on binocular drone of claim 1, it is characterized in that: Also includes: The target fusion processing results are displayed in real time.

5. A high-speed scene vehicle target fusion system based on binocular drone, characterized in that: include: The configuration module is used to: configure the video collection screens of two collection devices carried on the hovering drone for collecting data on the road section to be detected, and perform pixel coordinate calibration processing on the common area of ​​the collection screens of the two collection devices for the road section to be detected; The acquisition module is used to: acquire video data of the road section to be detected based on the two configured acquisition devices, and determine the target queue corresponding to each acquisition device within a preset period, wherein the target queue includes basic information of at least one target vehicle extracted based on the video data and arranged in time sequence; The fusion module is used to: for any of the two acquisition devices, in combination with the pixel coordinate calibration processing results, sequentially map the detection frame corresponding to the target vehicle in the target queue corresponding to the acquisition device to the perspective of the other acquisition device of the two acquisition devices, and determine the same target vehicle in the common area for target fusion processing.

6. The high-speed scene vehicle target fusion system based on binocular drone according to claim 5 is characterized in that: Also includes: The alarm module is used to: determine in real time the driving trajectory and speed of each target vehicle in the video data collected by any of the two acquisition devices, determine whether the target vehicle has violated traffic behavior based on the driving trajectory and speed corresponding to any target vehicle, generate an approval image for the target vehicle that has violated traffic behavior, and generate alarm information based on the approval image.

7. The high-speed scene vehicle target fusion system based on binocular drone according to claim 5 is characterized in that: Also includes: The positioning module is used to: upload the longitude and latitude coordinates of any target vehicle in real time for the video data collected by any of the two collection devices; for the target vehicle in the public area, when the front of the target vehicle enters the public area, the collection device corresponding to the area where the body part that has not entered the public area is located continues to provide video data for determining the longitude and latitude coordinates of the target vehicle.

8. The high-speed scene vehicle target fusion system based on binocular drone according to claim 5 is characterized in that: Also includes: The display module is used to display the target fusion processing results in real time.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images