Visual intelligent identification technology applied to unmanned aerial vehicle

Abstract

The invention relates to the technical field of unmanned aerial vehicle vision, and particularly provides a vision intelligent identification technology for an unmanned aerial vehicle, which comprises the following steps: preprocessing each road image frame to obtain a plurality of road image frames to be detected; performing static identification on the to-be-detected image frames of the roads to detect road conditions; based on an improved FPGA hardware acceleration platform, carrying out real-time target detection and dynamic feature analysis on a plurality of continuous road image frames to be detected to obtain the position of a target object, and carrying out dynamic tracking on the position of the target object to obtain a time sequence position sequence of the target object; based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle, obtaining a predicted position sequence of the target object and a predicted position sequence of the unmanned aerial vehicle to determine a risk level and trigger early warning; the target can be efficiently and accurately recognized in a dynamic complex environment through visual intelligent recognition of the unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The application relates to the field of unmanned aerial vehicle vision technology, and particularly provides a visual intelligent identification technology used on an unmanned aerial vehicle. BACKGROUND

[0002] Traditional methods use artificial inspection, monitoring equipment and the like to monitor targets, but the traditional methods have the problems of low efficiency, monitoring blind spots and low monitoring accuracy. With the development of science and technology, unmanned aerial vehicles are widely used in many fields, such as power inspection, agriculture, emergency management and environmental monitoring and the like, and higher requirements are put forward for visual intelligent identification technology of unmanned aerial vehicles.

[0003] With the development of computer vision technology and artificial intelligence, the development of unmanned aerial vehicle applications is promoted. Early visual identification of unmanned aerial vehicles mainly identifies target features through traditional computer vision algorithms such as edge detection and template matching, but these traditional methods mainly target a single scene and have poor adaptability, cannot cope with complex and dynamic environments, and thus cannot accurately identify targets. Moreover, in the process of high-speed flight of the unmanned aerial vehicle, the traditional visual identification technology may have a delay problem in target identification and positioning, and cannot realize efficient and accurate identification of targets in dynamic and complex environments. SUMMARY

[0004] The application provides a visual intelligent identification technology used on an unmanned aerial vehicle to solve the problem that the visual intelligent identification of the unmanned aerial vehicle cannot be realized in a dynamic environment.

[0005] The application provides a visual intelligent identification technology used on an unmanned aerial vehicle, and the method comprises the following steps: obtaining a video image of a current road and decomposing the video image into a plurality of road image frames; preprocessing each road image frame to obtain a plurality of road to-be-tested image frames; performing static identification on each road to-be-tested image frame to detect road conditions; based on an improved FPGA hardware acceleration platform, performing real-time target detection and dynamic feature analysis on a plurality of continuous road to-be-tested image frames to obtain positions of target objects, the target objects including vehicles and pedestrians; performing dynamic tracking on the positions of the target objects to obtain a time sequence position sequence of the target objects; obtaining a predicted position sequence of the target objects and a predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target objects and a time sequence position sequence of the unmanned aerial vehicle to determine a risk level and trigger a warning.

[0006] In some embodiments, the real-time target detection and dynamic feature analysis of the continuous plurality of road image frames to be detected obtain the position of the target object, comprising: The Farneback algorithm is used to obtain the displacement vector of the pixels in the continuous plurality of road image frames to be detected, and the background motion area and the target motion area of the continuous plurality of road image frames to be detected are distinguished by the displacement vector; The YOLOv8 target detection algorithm is used to detect the target motion area in real time to obtain the confidence of the target category of the target object; if the confidence of the target category of the target object is greater than or equal to the confidence threshold, the position of the target object is output, and the target object includes a vehicle and a pedestrian.

[0007] In some embodiments, the position of the target object is dynamically tracked to obtain a time sequence position sequence of the target object, comprising: The DeepSORT algorithm combined with the Kalman filter algorithm is used to dynamically track the position of the target object to obtain a time sequence position sequence of the target object; The angle between the baseline direction vector and the vehicle motion direction vector of the target object is calculated, and the baseline is the tangent of the lane center line; if the angle is greater than the angle threshold, it is determined that the vehicle of the target object is reverse; The long short-term memory network LSTM algorithm based on deep learning is used to identify the motion state mutation of the vehicle of the target object from the time sequence position sequence of the vehicle to obtain a probability vector of each accident category of the vehicle, and the maximum probability value is extracted from the probability vector; when the maximum probability value is greater than the probability threshold, it is determined that the vehicle has an accident.

[0008] In some embodiments, the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle are obtained based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle to determine the risk level and trigger the early warning, comprising: The Transformer algorithm based on deep learning is used to predict based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle to obtain the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle; The distance and the velocity component of the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle at the same time point are obtained; the ratio of the distance to the velocity component is the TTC collision time, and if the TTC collision time is less than the time threshold, it is determined that there is a collision risk and the early warning is triggered.

[0009] The present invention provides a visual intelligent recognition technology for drones. Based on an improved FPGA hardware acceleration platform, it performs real-time target detection in diverse and complex dynamic scenarios, improving the accuracy of target recognition and positioning, and also enhancing the accuracy and stability of drone visual recognition. This technology enables drone visual intelligent recognition to accurately and efficiently identify targets in dynamic and complex environments. Based on dynamic tracking results, it predicts the positions of the drone and the target in advance, predicts risks, and triggers timely risk warnings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 The present invention provides a flow chart of a visual intelligent recognition technology for use on drones. DETAILED DESCRIPTION

[0012] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0013] The following combination Figure 1 The embodiment shown describes the technical solution of the present invention: This application provides an embodiment of a visual intelligent recognition technology used on a drone, referring to Figure 1 As shown, the visual intelligent recognition technology used on the drone provided in this embodiment includes the following steps: S110: Acquire a video image of the current road, and decompose the video image into multiple road image frames; pre-process each road image frame to obtain multiple road image frames to be tested.

[0014] In some embodiments, the implementation of step S110 (obtaining a video image of the current road and decomposing the video image into multiple road image frames; and preprocessing each road image frame to obtain multiple road image frames to be tested) may include: It should be noted that the visual recognition technology of the unmanned aerial vehicle is widely used in different fields, such as the traffic field, power inspection, environmental monitoring and the like. The visual recognition technology used on the unmanned aerial vehicle is to perceive the environmental image through the visual sensor, to pre-process the image, to further identify the target, and to make a decision based on the identified result, so as to realize the intelligent recognition of the target.

[0015] It should be noted that the traffic condition monitoring includes intelligent and accurate identification of the highway, vehicle behavior monitoring and the like. The vehicle on the highway is moving at a high speed, with a speed of 60-120km / h, and the traffic flow is large and the dynamic environment is relatively complex. The unmanned aerial vehicle can cruise and intelligently monitor in the low altitude, collect the image data of the vehicle in real time by using the high-definition camera, detect and extract the features of the target by using the intelligent algorithm, and finally analyze the features and make safety warning and the like.

[0016] It should be noted that the high-definition camera is carried on the unmanned aerial vehicle, and the video stream of the road is continuously collected by using the camera, including the image data of the vehicle, the pedestrian and the traffic facility. The high frame rate camera is used to reduce the motion blur, and the electronic shutter adaptive exposure is combined to ensure the definition of the target contour. The unmanned aerial vehicle captures the turning of the vehicle at the intersection through continuous shooting, and provides the basis for traffic flow analysis.

[0017] Specifically, the IMU inertial measurement unit is linked with the visual sensor to monitor the shaking amplitude of the unmanned aerial vehicle in real time; and the high-resolution RGB camera is carried on the unmanned aerial vehicle to collect the road image frame in real time.

[0018] It should be noted that the high-resolution RGB camera (the resolution is required to be greater than or equal to 1080P, and the frame rate is required to be greater than or equal to 25fps) is used to collect data, including the images of the lane, the intersection, the sidewalk and the surrounding area, to extract the target features of the vehicle and the pedestrian, and to extract the visual features such as color, texture and shape. If in low light or bad weather, the infrared camera can be used to collect thermal imaging data to highlight the thermal radiation features of the vehicle and the pedestrian. Since the weather conditions will affect the quality of the image, the meteorological data such as rain, fog and light intensity also need to be collected, and the meteorological data of the bad weather are pre-processed to reduce the influence of the bad weather on the quality of the image.

[0019] It should be noted that since there is a certain limitation in relying only on the visual sensing data, the state data of the unmanned aerial vehicle also needs to be concerned. In the traffic supervision, such as the highway and the bridge, the unmanned aerial vehicle which is close to the traffic facility needs to be focused on to prevent the unmanned aerial vehicle from interfering with the traffic condition and causing traffic accidents.

[0020] Specifically, the GPS is used to provide the latitude and longitude information, the ground speed data of the GPS and the acceleration integration result of the inertial measurement unit IMU are fused, and the flight speed of the unmanned aerial vehicle and the time sequence position sequence of the unmanned aerial vehicle are obtained in real time.

[0021] It should be noted that the angular velocity of the IMU combines the position change of the GPS to obtain the heading of the UAV; the relative barometric pressure height measured by the barometer is fused with the absolute height output by the GPS to obtain the height of the UAV.

[0022] It should be noted that the collected original image can be affected by the shaking of the UAV, changes in light and weather conditions, and needs to be preprocessed to remove noise and restore a clear image, thereby laying a good foundation for the next step of target recognition. The FPGA hardware acceleration image vision processing module is adopted to realize the full-link parallel pipeline processing of video acquisition, preprocessing, encoding and transmission. The function of the defogging algorithm is to eliminate image blur.

[0023] The median filter is used for denoising processing on multiple road image frames. It should be noted that the quality of the image will be affected by the influence of light and weather; the image needs to be preprocessed before the next step of feature extraction. Image preprocessing can remove blurred, overexposed or underexposed images, thereby improving the quality of the image. The gradient amplitude is directly related to the sharpness of the image, and the detection of the exposure is to avoid the loss of features due to exposure problems. Gradient amplitude and exposure detection algorithm are used to screen the quality of the road image and lay a good foundation for the next step of target feature extraction.

[0024] It should be noted that the threshold amplitude is set according to the typical sharpness of the road image. If the average of the gradient amplitude is lower than the threshold, it is judged as a blurred image and is rejected. If the average of the gradient amplitude is higher than the threshold, it is judged as a clear image, which is used for subsequent target detection, lane line recognition, etc.

[0025] Specifically, it is judged whether the gradient amplitude of each road image frame after denoising processing is greater than the amplitude threshold; if so, it is judged whether the overexposure area ratio of the road image frame is greater than the ratio threshold; if so, the road image frame is determined as a road to-be-measured image frame.

[0026] For example, if the gradient amplitude of the i-th frame of road image after denoising processing is higher than the amplitude threshold, it is a clear image and is determined as the i-th frame of road to-be-measured image. Otherwise, it is a blurred image and is rejected.

[0027] It should be noted that the image pHash hash algorithm is used to delete repeated images (hash similarity greater than 95%); the overexposure area is removed. Generally, when the pixel value is greater than or equal to 240, it is overexposed, because 240 is close to the maximum value 255, and details are easily lost; when the overexposure area ratio in the image is greater than 30% and the underexposure area ratio is greater than 50%, the whole image is rejected to ensure the quality of the retained image.

[0028] The preset overexposure threshold of each frame of road image is pixel value 240-250, and the area in the brightness channel with pixel value greater than or equal to the overexposure threshold is marked as 1, and the rest is 0. The overexposure area ratio = (the number of "1" pixels in the overexposure mask / the total number of image pixels) * 100%; If the overexposure area ratio of the i-th frame of road image after denoising processing is greater than the ratio threshold, the ratio threshold is 30%, the image is rejected.

[0029] It should be noted that the unmanned aerial vehicle needs to monitor the change of data in real time in a complex environment, which requires coping with target occlusion, motion, etc. In the visual recognition of the unmanned aerial vehicle, the environment will interfere, such as light, which will cause the contrast between the target and the background to decrease; fog, rain, dust and other bad weather will make the image blurred or distorted.

[0030] It should be noted that the unmanned aerial vehicle can be flexibly covered in high or medium altitude, and can be applied in visual recognition in scenes such as highway management and traffic monitoring. In visual recognition, the visual sensor can be used to monitor the traffic environment, identify the target and analyze the traffic state, so as to provide data support for traffic management and safety warning.

[0031] It should be noted that in strong light environment, the target will be overexposed, and in low light environment, the noise will be enhanced, so direct use of fixed confidence threshold will cause missed detection. In addition, attention should be paid to the space-time problem of target positioning and tracking, and the time stamp alignment of multi-source data can improve the accuracy of dynamic positioning of the target, such as GPS signal triggering image acquisition, GPS and sensor sampling, which should ensure that all time stamp errors are less than or equal to 1ms, and the IMU inertial measurement unit data of the unmanned aerial vehicle is used to interpolate and supplement the sampling interval of the sensor.

[0032] S120: performing static recognition on each road image frame to be detected to detect the road condition.

[0033] In some embodiments, the implementation of the above step S120 (performing static recognition on each road image frame to be detected to detect the road condition) can include: It should be noted that single frame can accurately detect static target and obtain static target information, attribute characteristics, etc., which can provide basic data support for dynamic correlation analysis in the next step. First, the static information of single frame image is analyzed, and then the dynamic information is analyzed by time sequence video frame correlation. First, the single frame image is accurately positioned, and then the unmanned aerial vehicle needs to process the continuous frames of images in real time to realize the correlation of dynamic monitoring of road traffic state.

[0034] It should be noted that semantic segmentation can divide different regions, and each pixel in the image can be divided into different categories; the traffic road uses semantic segmentation on the unmanned aerial vehicle, which can distinguish sky, ground, pedestrians, obstacles, realize the self-adaptation of the environment, and extract the target area. High-resolution processing can be performed on the target area to timely focus on its changes. By determining the key area through semantic segmentation and performing detailed processing on the key area, the amount of calculation can be reduced.

[0035] It should be noted that the semantic segmentation algorithm mainly processes single-frame images, classifies each pixel in each image (each frame) and divides it into different semantic regions such as roads, vehicles, pedestrians, sky, etc. By dividing different regions using semantic segmentation, detection of non-driving areas such as sky, green belts, etc. can be reduced.

[0036] Specifically, the U-Net model of the semantic segmentation algorithm is used for pixel-level classification of each road image frame to be measured, and each road image frame to be measured is divided into a plurality of semantic regions, including a drivable area, a traffic facility region, and a non-driving area; the drivable area includes a road surface area; the traffic facility region includes a traffic sign region and a lane line region; and the non-driving area includes sky, green belts and sidewalks.

[0037] It should be noted that the lane lines and other static targets in the drivable area are detected, and the state of the road and the traffic facility is identified. Single-frame images are static images, and in traffic state monitoring, core targets such as vehicles, pedestrians, traffic signs, lane lines, and road surface states are located and identified, basic information of road traffic such as whether the traffic sign is missing, whether the lane line is clear, and whether the road surface state is healthy is identified, to ensure that the static traffic facility is not missing or damaged, and to first ensure the safety of the traffic road; the number of vehicles is also accurately counted to support the data of traffic flow.

[0038] It should be noted that single-frame static analysis can identify the current road state, such as traffic signals, traffic signs, lane lines, and other fixed traffic facilities, and if there is an abnormality in the traffic facility, it needs to be repaired in time.

[0039] It should be noted that the autoencoder identifies whether the traffic sign region exists, and the autoencoder automatically extracts key features including color features and shape features from the input image through the network structure; the autoencoder realizes anomaly detection through the process of encoding to decoding, and the output of the algorithm is a reconstructed image and an anomaly score, and finally the anomaly score is used to judge whether the traffic sign is missing, and if it is missing, it needs to be repaired in time.

[0040] Specifically, the autoencoder anomaly detection algorithm is used to identify that each road image frame to be measured is divided into a traffic sign region, a lane line region, and a road surface region through semantic segmentation. An autoencoder anomaly detection algorithm is used to identify traffic sign areas. The autoencoder reconstructs the traffic sign area by learning the color and shape characteristics of the traffic sign, and calculates the anomaly score of the traffic sign area based on the reconstruction difference. If the anomaly score is greater than the score threshold, the traffic sign is determined to be missing and requires timely repair; if the anomaly score is less than or equal to the score threshold, the traffic sign is determined to be present and normal.

[0041] It should be noted that to determine whether the lane line area is clear, the lane line area is analyzed through pixel contrast. A clear lane line should have a significant grayscale jump. When the contrast is lower than the threshold, it is blurred. If the lane line is blurred, it will directly affect the judgment of the vehicle's driving trajectory. It needs to be handled in time, and temporary warnings must be added immediately. It also needs timely repairs and regular maintenance.

[0042] Specifically, according to the grayscale values ​​of the lane line area and the road surface area, based on the lane line mask of the lane line area and the mask of the road surface area, the lane line pixel grayscale value within the mask is extracted and recorded as , the grayscale value of the road background 5-10 pixels outside the mask edge is recorded as ,calculate and The average grayscale difference between the two. If the average grayscale difference is greater than or equal to the grayscale difference threshold, the lane marking area is considered normal. If the average grayscale difference is less than the grayscale difference threshold, the lane marking area is considered blurred and requires prompt treatment. Based on experience, the grayscale difference threshold is set to 30.

[0043] It should be noted that unhealthy road surface conditions will also affect driving safety. It is necessary to detect whether the road surface is healthy, detect and identify abnormalities such as cracks and potholes, and carry out repairs in a timely manner when abnormal conditions occur.

[0044] Specifically, the pavement area is grayscaled, and the grayscaled pavement area is identified using a U-Net variant defect detection algorithm to obtain a defect area of ​​the pavement area; the total area of ​​the defect area is calculated; and the total area of ​​the pavement area is calculated; The ratio of the total area of ​​the defective area to the total area of ​​the road surface area is the defect ratio of the road surface area; when the defect ratio of the road surface area is less than the first defect ratio threshold, the road is judged to be healthy; when the defect ratio of the road surface area is greater than the first defect ratio threshold and less than the second defect ratio threshold, the road is judged to be slightly abnormal; when the defect ratio of the road surface area is greater than the second defect ratio threshold, the road is judged to be seriously abnormal. The first defect ratio threshold is 1%, and the second defect ratio threshold is 5%. It should be noted that when judging the severity of defects, if there are no defects or only minor defects that do not affect the safety of the vehicle; if there are a small number of cracks or shallow potholes, timely attention or repairs should be paid; if there are large-scale cracks or deep potholes, emergency treatment is required.

[0045] S130: Based on the improved FPGA hardware acceleration platform, real-time target detection and dynamic feature analysis are performed on the continuous multiple road image frames to be detected to obtain the position of the target object, and the target object includes vehicles and pedestrians.

[0046] In some embodiments, the implementation of the above step S130 (based on the improved FPGA hardware acceleration platform, real-time target detection and dynamic feature analysis are performed on the continuous multiple road image frames to be detected to obtain the position of the target object, and the target object includes vehicles and pedestrians) can include: It should be noted that target detection is one of the core tasks of visual intelligent recognition on unmanned aerial vehicles, and only algorithms suitable for the scene can achieve accurate recognition. Identify and locate target features in images, and output target position and category information.

[0047] It should be noted that the driving speed, trajectory on the road, and the moving direction of pedestrians in the traffic system are also dynamically changing, and dynamic target areas need to be identified, such as obvious displacement in multiple image frames, which is determined as a target dynamic area. In image processing of unmanned aerial vehicles, computer vision algorithms are used for target detection and recognition, and the target position and related information are located.

[0048] It should be noted that hardware acceleration is used to adapt to the embedded platform of the unmanned aerial vehicle to ensure the real-time and reliability of the unmanned aerial vehicle recognition. The unmanned aerial vehicle can accurately recognize the target in the flight state and in complex environment, thereby improving the reliability and adaptability of the unmanned aerial vehicle target detection. Real-time attention is paid to the position and state of the unmanned aerial vehicle, and corresponding reactions are made quickly.

[0049] It should be noted that in the road target detection of the traffic system, the FPGA hardware acceleration method is used, which uses parallel processing to achieve efficient acceleration of visual algorithms. FPGA can provide computing power, and FPGA can efficiently perform image preprocessing (deblurring, noise reduction), feature extraction, and tracking correlation tasks using parallel computing, and meet the low delay requirement of less than or equal to 50ms response time for unmanned aerial vehicle tracking. It can break through the bottleneck of CPU / GPU serial processing and reduce the delay. FPGA is a parallel control driven by hardware logic clock, which does not need to rely on the operating system, can eliminate the risk of software crash from the bottom, and can achieve high reliability.

[0050] Specifically, based on the improved FPGA hardware acceleration platform: the FPGA hardware acceleration unit carried on the unmanned aerial vehicle, the FPGA hardware acceleration unit is configured with a DSP array, the DSP array is used to perform parallel multi-channel convolution operation, an optimized activation function module is used to realize real-time detection; a dynamic feature learning module is used to realize real-time update of features.

[0051] It's important to note that raw images captured by drones are typically subject to dynamic interference, such as motion blur and lighting imbalance, requiring preprocessing. First, an optical flow algorithm is used to calculate inter-frame motion vectors. FPGA hardware acceleration is then used to perform jitter compensation (eliminating background offset caused by the drone's own motion) and deblur preprocessing, resulting in a stable image sequence output. Motion compensation preprocessing and inverse motion blur filtering are also performed on the images to enhance edge features of highly moving objects, thereby improving feature extraction accuracy.

[0052] It's important to note that due to background motion, such as the drone's own jitter and the swaying of trees, complex background suppression is necessary to filter out the interference of background dynamics. By comparing pixel motion vectors with the overall motion trend of the background, we can locate pixel areas with abnormal motion, indicating potential targets.

[0053] It's important to note that the drone's altitude fluctuates during flight, resulting in noticeable differences in the appearance of targets in the image, ranging from small targets at long range to large targets at close range. Furthermore, the complex background of drone images can increase the difficulty of target recognition. Therefore, it's important to first distinguish between background motion and target motion.

[0054] It should be noted that by calculating the displacement vectors of pixels in multiple frames, dynamic areas can be identified and the trajectory of vehicles and the movement direction of pedestrians can be monitored in a timely manner. Due to the limited computing power of the FPGA hardware platform on the drone, a lightweight algorithm is required for target detection.

[0055] The optical flow method uses the Farneback algorithm to calculate the displacement vector of pixels in multiple frames of road images to be tested, which is used to assist in determining the overall movement direction and distinguish between background movement areas and target movement areas (including vehicle movement and pedestrian movement).

[0056] It should be noted that the screened target motion area will reduce background interference, but the optical flow method only determines the target motion area after processing, and it has not yet been determined whether it is a vehicle or a pedestrian in motion, and further target detection is needed.

[0057] It should be noted that target detection is to quickly identify, locate and classify vehicles, pedestrians and other targets in complex traffic scenes to obtain the target's location.

[0058] It's important to note that dynamic analysis exploits the dynamic characteristics of a target, such as analyzing whether a vehicle's trajectory is running a red light or driving against traffic, or whether a pedestrian is jaywalking, providing data for safety warnings and traffic enforcement. Because drone motion can cause target position shifts, image feature matching with ORB features allows for spatiotemporal alignment of multiple frames to eliminate target position shifts caused by the drone's own motion.

[0059] It should be noted that the driving speed, driving direction, vehicle distance, moving direction of pedestrians, real-time state of traffic signal lights and other dynamic target data of vehicles on the road need to be extracted from continuous image frames for the next step of judging whether there is a violation and predicting risk and other situation analysis.

[0060] It should be noted that the target detection algorithm such as YOLO algorithm is used for unmanned aerial vehicle target detection, which can perform target detection in dynamic changes and can perform real-time detection on images. According to the real-time changes, the position and related information of the target are paid attention to in real time.

[0061] It should be noted that dynamic data such as position is pixel coordinates, speed is calculated using inter-frame displacement, and the time interval of adjacent unmanned aerial vehicle frames is, for example, 30 frames / second, and the interval is approximately 0.033 seconds. The displacement of the vehicle in the pixel plane is calculated.

[0062] Specifically, the YOLOv8 target detection algorithm is used to detect vehicles and pedestrians in the target motion area in real time, and the confidence of the vehicle and pedestrian target categories is obtained. Low-confidence targets are filtered, and the confidence threshold is pre-set to 0.5. If the confidence of the vehicle target category is less than the confidence threshold, the vehicle target category is removed. If the confidence of the pedestrian target category is less than the confidence threshold, the pedestrian target category is removed. If the confidence of the vehicle and pedestrian target categories is greater than or equal to the confidence threshold, the position of the vehicle and pedestrian is output. And output the bounding box in the target pixel coordinate system.

[0063] It should be noted that the background will shift due to the movement of the unmanned aerial vehicle itself. After the preprocessed image is detected, the dynamic threshold adjustment method (such as automatically optimizing the confidence threshold according to the light intensity) is used to improve the recognition accuracy.

[0064] Specifically, the NMS confidence threshold is dynamically optimized based on the target confidence distribution. It should be noted that after the target is determined, the next step is to perform target tracking, analyze the trajectory features of the target, and judge abnormal behavior events.

[0065] It should be further noted that the existing target recognition algorithm generally relies on image analysis of a single frame and lacks the ability to continuously learn and update the target features in real time. In addition, the FPGA hardware reduction on the unmanned aerial vehicle may lack real-time performance, and the complex dynamic environment on the road requires real-time monitoring of the traffic road conditions. Therefore, on the basis of FPGA hardware acceleration, a dynamic feature learning module can be added to realize dynamic updating of the model and optimization of target recognition. For example, when a new target such as a new type of intelligent networked car is added, a special detection head is added in the modular incremental learning and loaded onto the FPGA on-chip RAM.

[0066] It should be noted that the FPGA has the characteristics of parallel computing architecture and low delay. The FPGA hardware acceleration detection can be optimized and adapted, and the optimal update strategy can be dynamically selected by using reinforcement learning.

[0067] Specifically, INT16 is used to reduce the amount of calculation, and redundant convolution kernels are pruned. The FPGA DSP array is used for multi-channel convolution operation, and real-time detection of 30+FPS is realized by optimizing the activation function.

[0068] On the FPGA hardware acceleration platform, a special acceleration unit is designed as a dynamic feature learning module. The CNN convolution layer is used for feature extraction, the matrix operation of incremental learning is used for feature update, and the configurable logic unit (LUT) of the FPGA is used for low-delay processing.

[0069] It should be noted that since the dynamic target feature is easily affected by changes such as light changes and angle rotations, it is necessary to realize long-term stable recognition of the target. If the feature template is directly updated using each frame feature, the feature template may gradually deviate from the real feature of the target, and feature drift may occur.

[0070] Specifically, the incremental learning method is adopted. First, the feature vector of the effective area (unobstructed / low blur area) of each frame target is extracted, the cosine similarity with the historical template is calculated, if the similarity is higher than the threshold 0.85, the weighted average is used to update the template, (new feature weight 0.3+ historical template weight 0.7), to prevent feature drift; if the similarity is lower than the threshold 0.6, the feature memory pool is triggered to call, and the highest quality feature is selected from the historical frames to supplement the template.

[0071] It should be noted that the target dynamic analysis result can be used to benefit static target detection. When an abnormal trajectory such as sudden disappearance of the target is found, it can be recorded as a “suspicious static target detection result”, and then single-frame static detection is performed to perform secondary verification on the frame image. The threshold value of the detection can be adjusted, which can reduce the false judgment.

[0072] S140: dynamically tracking the position of the target object to obtain a time sequence position sequence of the target object.

[0073] In some embodiments, the implementation of the above step S140 (dynamically tracking the position of the target object to obtain a time sequence position sequence of the target object) can include: It should be noted that the movement of the unmanned aerial vehicle itself and the movement of the target will cause dynamic changes in the content of the image, and the algorithm for processing also needs to have the ability to process dynamic scenes. The target tracking task is a key task of unmanned aerial vehicle visual intelligent recognition.

[0074] It should be noted that the motion trajectory of the target is tracked in the continuous image sequence, and the position of the target is visible at any time. The target tracking algorithm is also used to continuously track the target vehicle, so as to ensure that the tracking is continued under the conditions of lane changing, shielding and the like. At the same time, in the target tracking process, the flight control system of the unmanned aerial vehicle is also combined to adjust the route autonomously, so as to maintain a safe distance from the target vehicle.

[0075] It should be noted that the target vehicle is tracked, the rapid real-time tracking and rapid positioning are realized, the positioning position is rescued in time, the driving state of the key vehicle such as the large truck and the dangerous goods transport vehicle is monitored, and the safety hidden danger is prevented in advance. The tracking data of multiple vehicles are analyzed, the traffic flow and the driving trajectory are analyzed, and the traffic system is assisted and supported.

[0076] It should be noted that the flight speed of the unmanned aerial vehicle is dynamically adjusted according to the target motion speed, and the target is kept in the center of the monitoring field of view of the unmanned aerial vehicle.

[0077] It should be noted that the unmanned aerial vehicle needs to identify and track the moving target such as a pedestrian, an obstacle and a vehicle in real time when flying dynamically, and higher requirements are put forward for the identification method, and the speed and accuracy need to be met. The scene also has the problem of shielding, and solving the problem of shielding can improve the tracking stability in a complex environment.

[0078] It should be noted that the dynamic information is analyzed based on continuous video frames, and is the identification of the motion trajectory of the target. The target tracking is used to continuously track the motion trajectory of the target. In traffic monitoring, the vehicle and pedestrian targets are in a dynamic moving state, and there may be shielding, crossing and the like, so the target tracking algorithm needs to associate the same target in different frames (continuous frames), record the pixel coordinates of the target in different frames, and output the continuous trajectory.

[0079] The image sequence formed by using continuous frames can record the motion trajectory of the target, and the target tracking algorithm is adapted to dynamic targets such as moving vehicles and pedestrians.

[0080] It should be noted that the DeepSORT algorithm based on deep learning + Kalman filter algorithm is adopted, so that the continuous tracking of the target in a complex scene can be realized. The target detection result can also be combined to predict the trajectory of the target motion, so as to solve the problem of shielding tracking (shielding recovery success rate > 85%.

[0081] It should be noted that the DeepSORT algorithm is adopted, a motion module (Kalman filter algorithm) is introduced on the basis of real-time tracking, and long-time stable tracking is realized. The DeepSORT algorithm assigns a unique ID number to the vehicle.

[0082] Specifically, the DeepSORT algorithm combined with the Kalman filter algorithm is used to dynamically track the position of the target object, and a time sequence position sequence of the target object is obtained.

[0083] It should be noted that based on the target detection and tracking results, further feature analysis and abnormal accident analysis are needed, such as judging the vehicle congestion condition, detecting traffic accidents, etc.

[0084] It should be noted that based on the mutation of the target motion state, traffic accidents can be detected, such as sudden stopping of a vehicle, stopping after multiple vehicle trajectories intersect, or changes in the shape of a vehicle after a collision, etc. When an accident occurs, emergency rescue and handling should be responded in time.

[0085] The angle between the reference line direction vector and the vehicle motion direction vector of the target object is calculated, and the reference line is the tangent of the lane center line; if the angle is greater than the angle threshold, it is determined that the vehicle of the target object is reverse; Specifically, the long short-term memory network LSTM algorithm based on deep learning is used to identify the mutation of the motion state of the vehicle of the target object, and a vehicle accident category probability vector is obtained. The maximum probability value is extracted from the probability vector; when the maximum probability value is greater than the probability threshold, it is determined that the vehicle has an accident; the probability threshold is set to 0.5 according to experience.

[0086] S150: Based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle, the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle are obtained to determine the risk level and trigger the warning.

[0087] In some embodiments, the implementation of the above step S150 (obtaining the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle to determine the risk level and trigger the warning) can include: It should be noted that control instructions can be generated according to the results of intelligent identification of the type and position of the target, and then the functions of autonomous navigation, obstacle avoidance, and inspection of the unmanned aerial vehicle can be realized. Since the unmanned aerial vehicle is small, it is more advantageous to avoid the danger of the unmanned aerial vehicle in advance if the existence of the unmanned aerial vehicle is monitored at a farther distance. Therefore, the position of the unmanned aerial vehicle needs to be predicted in advance, and the risk of avoiding the risk is handled as much as possible in advance.

[0088] It should be noted that the speed and direction of the detected pedestrians and vehicles are extracted, the positions after a few seconds in the future are predicted in advance, and the risk of collision is avoided in advance.

[0089] It should be noted that the motion trajectory of the dynamic target area is tracked, the moving direction is predicted in advance, and collision is prevented. The target is, for example, a pedestrian, an obstacle, etc.; if the target is blocked, the position is predicted by the historical motion trajectory, and the background is combined for suppression.

[0090] It should be noted that the position of the unmanned aerial vehicle is predicted in advance, which can improve the adaptability in a dynamic scene. Since the delay of the unmanned aerial vehicle in high-speed flight and tracking of a high-speed target can miss the best adjustment time, affecting the obstacle avoidance of the unmanned aerial vehicle. By predicting the position of the unmanned aerial vehicle at a future time, control instructions can be generated in advance to offset the errors caused by the delay, ensuring that the actual motion of the unmanned aerial vehicle is consistent with the expected trajectory. For example, when tracking a high-speed target, the position of the unmanned aerial vehicle is predicted in advance, the camera view angle and the heading of the unmanned aerial vehicle can be adjusted in advance to prevent the target from leaving the field of view and ensure that the target is always within the monitoring range.

[0091] It should be noted that the future position of the unmanned aerial vehicle is predicted by using the law of historical data, and the real-time applicability depends on the hardware acceleration FPGA. In combination with the Transformer algorithm based on deep learning, long time series are captured. The Transformer algorithm is trained by using 100 frames of position data of the historical target time series position sequence, which can predict the position of the next 10 frames, and also realizes low delay (less than 50 ms) and high precision (error less than 3 meters) real-time prediction.

[0092] The time series position sequence of the target object and the time series position sequence of the unmanned aerial vehicle are predicted by using the Transformer algorithm based on deep learning to obtain the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle. The minimum relative distance between the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle within s seconds is calculated by using the Euclidean distance. If the minimum relative distance is less than a preset threshold y, it is determined that there is a high risk. The preset threshold y is 5 meters (5 meters is the safety radius of the unmanned aerial vehicle).

[0093] It should be noted that there can be potential collisions between vehicles, pedestrians and unmanned aerial vehicle dynamic targets. Based on the target dynamic tracking result and the trajectory prediction, real-time risk warning is performed.

[0094] It should be noted that whether there is a collision risk is determined by calculating the TTC collision time. Under the premise that the unmanned aerial vehicle dynamic target maintains the current motion trend, the earliest time when the two are reduced to within the safety threshold is calculated. When the time is less than a preset threshold of 5 seconds, a collision warning is triggered. The premise is that the position, speed and motion trajectory of the unmanned aerial vehicle are obtained by GPS and IMU.

[0095] First, the coordinate system is unified, the distance and velocity component of the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle at the same time point are calculated, the velocity component is the approaching direction, and the velocity component of both in the approaching direction (taking positive value); the ratio of the distance and the velocity component is the TTC collision time, if the TTC collision time is less than the preset safety threshold r, then it is determined that there is a collision risk warning, if the TTC collision time is less than the preset safety threshold r for 2 seconds, then it is determined that there is a collision risk, and the warning is triggered.

[0096] It should be noted that in traffic monitoring, especially on highways, predicting the position of the unmanned aerial vehicle in advance can ensure the safe operation of the unmanned aerial vehicle and improve the efficiency of traffic monitoring. Predicting the position of the unmanned aerial vehicle in advance on a complex background highway can avoid collision through trajectory planning in advance, thereby ensuring safety. The prediction of the position of the unmanned aerial vehicle can dynamically adjust the route of the unmanned aerial vehicle and reduce the blind area of monitoring.

[0097] Specifically, when an abnormality such as "collision occurs" is detected, an alarm is triggered immediately, and the location is returned to the system platform to trigger emergency handling immediately.

[0098] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual intelligent recognition technology used on drones, characterized in that: include: Acquire a video image of a current road, and decompose the video image into a plurality of road image frames; Preprocessing each of the road image frames to obtain a plurality of road image frames to be tested; Performing static recognition on each of the road to-be-tested image frames to detect road conditions; Based on the improved FPGA hardware acceleration platform, real-time target detection and dynamic feature analysis are performed on a plurality of consecutive road test image frames to obtain the position of the target object, which includes vehicles and pedestrians; Dynamically tracking the position of the target object to obtain a temporal position sequence of the target object; The predicted position sequence of the target object and the predicted position sequence of the drone are obtained based on the time-series position sequence of the target object and the time-series position sequence of the drone to determine the risk level and trigger an early warning.

2. The visual intelligent recognition technology used on a drone according to claim 1, characterized in that: The preprocessing of each of the road image frames to obtain a plurality of road image frames to be tested includes: Performing denoising processing on the plurality of road image frames by using median filtering; Determining whether the gradient amplitude of each of the road image frames after denoising is greater than an amplitude threshold; If yes, determining whether the proportion of the overexposed area of ​​the road image frame is greater than a proportion threshold; If so, it is determined that the road image frame is the road image frame to be tested.

3. The visual intelligent recognition technology used on a drone according to claim 1, characterized in that: The static recognition of each of the road to-be-tested image frames to detect the road condition includes: Dividing each of the road to-be-tested image frames into a plurality of semantic regions, and performing static recognition on each of the semantic regions to detect road conditions; Each of the road image frames to be tested is divided into a plurality of semantic regions, including: A U-Net model of a semantic segmentation algorithm is used to perform pixel-level classification on each of the road image frames to be tested, and each of the road image frames to be tested is divided into multiple semantic areas, wherein the multiple semantic areas include a drivable area, a traffic facility area, and a non-drivable area; the drivable area includes a road surface area; the traffic facility area includes a traffic sign area and a lane line area; and the non-drivable area includes the sky, green belts, and sidewalks.

4. The visual intelligent recognition technology used on a drone according to claim 3, characterized in that: The static recognition of each of the road to-be-tested image frames to detect the road condition further includes: An autoencoder anomaly detection algorithm is used to identify the traffic sign area. The autoencoder reconstructs the traffic sign area by learning the color and shape characteristics of the traffic sign. An anomaly score of the traffic sign area is calculated based on the difference in the reconstruction. If the anomaly score is greater than a score threshold, it is determined that the traffic sign in the traffic sign area is missing and requires timely repair. According to the grayscale values ​​of the lane line area and the road surface area, based on the lane line mask of the lane line area and the mask of the road surface area, respectively extract and , calculate the and stated If the average grayscale difference is less than the grayscale difference threshold, it is determined that the lane line area is blurred and needs to be processed in time.

5. The visual intelligent recognition technology used on a drone according to claim 3, characterized in that: The static recognition of each of the road to-be-tested image frames to detect the road condition further includes: grayscale the road surface area, identify the grayscale road surface area using a U-Net variant defect detection algorithm to obtain defect areas of the road surface area; calculate the total area of ​​the defect areas; and calculate the total area of ​​the road surface area; The ratio of the total area of ​​the defect area to the total area of ​​the road surface area is the defect ratio of the road surface area; when the defect ratio of the road surface area is less than the first defect ratio threshold, the road is judged to be in a healthy state; when the defect ratio of the road surface area is greater than the first defect ratio threshold and less than the second defect ratio threshold, the road is judged to be slightly abnormal; when the defect ratio of the road surface area is greater than the second defect ratio threshold, the road is judged to be seriously abnormal.

6. The visual intelligent recognition technology used on a drone according to claim 1, characterized in that: The improved FPGA hardware acceleration platform includes: An FPGA hardware acceleration unit for use on a drone is configured with a DSP array for performing parallel multi-channel convolution operations, an optimized activation function module for achieving real-time detection, and a dynamic feature learning module for achieving real-time feature updates.

7. The visual intelligent recognition technology used on a drone according to claim 1, characterized in that: The performing real-time target detection and dynamic feature analysis on a plurality of consecutive road image frames to obtain the position of the target object includes: Using a Farneback algorithm to obtain displacement vectors of pixels in the plurality of consecutive road image frames to be tested, and distinguishing background motion areas and target motion areas in the plurality of consecutive road image frames to be tested by using the displacement vectors; The YOLOv8 target detection algorithm is used to detect the target motion area in real time to obtain the confidence of the target category of the target object; if the confidence of the target category of the target object is greater than or equal to the confidence threshold, the position of the target object is output, and the target object includes vehicles and pedestrians.

8. The visual intelligent recognition technology used on a drone according to claim 1, characterized in that: The dynamically tracking the position of the target object to obtain a temporal position sequence of the target object includes: The DeepSORT algorithm is combined with the Kalman filter algorithm to dynamically track the position of the target object to obtain the time series position sequence of the target object.

9. The visual intelligent recognition technology used on a drone according to claim 8, characterized in that: After dynamically tracking the position of the target object to obtain the time sequence position sequence of the target object, the method further includes: Calculating an angle between a reference line direction vector and a vehicle motion direction vector of the target object, wherein the reference line is a tangent to the lane centerline; if the angle is greater than an angle threshold, determining that the vehicle of the target object is traveling in the wrong direction; A long short-term memory network (LSTM) algorithm based on deep learning is used to identify sudden changes in the vehicle's motion state in the time series position sequence of the target vehicle, obtain probability vectors of each accident category of the vehicle, and extract the maximum probability value from the probability vectors; when the maximum probability value is greater than a probability threshold, it is determined that the vehicle has an accident.

10. The visual intelligent recognition technology used on a drone according to claim 1, characterized in that: The step of obtaining the predicted position sequence of the target object and the predicted position sequence of the drone based on the time-series position sequence of the target object and the time-series position sequence of the drone to determine the risk level and trigger an early warning includes: Based on the temporal position sequence of the target object and the temporal position sequence of the drone, a Transformer algorithm based on deep learning is used to perform prediction to obtain a predicted position sequence of the target object and a predicted position sequence of the drone; The distance and velocity components between the predicted position sequence of the target object and the predicted position sequence of the UAV at the same time point are obtained; the ratio of the distance to the velocity component is the TTC collision time. If the TTC collision time is less than the time threshold, it is determined to be a collision risk and an early warning is triggered.

Images