A visual intelligent identification technology used on a drone

Abstract

The present application relates to the field of unmanned aerial vehicle vision technology, and specifically provides a visual intelligent identification technology used on an unmanned aerial vehicle, which pre-processes each road image frame to obtain a plurality of road image frames to be measured, performs static identification on each road image frame to be measured to detect road conditions, performs real-time target detection and dynamic feature analysis on a plurality of continuous road image frames to be measured based on an improved FPGA hardware acceleration platform to obtain the position of a target object, dynamically tracks the position of the target object to obtain a time sequence position sequence of the target object, and obtains a predicted position sequence of the target object and a predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle to determine a risk level and trigger a warning, so that the visual intelligent identification of the unmanned aerial vehicle can efficiently and accurately identify targets in a dynamic and complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle vision technology, and specifically provides a visual intelligent recognition technology used on an unmanned aerial vehicle. BACKGROUND

[0002] Traditional methods use artificial inspection, monitoring equipment and other methods for target monitoring, but the traditional methods have the problems of low efficiency, monitoring blind area and low monitoring accuracy. With the development of science and technology, unmanned aerial vehicles are widely used in many fields, such as power inspection, agriculture, emergency management and environmental monitoring, etc., which also puts forward higher requirements for the visual intelligent recognition technology of unmanned aerial vehicles.

[0003] With the development of computer vision technology and artificial intelligence, the development of unmanned aerial vehicle applications is promoted. Early unmanned aerial vehicle vision recognition mainly identifies target features through traditional computer vision algorithms such as edge detection and template matching, but these traditional methods mainly target a single scene and have poor adaptability, cannot cope with complex and dynamic environments, and thus cannot accurately identify targets. Moreover, in the process of high-speed flight of the unmanned aerial vehicle, the traditional visual recognition technology may have a delay problem in target recognition and positioning, and cannot achieve efficient and accurate target recognition of the visual intelligent recognition of the unmanned aerial vehicle in a dynamic and complex environment. SUMMARY

[0004] The present application provides a visual intelligent recognition technology used on an unmanned aerial vehicle to solve the problem that the visual intelligent recognition of the unmanned aerial vehicle cannot be achieved in a dynamic environment.

[0005] The present application provides a visual intelligent recognition technology used on an unmanned aerial vehicle, which comprises the following steps:

[0006] Obtaining a video image of a current road and decomposing the video image into a plurality of road image frames;

[0007] Preprocessing each road image frame to obtain a plurality of road test image frames;

[0008] Performing static recognition on each road test image frame to detect road conditions;

[0009] Based on an improved FPGA hardware acceleration platform, performing real-time target detection and dynamic feature analysis on a plurality of consecutive road test image frames to obtain the position of a target object, the target object including a vehicle and a pedestrian;

[0010] Performing dynamic tracking on the position of the target object to obtain a time sequence position sequence of the target object;

[0011] obtain a predicted position sequence of the target object and a predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target object and a time sequence position sequence of the unmanned aerial vehicle, to determine a risk level and trigger a warning.

[0012] In some embodiments, the real-time target detection and dynamic feature analysis on the continuous plurality of road image frames to be detected obtain the position of the target object, including:

[0013] The Farneback algorithm is used to obtain the displacement vector of the pixels in the continuous plurality of road image frames to be detected, and the background motion area and the target motion area of the continuous plurality of road image frames to be detected are distinguished by the displacement vector;

[0014] The YOLOv8 target detection algorithm is used to detect the target motion area in real time to obtain the confidence of the target category of the target object; if the confidence of the target category of the target object is greater than or equal to a confidence threshold, the position of the target object is output, and the target object includes a vehicle and a pedestrian.

[0015] In some embodiments, the dynamic tracking of the position of the target object obtains the time sequence position sequence of the target object, including:

[0016] The DeepSORT algorithm combined with the Kalman filtering algorithm is used to dynamically track the position of the target object to obtain the time sequence position sequence of the target object;

[0017] An included angle between a baseline direction vector and a vehicle motion direction vector of the target object is calculated, the baseline is a tangent of a lane center line; if the included angle is greater than an included angle threshold, it is determined that the vehicle of the target object is reverse driving;

[0018] The long short-term memory network LSTM algorithm based on deep learning is used to identify the motion state mutation of the vehicle of the target object based on the time sequence position sequence of the vehicle to obtain a probability vector of each accident category of the vehicle, and the maximum probability value is extracted from the probability vector; when the maximum probability value is greater than a probability threshold, it is determined that the vehicle has an accident.

[0019] In some embodiments, the obtaining of the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle to determine a risk level and trigger a warning includes:

[0020] The Transformer algorithm based on deep learning is used to predict based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle to obtain the predicted position sequence of the target object and the predicted position sequence of the unmanned aerial vehicle;

[0021] The distance and velocity components of the predicted position sequence of the target object and the predicted position sequence of the UAV at the same time point are obtained; the ratio of the distance to the velocity component is the TTC collision time. If the TTC collision time is less than a time threshold, it is determined to be a collision risk and an early warning is triggered.

[0022] This application provides a visual intelligent recognition technology for drones. Based on an improved FPGA hardware acceleration platform, it enables real-time target detection to handle various complex dynamic scenarios, improving the accuracy of target recognition and localization, as well as the accuracy and stability of drone visual recognition. It achieves accurate and efficient target identification in dynamic and complex environments, predicts the drone's position relative to the target based on dynamic tracking results, anticipates risks, and triggers timely risk warnings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a visual intelligent recognition technology for use on drones provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The following is combined Figure 1 The illustrated embodiments describe the technical solution of the present invention:

[0027] This application provides an embodiment of visual intelligent recognition technology for use on drones, referring to... Figure 1 As shown, the visual intelligent recognition technology for use on drones provided in this embodiment includes the following steps:

[0028] S110: Acquire the video image of the current road and decompose the video image into multiple road image frames; preprocess each road image frame to obtain multiple road image frames to be tested.

[0029] In some embodiments, the implementation of the above step S110 (acquiring a video image of the current road, and decomposing the video image into a plurality of road image frames; pre-processing each road image frame to obtain a plurality of road to-be-tested image frames) can include:

[0030] It should be noted that the visual recognition technology of the unmanned aerial vehicle is widely used in different fields, such as the transportation field, power inspection, environmental monitoring, etc. The visual recognition technology used on the unmanned aerial vehicle is to perceive the environmental image through the visual sensor, to pre-process the image, to then perform target recognition, and to make a decision based on the recognized result, so as to realize intelligent recognition of the target.

[0031] It should be noted that the traffic condition monitoring includes intelligent and accurate identification of the highway, vehicle behavior monitoring, etc. The vehicles on the highway are moving at high speed, with a speed of 60-120 km / h, and the traffic flow is large and the dynamic environment is relatively complex. The unmanned aerial vehicle can perform intelligent monitoring by cruising in the low altitude, using the high-definition camera to collect the image data of the vehicle in real time, using the intelligent algorithm to perform target detection and feature extraction, and finally performing feature analysis and safety warning, etc.

[0032] It should be noted that the high-definition camera is mounted on the unmanned aerial vehicle, and the camera continuously collects the road video stream, including the image data of the vehicles, pedestrians, and traffic facilities, etc. The high-frame-rate camera is used to reduce the motion blur, and the electronic shutter adaptive exposure is combined to ensure the clarity of the target contour. The unmanned aerial vehicle captures the turning of the vehicle at the intersection through continuous shooting, to provide the basis for traffic flow analysis.

[0033] Specifically, the IMU inertial measurement unit is linked with the visual sensor to monitor the shaking amplitude of the unmanned aerial vehicle in real time; and the high-resolution RGB camera is mounted on the unmanned aerial vehicle to collect the road image frames in real time.

[0034] It should be noted that the high-resolution RGB camera (with a resolution ≥1080P and a frame rate ≥25fps) is used to collect data, including the images of the lanes, intersections, sidewalks, and surrounding areas, to extract the target features of the vehicles and pedestrians, and to extract the visual features such as color, texture, and shape. If in low light or bad weather, the infrared camera can be used to collect thermal imaging data to highlight the thermal radiation features of the vehicles and pedestrians. Since the weather conditions will affect the quality of the image, the meteorological data such as rain, fog, and light intensity also need to be collected, and the meteorological data of the bad weather are pre-processed to reduce the influence of the bad weather on the quality of the image.

[0035] It should be noted that due to the limitation of relying only on visual sensing data, the state data of the unmanned aerial vehicle also needs to be concerned. When traffic supervision is needed, such as highways and bridges, the unmanned aerial vehicle that is close to the traffic facilities needs to be focused on to prevent the unmanned aerial vehicle from interfering with the traffic conditions and causing traffic accidents.

[0036] Specifically, the longitude and latitude information is provided by the GPS, the ground speed data of the GPS and the acceleration integration result of the inertial measurement unit (IMU) are fused, and the flight speed of the unmanned aerial vehicle and the time sequence position sequence of the unmanned aerial vehicle are obtained in real time.

[0037] It should be noted that the angular velocity of the IMU is combined with the position change of the GPS to obtain the heading of the unmanned aerial vehicle; the relative barometric height measured by the barometer is fused with the absolute height output by the GPS to obtain the height of the unmanned aerial vehicle.

[0038] It should be noted that the collected original image may be affected by the shaking of the unmanned aerial vehicle, changes in light and weather conditions, and needs to be preprocessed to remove noise and restore a clear image, thereby laying a good foundation for the next step of target recognition. The FPGA hardware acceleration image visual processing module is adopted to realize the full-link parallel pipeline processing of video acquisition, preprocessing, encoding and transmission. The function of the defogging algorithm is to eliminate image blur.

[0039] The median filter is used for denoising processing on multiple road image frames.

[0040] It should be noted that the quality of the image will be affected by the light and weather; the image needs to be preprocessed before the next step of feature extraction. Image preprocessing can remove blurred, overexposed or underexposed images, thereby improving the quality of the image. The gradient amplitude is directly related to the sharpness of the image, and the exposure detection is to avoid the loss of features due to exposure problems. Gradient amplitude and exposure detection algorithm are used to screen the quality of the road image and lay a good foundation for the next step of target feature extraction.

[0041] It should be noted that the threshold amplitude is set according to the typical sharpness of the road image, if the average of the gradient amplitude is lower than the threshold, it is judged as a blurred image and is rejected; if the average of the gradient amplitude is higher than the threshold, it is judged as a clear image, and the image is used for subsequent target detection, lane line recognition, etc.

[0042] Specifically, it is judged whether the gradient amplitude of each road image frame after denoising processing is greater than the amplitude threshold; if yes, it is judged whether the overexposure area ratio of the road image frame is greater than the ratio threshold; if yes, the road image frame is determined as a road image frame to be measured.

[0043] Exemplarily, if the gradient amplitude of the i-th frame of road image after the denoising processing is higher than the amplitude threshold value, the i-th frame of road image is a clear image, and is determined as a road image to be detected; otherwise, the i-th frame of road image is a fuzzy image, and is removed.

[0044] It should be noted that the repeated images (hash similarity greater than 95%) are removed by using the image pHash hash algorithm; the overexposed area is removed, and generally, when the pixel value is greater than or equal to 240, it is overexposed, because 240 is close to the maximum value 255, and the details are easy to be lost; when the proportion of the overexposed area in the image is greater than 30% and the proportion of the underexposed area is greater than 50%, the whole image is removed to ensure the quality of the remaining image.

[0045] The overexposure threshold value of each frame of road image to be detected is preset as a pixel value of 240-250, and the area in the brightness channel with a pixel value greater than or equal to the overexposure threshold value is marked as 1, and the rest is 0.

[0046] The proportion of the overexposed area is (the number of pixels with value 1 in the overexposure mask / the total number of pixels in the image) * 100%.

[0047] If the proportion of the overexposed area of the i-th frame of road image after the denoising processing is greater than the proportion threshold value, the proportion threshold value is 30%, the image is removed.

[0048] It should be noted that the unmanned aerial vehicle can monitor the change of data in real time in a complex environment, which puts forward requirements for coping with target occlusion and motion. In the visual recognition of the unmanned aerial vehicle, the environment will interfere, for example, light will cause the contrast between the target and the background to decrease; fog, rain, dust and other bad weather will make the image blurred or distorted.

[0049] It should be noted that the unmanned aerial vehicle can be flexibly covered in high or medium altitude, and can be applied in visual recognition in scenes such as highway management and traffic monitoring. In visual recognition, the traffic environment, the target to be recognized and the traffic state are analyzed through the visual sensor, to provide data support for traffic management and safety warning.

[0050] It should be noted that in a strong light environment, the target will be overexposed, and in a low light environment, the noise will be enhanced, and directly using a fixed confidence threshold value will cause missed detection. In addition, attention should be paid to the space-time problem of target positioning and tracking, and the time stamp alignment of multiple source data can improve the accuracy of dynamic positioning of the target, for example, the GPS signal triggers the collection of images, GPS and sensor sampling, and it is necessary to ensure that the error of all time stamps is less than or equal to 1 ms, and the IMU inertial measurement unit data of the unmanned aerial vehicle is used to interpolate and supplement the sampling interval of the sensor.

[0051] S120: performing static recognition on each road image frame to be detected to detect the road condition.

[0052] In some embodiments, the implementation of the above step S120 (static recognition on each road image frame to be detected to detect road conditions) can include:

[0053] It should be noted that single frame can accurately detect static targets to obtain static target information, attribute features, etc., to provide basic data support for the next step of dynamic correlation analysis. First, the static information of the single frame image is analyzed, and then the dynamic information is analyzed through time sequence video frame correlation. First, the single frame image is accurately positioned, and then the multiple frame images are analyzed. The unmanned aerial vehicle needs to process the continuous frame images in real time to realize the correlation dynamic monitoring of the road traffic state.

[0054] It should be noted that semantic segmentation can divide different regions and divide each pixel in the image into different categories. The traffic road uses semantic segmentation on the unmanned aerial vehicle, which can distinguish sky, ground, pedestrians, and obstacles, realize environmental self-adaptation, and extract target regions. The target region can be processed with high resolution to timely focus on changes. By determining the key region through semantic segmentation and performing detailed processing on the key region, the amount of calculation can be reduced.

[0055] It should be noted that the semantic segmentation algorithm mainly processes single frame images, classifies each pixel in each image (each frame) and divides it into different semantic regions such as roads, vehicles, pedestrians, and sky. By dividing different regions using semantic segmentation, detection of non-driving regions such as sky and green belts can be reduced.

[0056] Specifically, the U-Net model of the semantic segmentation algorithm is used for pixel-level classification on each road image frame to be detected, and each road image frame to be detected is divided into a plurality of semantic regions, including a drivable region, a traffic facility region, and a non-driving region. The drivable region includes a road surface region; the traffic facility region includes a traffic sign region and a lane line region; and the non-driving region includes sky, green belt and sidewalk.

[0057] It should be noted that the lane lines and other static targets are detected in the drivable region. First, the static targets are detected to identify the state of the road and the traffic facility. The single frame image is a static image. In traffic state monitoring, the core targets such as vehicles, pedestrians, traffic signs, lane lines, and road surface states are positioned and identified. The basic information of the road traffic such as whether the traffic sign is missing, whether the lane line is clear, and whether the road surface state is healthy is identified to ensure that the static traffic facility is not missing or damaged, which can first ensure the safety of the traffic road. The number of vehicles is also accurately counted to provide data support for traffic flow.

[0058] It should be noted that single-frame static analysis can identify the current time road state, such as traffic signal, traffic sign, lane line and other fixed traffic facilities, and if the traffic facilities are abnormal, they need to be repaired in time.

[0059] It should be noted that the autoencoder identifies whether the traffic sign region exists, and the autoencoder automatically extracts key features including color features and shape features in the input image through network structure; the autoencoder realizes anomaly detection through the process of encoding to decoding, and the output of the algorithm is a reconstructed image and an anomaly score, and finally the anomaly score is used to judge whether the traffic sign is missing, and if missing, it needs to be repaired in time.

[0060] Specifically, the autoencoder anomaly detection algorithm is used to identify each road image frame to be tested, which is divided into a traffic sign region, a lane line region and a road surface region through semantic segmentation;

[0061] The autoencoder anomaly detection algorithm is used to identify the traffic sign region, and the autoencoder realizes the reconstruction of the traffic sign region by learning the color and shape features of the traffic sign. According to the reconstruction difference, the anomaly score of the traffic sign region is calculated. If the anomaly score is greater than the score threshold, it is determined that the traffic sign is missing and needs to be repaired in time. If the anomaly score is less than or equal to the score threshold, it is determined that the traffic sign exists and is normal.

[0062] It should be noted that whether the lane line region is clear is determined by analyzing the pixel contrast. Clear lane lines should have significant gray level jumps. When the contrast is lower than the threshold, the lane lines are blurred. If the lane lines are blurred, it will directly affect the judgment of the vehicle driving trajectory, which needs to be handled in time, immediately increase the temporary warning, and also needs to be repaired and maintained regularly.

[0063] Specifically, according to the gray values of the lane line region and the road surface region, the lane line pixel gray values in the mask are extracted based on the lane line mask of the lane line region and the mask of the road surface region, and the lane line pixel gray values in the mask are denoted as , the road surface background gray values of 5-10 pixels outside the mask edge are denoted as , and the average gray difference between and is calculated; if the average gray difference is greater than or equal to the gray difference threshold, it is determined that the lane line region is normal; if the average gray difference is less than the gray difference threshold, it is determined that the lane line region is blurred and needs to be handled in time. According to experience, the gray difference threshold is set to 30.

[0064] It should be noted that the road surface state is not healthy, which will also affect the safety of driving. It is necessary to detect whether the road surface is healthy. Abnormalities such as cracks and potholes are detected and identified, and the abnormal situation is repaired and handled in time.

[0065] Specifically, the road surface region is subjected to grayscale processing, and a U-Net variant defect detection algorithm is used to recognize the road surface region after the grayscale processing to obtain a defect region of the road surface region; a total area of the defect region is counted; and a total area of the road surface region is counted.

[0066] The ratio of the total area of the defect region to the total area of the road surface region is a defect ratio of the road surface region; when the defect ratio of the road surface region is less than a first defect ratio threshold, the road is determined to be in a healthy state; when the defect ratio of the road surface region is greater than the first defect ratio threshold and less than a second defect ratio threshold, the road is determined to be in a mild abnormality; and when the defect ratio of the road surface region is greater than the second defect ratio threshold, the road is determined to be in a serious abnormality. The first defect ratio threshold is 1%, and the second defect ratio threshold is 5%,

[0067] It should be noted that the severity of the defect is determined. If there is no defect or only a slight defect that does not affect the safety of the vehicle; if there is a small amount of cracks or shallow pits, attention or maintenance should be paid in time; if there is a large area of cracks or deep pits, emergency treatment is required.

[0068] S130: Based on the improved FPGA hardware acceleration platform, real-time target detection and dynamic feature analysis are performed on a plurality of continuous road image frames to be tested to obtain the position of the target object, and the target object includes a vehicle and a pedestrian.

[0069] In some embodiments, the implementation of the above step S130 (real-time target detection and dynamic feature analysis are performed on a plurality of continuous road image frames to be tested based on the improved FPGA hardware acceleration platform to obtain the position of the target object, and the target object includes a vehicle and a pedestrian) can include:

[0070] It should be noted that target detection is one of the core tasks of visual intelligent recognition on the unmanned aerial vehicle, and an algorithm suitable for the scene should be used to achieve accurate recognition. The target feature is recognized and positioned in the image, and the target position and category information are output.

[0071] It should be noted that the moving speed, trajectory on the road, and moving direction of the pedestrian in the traffic system are also dynamically changing, and the dynamic target region needs to be identified. For example, if there is a significant displacement in multiple image frames, it is determined to be a target dynamic region. In the image processing of the unmanned aerial vehicle, a computer vision algorithm is used for target detection and recognition to locate the target position and related information.

[0072] It should be noted that the hardware acceleration is used to adapt to the embedded platform of the unmanned aerial vehicle to ensure the real-time performance and reliability of the unmanned aerial vehicle recognition. The unmanned aerial vehicle can accurately recognize the target in the flight state and in a complex environment, thereby improving the reliability and adaptability of the target detection of the unmanned aerial vehicle. The position and state of the unmanned aerial vehicle are monitored in real time, and a corresponding response is made quickly.

[0073] It should be noted that in the road target detection of the traffic system, the FPGA hardware acceleration method is used, which can realize efficient acceleration of visual algorithms by using parallel processing. FPGA can provide computing power, and FPGA can efficiently perform image preprocessing (deblurring, noise reduction), feature extraction and tracking correlation, and meet the low delay requirement of less than or equal to 50ms response time of unmanned aerial vehicle tracking, which can break through the bottleneck of CPU / GPU serial processing and reduce the delay. FPGA is a parallel control driven by hardware logic clock, which does not need to rely on the operating system, can eliminate the risk of software crash from the bottom, and can realize high reliability.

[0074] Specifically, based on the implementation of the improved FPGA hardware acceleration platform: the FPGA hardware acceleration unit carried on the unmanned aerial vehicle, the FPGA hardware acceleration unit is configured with a DSP array, the DSP array is used to perform parallel multi-channel convolution operation, an optimized activation function module is used to realize real-time detection; a dynamic feature learning module is used to realize real-time updating of features.

[0075] It should be noted that the original image collected by the general unmanned aerial vehicle will be disturbed by dynamic motion blur, light imbalance and other dynamic disturbances, and needs to be preprocessed. First, the optical flow algorithm is used to calculate the inter-frame motion vector, and the FPGA hardware acceleration is used to compensate for the image jitter (eliminate the background offset caused by the unmanned aerial vehicle itself) and deblur preprocessing, and output a stable image sequence. Motion compensation preprocessing is also needed, and the image is subjected to motion blur inverse filtering processing, which can enhance the edge features of high-speed moving targets, and thus improve the accuracy of feature extraction.

[0076] It should be noted that due to background motion such as unmanned aerial vehicle shaking, tree swaying and the like, complex background suppression is needed to filter out the background dynamic interference. By comparing the pixel motion vector with the overall motion trend of the background, the motion abnormal pixel area can be located, which is the potential target.

[0077] It should be noted that the height of the unmanned aerial vehicle will change when it is flying, which will cause the target to be significantly different in the image, ranging from small targets at a long distance to large targets at a close distance. In addition, the complex background of the unmanned aerial vehicle image will increase the difficulty of target recognition, therefore, it is necessary to first distinguish whether it is background motion or target motion.

[0078] It should be noted that by calculating the displacement vector of the pixels in multiple frames of images, the dynamic area is identified, and the trajectory of the vehicle and the moving direction of the pedestrian are timely paid attention to. Since the FPGA hardware platform carried by the unmanned aerial vehicle has limited computing power, a lightweight algorithm needs to be selected for target detection.

[0079] The optical flow method is used to calculate the displacement vector of pixels in a plurality of continuous frames of the road image to be detected, which is used to assist in judging the overall moving direction and distinguishing the background motion region and the target motion region (including vehicle driving and pedestrian movement).

[0080] It should be noted that the target motion region screened reduces the background interference, but the optical flow method only determines the target motion region after processing, and it is not yet determined whether the vehicle or the pedestrian is moving, and further target detection is required.

[0081] It should be noted that target detection is to quickly identify, locate and classify vehicles, pedestrians and other targets from complex traffic scenes to obtain the position of the target.

[0082] It should be noted that dynamic analysis is to mine the dynamic characteristics of the target, such as analyzing whether the vehicle trajectory violates the red light or reverses, and whether the pedestrian crosses the road, etc., to provide data basis for safety warning and traffic execution. Since the movement of the unmanned aerial vehicle may cause the shift of the target position, the image feature matching ORB feature is used to perform space-time alignment on multiple image frames to eliminate the shift of the target position caused by the movement of the unmanned aerial vehicle itself.

[0083] It should be noted that the driving speed, driving direction, vehicle distance of the vehicle on the road, the moving direction of the pedestrian, and the real-time state of the traffic signal light are dynamic target data that need to be extracted from continuous image frames for the next step of judging whether there is a violation and predicting risk and other situation analysis.

[0084] It should be noted that the target detection algorithm such as YOLO algorithm is used for unmanned aerial vehicle target detection, which can detect the target in dynamic changes and can detect the image in real time. According to the real-time changes, the position and related information of the target are paid attention to in real time.

[0085] It should be noted that the dynamic data such as the position is pixel coordinates, the speed is calculated by using the inter-frame displacement, the time interval of adjacent unmanned aerial vehicle frames is, for example, 30 frames / second, and the interval is approximately 0.033 seconds, and the displacement of the vehicle in the pixel plane is calculated.

[0086] Specifically, the YOLOv8 target detection algorithm is used to detect the vehicle and pedestrian in the target motion region in real time to obtain the confidence of the vehicle and pedestrian target categories; low-confidence targets are filtered, and the confidence threshold is pre-set to 0.5; if the confidence of the vehicle target category is less than the confidence threshold, the vehicle target category is removed; if the confidence of the pedestrian target category is less than the confidence threshold, the pedestrian target category is removed; if the confidence of the vehicle and pedestrian target categories is greater than or equal to the confidence threshold, the position of the vehicle and pedestrian is output; and the bounding box in the target pixel coordinate system is output.

[0087] It should be noted that the background shift caused by the movement of the unmanned aerial vehicle itself, and then the target detection is performed on the preprocessed image, and the dynamic threshold adjustment method (such as automatically optimizing the confidence threshold according to the light intensity) is used to improve the recognition accuracy.

[0088] Specifically, the NMS confidence threshold is dynamically optimized based on the target confidence distribution.

[0089] It should be noted that after determining the target, the next step is to perform target tracking, analyze the target trajectory features, and judge abnormal behavior events.

[0090] Further, the existing target recognition algorithm generally relies on single-frame image analysis, lacks continuous learning and real-time updating capability of target features. In addition, the FPGA hardware reduction on the unmanned aerial vehicle may lack real-time performance, and the complex dynamic environment on the road requires real-time monitoring of the traffic road conditions. Therefore, on the basis of FPGA hardware acceleration, a dynamic feature learning module can be added to realize dynamic updating of the model and optimization of target recognition. For example, when a new target such as a new intelligent networked car is added, a special detection head is added in the modular incremental learning and loaded into the FPGA on-chip RAM.

[0091] It should be noted that FPGA has parallel computing architecture and low delay characteristics. FPGA hardware acceleration detection can be optimized and adapted, and reinforcement learning can be used to dynamically select the optimal update strategy.

[0092] Specifically, INT16 is used to reduce the amount of calculation, and redundant convolution kernels are pruned. FPGA's DSP array parallel multi-channel convolution operation is used, and real-time detection of 30+FPS is realized by optimizing the activation function.

[0093] On the FPGA hardware acceleration platform, a special acceleration unit is designed as a dynamic feature learning module, CNN convolution layers are used for feature extraction, incremental learning matrix operations are used for feature updating, and FPGA's configurable logic unit (LUT) is used for low-delay processing.

[0094] It should be noted that since the dynamic target features are easily affected by changes such as light changes and angle rotations, long-term stable recognition of the target needs to be realized. If the feature template is directly updated using each frame of feature, the feature template may gradually deviate from the true features of the target, and feature drift may occur.

[0095] Specifically, an incremental learning method is adopted, a feature vector of an effective area (an unobstructed / low blur area) of each frame target is first extracted, a cosine similarity with a historical template is calculated, if the similarity is higher than a threshold 0.85, a weighted average is used to update the template (new feature weight 0.3 + historical template weight 0.7), to prevent feature drift; if the similarity is lower than the threshold 0.6, a feature memory pool is triggered to call, and the highest quality feature is selected from historical frames to supplement the template.

[0096] It should be noted that the target dynamic analysis result can be used to benefit the static target detection. When an abnormal trajectory such as sudden disappearance of the target is found, it can be recorded as a "suspicious static target detection result", and then a single-frame static detection is performed to perform secondary verification on the frame image. The threshold of the detection can be adjusted, and the false judgment can be reduced.

[0097] S140: dynamically tracking the position of the target object to obtain a time sequence position sequence of the target object.

[0098] In some embodiments, the implementation of the above step S140 (dynamically tracking the position of the target object to obtain a time sequence position sequence of the target object) can include:

[0099] It should be noted that due to the movement of the unmanned aerial vehicle itself and the movement of the target, the content of the image will dynamically change, and the processing algorithm also needs to have the ability to process dynamic scenes. The target tracking task is a key task of the unmanned aerial vehicle visual intelligent recognition.

[0100] It should be noted that the motion trajectory of the target is tracked in the continuous image sequence, and the position of the target can be seen at any time. The target tracking algorithm is also used to continuously track the target vehicle, so as to ensure that the tracking is continued under the conditions of lane changing and occlusion of the vehicle. At the same time, in the target tracking process, the flight control system of the unmanned aerial vehicle is also combined to adjust the route autonomously, so as to maintain a safe distance from the target vehicle.

[0101] It should be noted that the faulty vehicle is tracked to realize rapid real-time tracking and rapid positioning, and rescue is performed in time according to the positioning position; the driving state of the key vehicle is monitored, such as a large truck and a dangerous goods transport vehicle, to prevent safety hazards in advance. The tracking data of multiple vehicles is analyzed to analyze the traffic flow and the driving trajectory, and to assist in the dredging of the traffic system.

[0102] It should be noted that the flight speed of the unmanned aerial vehicle is dynamically adjusted according to the target motion speed, and the target is kept in the center of the monitoring field of view of the unmanned aerial vehicle.

[0103] It should be noted that the unmanned aerial vehicle needs to identify and track the moving target in real time in dynamic flight, such as pedestrians, obstacles, vehicles and the like, and higher requirements are put forward for the identification method, which needs to meet the speed and accuracy. The scene also has the problem of occlusion, and solving the problem of occlusion can improve the tracking stability in complex environment.

[0104] It should be noted that the dynamic information is analyzed based on continuous video frames, which is the identification of the target motion trajectory. In order to continuously track the motion trajectory of the target, the target tracking is in a dynamic moving state in traffic monitoring such as vehicle, pedestrian target, and there may be occlusion, intersection and the like, so the target tracking algorithm needs to associate the same target in different frames (continuous frames), record the pixel coordinates of the target in different frames, and output the continuous trajectory.

[0105] The image sequence formed by using continuous frames can record the motion trajectory of the target; the target tracking algorithm is adapted to dynamic targets such as moving vehicles and pedestrians.

[0106] It should be noted that the DeepSORT algorithm based on deep learning + Kalman filter algorithm can realize the continuous tracking of the target in complex scenes. In addition, the target detection result can be combined to predict the trajectory of the target motion, and the problem of occlusion tracking can be solved (the success rate of occlusion recovery is > 85%.

[0107] It should be noted that the DeepSORT algorithm is used, and a motion module (Kalman filter algorithm) is introduced on the basis of real-time tracking to realize long-time stable tracking. The DeepSORT algorithm assigns a unique ID number to the vehicle.

[0108] Specifically, the DeepSORT algorithm combined with the Kalman filter algorithm is used to dynamically track the position of the target object, and the time sequence position sequence of the target object is obtained.

[0109] It should be noted that based on the target detection and tracking results, further feature analysis and abnormal accident analysis are needed, such as judging the vehicle congestion condition, detecting traffic accidents and the like.

[0110] It should be noted that based on the mutation of the target motion state, traffic accidents can be detected, such as sudden stopping of vehicles, stopping after multiple vehicle trajectories intersect, or change in shape after vehicle collision and the like. When an accident occurs, emergency rescue and handling should be responded in time.

[0111] The angle between the reference line direction vector and the vehicle motion direction vector of the target object is calculated, and the reference line is the tangent of the lane center line; if the angle is greater than the angle threshold, it is determined that the vehicle of the target object is reverse;

[0112] Specifically, the time sequence position sequence of the vehicle of the target object is recognized by using a long short-term memory network (LSTM) algorithm based on deep learning to identify the motion state mutation of the vehicle, to obtain a vehicle accident category probability vector, and to extract a maximum probability value from the probability vector; when the maximum probability value is greater than a probability threshold value, it is determined that the vehicle has an accident; the probability threshold value is set to 0.5 according to experience.

[0113] S150: Obtain a predicted position sequence of the target object and a predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle, to determine a risk level and trigger a warning.

[0114] In some embodiments, the implementation of the above step S150 (obtaining a predicted position sequence of the target object and a predicted position sequence of the unmanned aerial vehicle based on the time sequence position sequence of the target object and the time sequence position sequence of the unmanned aerial vehicle, to determine a risk level and trigger a warning) can include:

[0115] It should be noted that the control instructions can be generated according to the results of intelligently identifying the type and position of the target, and then the functions of autonomous navigation, obstacle avoidance, and inspection of the unmanned aerial vehicle can be realized. Since the unmanned aerial vehicle is small, it is more advantageous to avoid the danger of the unmanned aerial vehicle in advance if the existence of the unmanned aerial vehicle is monitored at a farther distance. Therefore, the position of the unmanned aerial vehicle needs to be predicted in advance, and the risk of avoiding the danger needs to be handled in advance.

[0116] It should be noted that the speed and direction of the motion parameters of the detected pedestrians and vehicles are extracted, the positions after a few seconds in the future are predicted in advance, and the risk of collision is avoided in advance.

[0117] It should be noted that the motion trajectory of the dynamic target area is tracked, the moving direction is predicted in advance, and collision is prevented. The target is, for example, a pedestrian or an obstacle; if the target is blocked, the position is predicted by the historical motion trajectory and combined with background suppression.

[0118] It should be noted that predicting the position of the unmanned aerial vehicle in advance can improve the adaptability in a dynamic scene. Since the delay of the unmanned aerial vehicle when flying at high speed and tracking a high-speed target may miss the best adjustment time, affecting the obstacle avoidance of the unmanned aerial vehicle. By predicting the position of the unmanned aerial vehicle at a future time, the control instructions can be generated in advance to offset the errors caused by the delay, and the consistency of the actual motion of the unmanned aerial vehicle with the expected trajectory is ensured, such as predicting the position of the unmanned aerial vehicle in advance when tracking a high-speed target, adjusting the camera view angle and the heading of the unmanned aerial vehicle in advance, preventing the target from leaving the field of view, and ensuring that the target is always within the monitoring range.

[0119] It needs to be explained that the future position of the unmanned aerial vehicle is predicted by using the law of historical data, and the real-time applicability depends on the hardware acceleration FPGA, which combines the deep learning-based Transformer algorithm to capture long time series. The Transformer algorithm is trained by using the historical target time series position sequence of 100 frames of position data, which can predict the future 10 frames of position and realize low delay (less than 50ms) and high precision (error less than 3m) real-time prediction.

[0120] The target object time series position sequence and the unmanned aerial vehicle time series position sequence are predicted by using the deep learning-based Transformer algorithm to obtain the target object predicted position sequence and the unmanned aerial vehicle predicted position sequence. The minimum relative distance between the target object predicted position sequence and the unmanned aerial vehicle predicted position sequence within s seconds in the future is calculated by using the Euclidean distance, and if the minimum relative distance is less than the preset threshold y, it is determined as high risk; the preset threshold y is 5 meters (5 meters is the safety radius of the unmanned aerial vehicle).

[0121] It needs to be explained that the vehicle, pedestrian and unmanned aerial vehicle dynamic target may have potential collision, based on the target dynamic tracking result and trajectory prediction, real-time risk warning is carried out.

[0122] It needs to be explained that whether there is a collision risk is judged by calculating the TTC collision time. Under the premise that the unmanned aerial vehicle dynamic target maintains the current motion trend, the earliest time when the two are reduced to within the safety threshold, when the time is less than the preset threshold 5 seconds, the collision warning is triggered. The premise is that the position, speed and motion trajectory of the unmanned aerial vehicle are obtained by GPS and IMU.

[0123] First, the coordinate system is unified, the distance and speed component of the target object predicted position sequence and the unmanned aerial vehicle predicted position sequence at the same time point are calculated, the speed component is the approaching direction, and the speed component (taking positive value) of the two in the approaching direction; the ratio of the distance and the speed component is the TTC collision time, if the TTC collision time is less than the preset safety threshold r, it is determined as collision risk warning, if the TTC collision time is less than the preset safety threshold r is 2 seconds, it is determined as collision risk, and the warning is triggered.

[0124] It needs to be explained that in traffic monitoring, especially on the highway, predicting the position of the unmanned aerial vehicle in advance can ensure the safe work of the unmanned aerial vehicle, and also can improve the efficiency of traffic monitoring. Predicting the position of the unmanned aerial vehicle in advance on the complex background of the highway can avoid collision in advance through trajectory planning, and then ensure safety. Based on the prediction of the position of the unmanned aerial vehicle, the flight route of the unmanned aerial vehicle can be dynamically adjusted to reduce the blind area of monitoring.

[0125] Specifically, when an abnormality such as "collision occurs" is detected, an alarm is triggered immediately, and the location is returned to the system platform to trigger emergency handling immediately.

[0126] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual intelligent recognition technology used on a drone, characterized in that, The method comprises the following steps: acquiring a video image of a current road and decomposing the video image into a plurality of road image frames; preprocessing each of the road image frames to obtain a plurality of road to-be-tested image frames; performing static recognition on each of the road to-be-tested image frames to detect road conditions; based on an improved FPGA hardware acceleration platform, performing real-time target detection and dynamic feature analysis on a plurality of continuous road to-be-tested image frames to obtain the positions of target objects, the target objects including vehicles and pedestrians; performing dynamic tracking on the positions of the target objects to obtain a time sequence of the positions of the target objects; based on the time sequence of the positions of the target objects and a time sequence of positions of a UAV, obtaining a predicted position sequence of the target objects and a predicted position sequence of the UAV to determine a risk level and trigger a warning; wherein the process of detecting road conditions comprises: dividing each of the road to-be-tested image frames into a plurality of semantic regions, and performing static recognition on each of the semantic regions to detect road conditions; performing pixel-level classification on each of the road to-be-tested image frames using a U-Net model of a semantic segmentation algorithm to divide each of the road to-be-tested image frames into a plurality of semantic regions, the plurality of semantic regions including drivable regions, traffic facility regions, and non-drivable regions; the drivable regions including road surface regions; the traffic facility regions including traffic sign regions and lane line regions; the non-drivable regions including sky, green belts, and sidewalks; performing recognition on the traffic sign regions using an autoencoder anomaly detection algorithm, the autoencoder reconstructing the traffic sign regions by learning color and shape features of the traffic signs, calculating an anomaly score of the traffic sign regions according to differences in the reconstruction, and determining that the traffic sign regions are missing traffic signs and need to be repaired if the anomaly score is greater than a score threshold; According to the gray value of the lane line area and the road surface area, based on the lane line mask of the lane line area and the mask of the road surface area, respectively extract and , calculate the average gray difference of the and the ; if the average gray difference is less than the gray difference threshold, it is determined that the lane line area is blurred, and timely processing is required; performing grayscale processing on the road surface regions, performing recognition on the road surface regions after the grayscale processing using a U-Net variant defect detection algorithm to obtain defect regions of the road surface regions, counting a total area of the defect regions, and counting a total area of the road surface regions; a proportion of the total area of the defect regions to the total area of the road surface regions is a defect proportion of the road surface regions; when the defect proportion of the road surface regions is less than a first defect proportion threshold, the road is determined to be in a healthy state; when the defect proportion of the road surface regions is greater than the first defect proportion threshold and less than a second defect proportion threshold, the road is determined to be in a mild abnormality; when the defect proportion of the road surface regions is greater than the second defect proportion threshold, the road is determined to be in a serious abnormality. 2.The visual intelligent recognition technology for use on a UAV of claim 1, wherein, The preprocessing of each of the road image frames to obtain a plurality of road to-be-tested image frames comprises: performing denoising processing on a plurality of the road image frames using a median filter; determining whether the gradient amplitude of each of the road image frames after the denoising processing is greater than an amplitude threshold; if yes, determining whether the overexposure region proportion of the road image frame is greater than a proportion threshold; if yes, determining that the road image frame is the road to-be-tested image frame. 3.The visual intelligent recognition technology for use on a drone of claim 1, wherein, The improved FPGA hardware acceleration platform comprises: An FPGA hardware acceleration unit for being carried on a UAV, the FPGA hardware acceleration unit is configured with a DSP array for performing parallel multi-channel convolution operation, an optimized activation function module for realizing real-time detection, and a dynamic feature learning module for realizing real-time updating of features.

4. The visual intelligence recognition technology for use on a drone of claim 1, wherein, The real-time target detection and dynamic feature analysis of the continuous multiple road image frames to be detected are performed to obtain the position of the target object, including: The Farneback algorithm is used to obtain the displacement vector of the pixels in the continuous multiple road image frames to be detected, and the background motion area and the target motion area of the continuous multiple road image frames to be detected are distinguished through the displacement vector; The YOLOv8 target detection algorithm is used to detect the target motion area in real time to obtain the confidence of the target category of the target object; if the confidence of the target category of the target object is greater than or equal to the confidence threshold, the position of the target object is output, and the target object includes a vehicle and a pedestrian.

5. The visual intelligence recognition technology for use on a drone of claim 1, wherein, The dynamic tracking of the position of the target object is performed to obtain the time sequence position sequence of the target object, including: The DeepSORT algorithm combined with the Kalman filtering algorithm is used to dynamically track the position of the target object to obtain the time sequence position sequence of the target object. 6.The visual intelligent recognition technology for use on a UAV of claim 5, wherein, After the dynamic tracking of the position of the target object to obtain the time sequence position sequence of the target object, it further includes: The angle between the reference line direction vector and the vehicle motion direction vector of the target object is calculated, the reference line is the tangent of the lane center line; if the angle is greater than the angle threshold, it is determined that the vehicle of the target object is reverse driving; The LSTM algorithm based on deep learning is used to identify the motion state mutation of the vehicle of the target object from the time sequence position sequence of the vehicle to obtain the probability vector of each accident category of the vehicle, and the maximum probability value is extracted from the probability vector; when the maximum probability value is greater than the probability threshold, it is determined that the vehicle has an accident.

7. The visual intelligence recognition technology for use on a drone of claim 1, wherein, The time sequence position sequence of the target object and the time sequence position sequence of the UAV are used to obtain the predicted position sequence of the target object and the predicted position sequence of the UAV to determine the risk level and trigger the warning, including: The Transformer algorithm based on deep learning is used to predict based on the time sequence position sequence of the target object and the time sequence position sequence of the UAV to obtain the predicted position sequence of the target object and the predicted position sequence of the UAV; The distance and the velocity component of the predicted position sequence of the target object and the predicted position sequence of the UAV at the same time point are obtained; the ratio of the distance to the velocity component is the TTC collision time, if the TTC collision time is less than the time threshold, it is determined that there is a collision risk, and the warning is triggered.

Images