Unmanned aerial vehicle real-time visual processing system based on DVS

Abstract

The invention relates to an unmanned aerial vehicle real-time visual processing system based on DVS, and the system comprises a collection and preprocessing unit which is used for obtaining image data collected by an unmanned aerial vehicle in real time, carrying out the preprocessing, and obtaining an event flow; the target detection and recognition unit is used for detecting and recognizing a target object by adopting DVS based on the preprocessed image data to obtain image data containing a recognition result; the event prediction unit is used for extracting the dynamic characteristics from the event flow, obtaining event data by combining the dynamic characteristics with the image data containing the recognition result, predicting a future event flow, and obtaining the future position and the future motion trail of the target object; the dynamic scene reconstruction unit is used for reconstructing a dynamic scene in real time according to the event data; and the dynamic vision SLAM unit is used for realizing self-positioning and environment map construction of the unmanned aerial vehicle by adopting an SLAM algorithm according to the event data. Compared with the prior art, the real-time performance, the accuracy, the dynamic scene processing, the SLAM capability and the like are remarkably improved, and the method is suitable for complex and dynamic unmanned aerial vehicle application scenes.

Description

Technical Field

[0001] The present invention relates to the field of visual processing technology, and in particular to a real-time visual processing system for an unmanned aerial vehicle based on DVS. Background Art

[0002] Currently, drone-based real-time vision processing systems play an important role in various applications. Traditional vision systems rely on frame sensors to capture static images, which are then analyzed by processing algorithms. However, this approach has some inherent limitations in dealing with fast-moving or dynamic environments, such as slow processing speed, high energy consumption, and response delays.

[0003] In recent years, the emergence of the Dobot Vision Studio (DVS) visual algorithm platform has effectively solved these challenges. Dobot Vision Studio uses advanced visual algorithms to process dynamic scenes. Its core is to capture and analyze scene changes in real time in an efficient way. Compared with traditional frame sensors, this platform can respond to changes in dynamic scenes more quickly while significantly reducing energy consumption. It can maintain high sensitivity in low light conditions, which is crucial for the application of drones in various environments.

[0004] Although the Dobot Vision Studio visual algorithm platform performs well in terms of energy efficiency and response speed, the real-time visual processing system for drones based on this platform still faces some technical challenges. One of the main issues is how to effectively handle dynamic scenes and predict the future position and trajectory of moving objects in real time. Existing systems may have difficulty in accurate positioning and navigation in complex or unknown environments due to the lack of accurate environmental models. Summary of the invention

[0005] The purpose of the present invention is to provide a DVS-based UAV real-time visual processing system for predicting the state of moving objects and UAVs in dynamic scenes.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A real-time visual processing system for unmanned aerial vehicles based on DVS, comprising:

[0008] Acquisition and preprocessing unit: used to acquire image data collected by the drone in real time, perform preprocessing, and obtain event streams based on the preprocessed image data;

[0009] A target detection and recognition unit: connected to the acquisition and preprocessing unit, the target detection and recognition unit is used to detect and recognize the target object based on the preprocessed image data using DVS to obtain image data containing recognition results;

[0010] An event prediction unit: connected to the target detection and recognition unit and the acquisition and preprocessing unit, the event prediction unit is used to extract dynamic features from the event stream, and combine the image data containing the recognition results to obtain event data, predict the future event stream according to the event data, and obtain the future position and future motion trajectory of the target object;

[0011] A dynamic scene reconstruction unit: connected to the event prediction unit, the dynamic scene reconstruction unit is used to reconstruct the dynamic scene in real time according to the event data;

[0012] Dynamic visual SLAM unit: connected to the event prediction unit, the dynamic visual SLAM unit is used to realize the self-positioning of the drone and the construction of the environment map by using the SLAM algorithm according to the event data.

[0013] Furthermore, the acquisition and preprocessing unit includes an image data acquisition module, an image data preprocessing module and an event stream acquisition module.

[0014] The image data acquisition module is used to obtain image data collected by the drone in real time;

[0015] The image data preprocessing module is used to perform preprocessing operations including noise filtering, timestamp synchronization and data alignment on the image data;

[0016] The event stream acquisition module is used to convert dynamic change analysis into event stream based on the pre-processed image data.

[0017] Furthermore, the target detection and recognition unit includes:

[0018] Classification feature extraction module: used to extract classification features of target objects from preprocessed image data;

[0019] DVS classifier: used to detect and identify target objects using the DVS classifier according to classification features, and obtain image data containing recognition results;

[0020] Target tracking module: used to track the target object in real time.

[0021] Furthermore, the event stream prediction unit includes:

[0022] Dynamic feature extraction module: used to extract dynamic features from event streams using feature extraction algorithms;

[0023] Data conversion module: used for converting the format of the image data containing the recognition result;

[0024] Multimodal data fusion module: used to fuse the image data containing the recognition results after format conversion with the dynamic features to obtain event data;

[0025] Event data processing module: used for compressing the event data and performing sparse representation learning;

[0026] Prediction module: used to predict future event flows based on event data learned through compression and sparse representation, and obtain the future position and future motion trajectory of the target object.

[0027] Furthermore, the event data processing module includes:

[0028] An event data compression submodule: used to compress the event data using a coding algorithm;

[0029] Sparse representation learning submodule: used to perform data conversion on compressed event data using sparse representation learning technology.

[0030] Furthermore, the dynamic scene reconstruction unit includes:

[0031] Dynamic scene modeling and map updating module: used to perform preliminary dynamic scene modeling and map updating using SLAM algorithm according to the event data and dynamic information of the environment around the drone;

[0032] Loading module: used to obtain the state and future state of the target object based on the event data and the future event stream, and load it into the dynamic scene to obtain the final dynamic scene;

[0033] Dynamic scene visualization and feedback module: used to visualize the final dynamic scene and provide feedback to the user.

[0034] Furthermore, the dynamic visual SLAM unit comprises:

[0035] Positioning and map building module: used to locate the drone in real time based on the event data using SLAM algorithm and build a map in an unknown environment;

[0036] UAV visualization module: used to display the positioning of the UAV on the map in real time.

[0037] Furthermore, it also includes an event data stream visualization module, which is connected to the event prediction unit and is used to visualize the predicted event stream.

[0038] Furthermore, it also includes a time correlation analysis module, which is connected to the event prediction unit and is used to analyze the time correlation between the event data to assist in predicting future event flows.

[0039] Furthermore, it also includes an event-driven adaptive control module, which is connected to the event prediction unit and is used to adjust the control strategy of the drone according to the event data using an adaptive control algorithm, including flight path, speed adjustment and attitude control.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention uses the DVS in the target detection and recognition unit to detect and recognize the target object to distinguish each target in the image, and extracts dynamic features from the event stream through the event prediction unit. Then, the recognition results are combined to obtain the event data of the target object. The future position and future motion trajectory of the moving object can be obtained through real-time prediction. In addition, through dynamic scene reconstruction, self-positioning of the drone and environmental map construction, the state prediction of the moving object and the drone prediction in the dynamic scene can be realized.

[0042] (2) The event flow prediction unit of the present invention continuously extracts the dynamic features of the object, merges them with the recognition results, and extracts the most significant features or feature combinations through compression and sparse representation learning technology, which helps to accurately predict the event flow, thereby obtaining a more accurate future position and future motion trajectory of the target object.

[0043] (3) By adding an event-driven adaptive control module and a time correlation analysis module, the present invention can significantly improve the system's response speed, energy efficiency, environmental adaptability and system security, making it perform better and more reliable in complex and dynamic real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] Example 1

[0047] This embodiment provides a real-time visual processing system for unmanned aerial vehicles based on DVS. Figure 1 As shown, the system includes an acquisition and preprocessing unit 1, a target detection and recognition unit 2, an event prediction unit 3, a dynamic scene reconstruction unit 4, and a dynamic visual SLAM unit 5.

[0048] The acquisition and preprocessing unit 1 includes an image data acquisition module 11, an image data preprocessing module 12 and an event stream acquisition module 13. The image data acquisition module 11 includes: a camera interface submodule 111, which is used to connect the camera or visual sensor on the drone and perform data transmission and control with the camera; an image capture submodule 112, which is responsible for acquiring real-time image data from the camera; a data transmission submodule 113, which is used to transmit the image data to the ground processing system by wireless or wired means; a synchronization submodule 114, which is used to ensure the time synchronization of the image data with other sensor data for subsequent processing. The image data preprocessing module 12 is used to perform preprocessing work such as noise filtering, timestamp synchronization and data alignment on the image data obtained from the camera to improve the data quality. The event stream acquisition module 13 is used to convert the dynamic change analysis into an event stream based on the preprocessed image data.

[0049] The target detection and recognition unit 2 uses the Dobot Vision Studio system and the image processing capability of the DVS visual algorithm platform to detect and recognize the target object. The unit includes a classification feature extraction module 21, a DVS classifier 22, and a target tracking module 23. The classification feature extraction module 21 extracts the classification features of the target object from the preprocessed image data, where the target object includes vehicles, pedestrians, buildings, etc. The DVS classifier 22 detects and recognizes the target object based on the classification features using a DVS classifier trained by a machine learning algorithm to obtain the category, position, and size of the target object. The target tracking module 23 is used to track the detected target object in real time. The unit supports real-time processing and can automatically generate a recognition report and analysis results of the target according to the set standards.

[0050] The event prediction unit 3 predicts possible future event flows based on the current event data for predictive control and response of the system. The unit includes a dynamic feature extraction module 31, a data conversion module 32, a multimodal data fusion module 33, an event data processing module 34 and a prediction module 35. The dynamic feature extraction module 31 is used to extract dynamic features from the event flow using a feature extraction algorithm, such as motion direction, speed, acceleration, etc. These features are crucial for understanding and analyzing the motion state of the target. The dynamic feature extraction module 31 includes a feature detection submodule 311, a feature description submodule 312, and a feature selection and tracking submodule 313. The feature detection submodule 311 is responsible for detecting possible feature points or feature areas in the input data. For video or event data, feature detection may involve methods such as optical flow detection, corner point detection, and motion edge detection to capture areas with significant changes in the data; the feature description submodule 312 generates descriptors for the detected feature points or areas; the feature selection and tracking submodule 313 is responsible for selecting stable and information-rich feature points or areas and tracking their changes over time for continuous dynamic data. The data conversion module 32 converts the image data containing the recognition results into a processable form. The multimodal data fusion module 33 uses a fusion algorithm to fuse the image data and dynamic features containing the recognition results after format conversion to obtain event data to enhance the system's ability to perceive the environment. The module first uses an extended Kalman filter EKF or an unscented Kalman filter UKF to integrate different data, and then dynamically adjusts and allocates the fusion weights of different data, and designs different levels of information fusion strategies based on task requirements and environmental conditions. Through data fusion, the system can perceive the surrounding environment more comprehensively and accurately, including dynamic obstacles, terrain changes, etc. It can also improve the real-time and robustness of the system, adapt to complex and rapidly changing environmental conditions, and help drones make more accurate and safe decisions in the future. The event data processing module 34 includes an event data compression submodule 341 and a sparse representation learning submodule 342. The event data compression submodule 341 compresses and encodes high-frequency event data to reduce the cost of data transmission and storage while retaining key information of the data. Specifically: using efficient encoding algorithms, such as differential encoding, Huffman encoding, or dictionary-based compression methods, to convert event data into a more compact format; adjusting the compression rate according to actual needs and system resources, balancing the degree of data compression and the complexity of decompression; ensuring that the compression and decompression process has as little impact on the real-time performance of the system as possible, and adapting to application scenarios that require rapid response. The sparse representation learning submodule 342 uses sparse representation learning technology to convert event data into a higher-level expression form to improve the information utilization of the data.Among them, sparse representation learning converts data into a high-level expression form. This conversion helps to improve the information utilization of the data, making subsequent processing and analysis more efficient and accurate. The implementation method of this submodule is: Dictionary learning: One of the commonly used methods of sparse representation learning is dictionary learning. In this context, a dictionary suitable for event data can be constructed to represent different motion patterns or scene features. The construction of the dictionary can be automatically learned based on the data, or it can be pre-defined and optimized. Sparse coding: Once a suitable dictionary is available, event data can be represented by sparse coding. The goal of sparse coding is to find the least non-zero coefficients to represent the data, so that important information in the data can be retained and data can be processed and transmitted more efficiently. Feature extraction and selection: Sparse representation learning can also be used to extract the most significant features or feature combinations from event data, which can be used to further analyze the state, environmental changes or behavior patterns of the target. The main functions of this submodule include: Feature extraction: Extract representative sparse features from event data to reduce data dimensions and retain key information. Noise reduction processing: Suppress noise and redundant information in the data through sparse representation, and enhance the stability and reliability of the data. Data reconstruction: Using the results of sparse representation learning, reconstruct the high-level expression of the original event data, providing a basis for complex pattern recognition and predictive modeling. The prediction module 35 predicts the future event flow based on the event data after the event data processing module 34, and obtains the future position and future motion trajectory of the target object.

[0051] The dynamic scene reconstruction unit 4 reconstructs the dynamic model of the scene based on the event data to provide an accurate environment model that can be used to improve the positioning and navigation system of the drone. The unit includes a dynamic scene modeling and map update module 41, a loading module 42, and a dynamic scene visualization and feedback module 43. The dynamic scene modeling and map update module 41 uses the SLAM algorithm to perform preliminary dynamic scene modeling and map update based on the event data and the dynamic information of the drone's surrounding environment. The loading module 42 obtains the state and future state of the target object based on the event data and the future event stream, and loads it in the dynamic scene to obtain the final dynamic scene. The dynamic scene visualization and feedback module 43 visualizes the final dynamic scene and feeds it back to the user.

[0052] The dynamic visual SLAM unit 5 uses event data to perform real-time SLAM for positioning and mapping of drones in unknown environments. The unit includes a positioning and map building module 51 and a drone visualization module 52. The positioning and map building module 51 uses a SLAM algorithm based on the event data to perform real-time positioning of the drone and build a map in the unknown environment. The drone visualization module 52 displays the positioning of the drone in the map in real time.

[0053] Example 2

[0054] This embodiment provides a real-time visual processing system for unmanned aerial vehicles based on DVS. Figure 1 As shown, the system includes not only an acquisition and preprocessing unit 1, a target detection and recognition unit 2, an event prediction unit 3, a dynamic scene reconstruction unit 4, and a dynamic visual SLAM unit 5, but also an event data stream visualization module 6, a time correlation analysis module 7, and an event-driven adaptive control module 8.

[0055] The event data stream visualization module 6 is connected to the event prediction unit 3 to visualize the predicted event stream to help the operator understand and monitor the working status of the system.

[0056] The time correlation analysis module 7 is connected to the event prediction unit 3 and is used to analyze the time correlation between the event data and explore the causal relationship of event triggering to assist in predicting future event flows and provide more accurate time clues and predictions for system decision-making.

[0057] The event-driven adaptive control module 8 is connected to the event prediction unit 3 and is used to adjust the control strategy of the UAV according to the event data using an adaptive control algorithm, including flight path, speed adjustment and attitude control, so as to optimize the flight stability and efficiency of the UAV.

[0058] The rest is the same as in Example 1.

[0059] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0060] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A real-time visual processing system for unmanned aerial vehicles based on DVS, characterized in that: include: An acquisition and preprocessing unit (1): used to acquire image data acquired by the drone in real time, perform preprocessing, and acquire an event stream based on the preprocessed image data; A target detection and recognition unit (2): connected to the acquisition and preprocessing unit (1), the target detection and recognition unit (2) is used to detect and recognize the target object based on the preprocessed image data using DVS to obtain image data containing recognition results; An event prediction unit (3): connected to the target detection and recognition unit (2) and the acquisition and preprocessing unit (1), the event prediction unit (3) is used to extract dynamic features from the event stream, and combine the image data containing the recognition result to obtain event data, and predict the future event stream based on the event data to obtain the future position and future motion trajectory of the target object; A dynamic scene reconstruction unit (4): connected to the event prediction unit (3), the dynamic scene reconstruction unit (4) is used to reconstruct the dynamic scene in real time according to the event data; A dynamic visual SLAM unit (5) is connected to the event prediction unit (3), and is used to realize the self-positioning of the drone and the construction of an environmental map by using a SLAM algorithm according to the event data.

2. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: The acquisition and preprocessing unit (1) comprises an image data acquisition module (11), an image data preprocessing module (12) and an event stream acquisition module (13). The image data acquisition module (11) is used to acquire image data collected by the drone in real time; The image data preprocessing module (12) is used to perform preprocessing operations including noise filtering, time stamp synchronization and data alignment on the image data; The event stream acquisition module (13) is used to convert the dynamic change analysis into event stream based on the pre-processed image data.

3. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: The target detection and recognition unit (2) comprises: Classification feature extraction module (21): used to extract classification features of target objects from preprocessed image data; DVS classifier (22): used to detect and identify the target object using the DVS classifier according to the classification features, and obtain image data including the identification result; Target tracking module (23): used to track the target object in real time.

4. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: The event stream prediction unit (3) comprises: Dynamic feature extraction module (31): used to extract dynamic features from the event stream using a feature extraction algorithm; A data conversion module (32): used for converting the format of the image data containing the recognition result; A multimodal data fusion module (33): used for fusing the image data containing the recognition result after format conversion with the dynamic features to obtain event data; An event data processing module (34): used for compressing the event data and performing sparse representation learning; Prediction module (35): used to predict future event streams based on event data learned through compression and sparse representation, and obtain the future position and future motion trajectory of the target object.

5. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: The event data processing module (34) comprises: An event data compression submodule (341): used for compressing the event data using a coding algorithm; Sparse representation learning submodule (342): used to perform data conversion on the compressed event data using sparse representation learning technology.

6. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: The dynamic scene reconstruction unit (4) comprises: Dynamic scene modeling and map updating module (41): used to perform preliminary dynamic scene modeling and map updating using SLAM algorithm according to the event data and dynamic information of the environment around the drone; A loading module (42): used to obtain the state and future state of the target object based on the event data and the future event stream, and load them into the dynamic scene to obtain a final dynamic scene; Dynamic scene visualization and feedback module (43): used to visualize the final dynamic scene and provide feedback to the user.

7. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: The dynamic visual SLAM unit (5) comprises: A positioning and map building module (51): used to use a SLAM algorithm to locate the drone in real time based on the event data and to build a map in an unknown environment; UAV visualization module (52): used to display the location of the UAV in the map in real time.

8. The DVS-based UAV real-time visual processing system according to claim 1 is characterized in that: It also includes an event data stream visualization module (6), which is connected to the event prediction unit (3) and is used to visualize the predicted event stream.

9. The DVS-based UAV real-time visual processing system according to claim 1, characterized in that: It also includes a time correlation analysis module (7) connected to the event prediction unit (3) and used to analyze the time correlation between the event data to assist in predicting future event flows.

10. The DVS-based UAV real-time visual processing system according to claim 1, characterized in that: It also includes an event-driven adaptive control module (8) connected to the event prediction unit (3) and used to adjust the control strategy of the drone based on the event data using an adaptive control algorithm, including flight path, speed adjustment and attitude control.

Images