A multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception

Abstract

The present application relates to the technical field of unmanned aerial vehicle control, in particular to a multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception. According to the frequency domain dispersion difference and the spectrum mode change degree of the radar signal of each time and the previous time, and the degree of change of the unmanned aerial vehicle attitude, the signal variation degree is obtained; the state prediction model is adjusted based on the signal variation degree, and the optimized prediction model of each time is generated; according to the prediction error between the prediction state data generated by the optimized prediction model of the current time and the actual state data of the previous time, the deviation degree of the corresponding prediction error in the adjacent time within the current analysis period, the model confidence weight of the current time is obtained, and the optimal state data of the current time is determined; the unmanned aerial vehicle obstacle avoidance is executed based on the optimal state data. The present application uses the signal variation degree to dynamically correct the sensor artifacts caused by the abnormal attitude of itself and the operation of the obstacle, which effectively avoids the misjudgment of the perception system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle control, and particularly relates to a multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles in complex scenes such as logistics distribution, emergency rescue and smart city, the autonomous flight capability of unmanned aerial vehicles has become a key guarantee for efficient task execution. However, unmanned aerial vehicles need to face complex and changeable flight environments during flight, especially the existence of obstacles, which makes the autonomous obstacle avoidance capability of unmanned aerial vehicles face severe challenges. Traditional unmanned aerial vehicle obstacle avoidance technology mainly relies on a single sensor to realize obstacle detection and avoidance, such as laser radar, ultrasonic wave, visual sensor, etc. However, due to the inherent physical limitations and single perception dimension of a single sensor, it cannot meet the high reliability and high safety requirements of obstacle avoidance. Multi-modal sensors complement each other, overcome the blind area of a single sensor, and greatly improve the reliability of the system in extreme, harsh or variable light environments.

[0003] During the flight of the unmanned aerial vehicle, due to the abnormal flight attitude such as high-frequency jitter or sharp turning, and the movement of obstacles, the perception model of the radar sensor carried by the unmanned aerial vehicle may be distorted, causing the received radar signal to have artifacts such as false echoes. Such abnormal signals are easy to cause misjudgment of the perception system, for example, mistaking the false echoes caused by the jitter of the unmanned aerial vehicle itself as a front obstacle, and then triggering the flight control system to perform unnecessary emergency obstacle avoidance actions, which seriously damages the flight stability and safety. SUMMARY

[0004] In order to solve the technical problem of high misjudgment rate of the obstacle avoidance system caused by the distortion of the perception signal due to the abnormal flight attitude of the unmanned aerial vehicle or the movement of the obstacle, the purpose of the present application is to provide a multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception, and the technical solution adopted is as follows:

[0005] The present application provides a multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception, which comprises:

[0006] A data acquisition module for acquiring real-time actual state data and radar signals of the unmanned aerial vehicle at each time during flight;

[0007] A signal variation evaluation module for obtaining the signal variation degree of each time according to the frequency domain dispersion difference and the spectral pattern change degree of the radar signals of each time and the previous time, and the degree of change of the unmanned aerial vehicle attitude;

[0008] a data correction module, configured to adjust a state prediction model based on the signal variability to generate an optimized prediction model at each time point; obtain a model confidence weight at the current time point according to a deviation degree of a prediction error between predicted state data generated by the optimized prediction model at the current time point and actual state data at a previous time point relative to corresponding prediction errors at adjacent time points in a current analysis period, and determine optimal state data at the current time point;

[0009] an unmanned aerial vehicle obstacle avoidance module, configured to perform unmanned aerial vehicle obstacle avoidance based on the optimal state data.

[0010] Further, the signal variability at each time point is obtained by:

[0011] the signal interference degree at each time point is obtained according to the frequency domain dispersion difference and the frequency spectrum pattern change degree of the radar signal at each time point and the previous time point thereof;

[0012] attitude data of the unmanned aerial vehicle at each time point is obtained, the attitude data is standardized to obtain standard attitude data, and the absolute value of the difference between the same kind of standard attitude data at each time point and the previous time point thereof is calculated, and the cumulative sum of all absolute values is taken as the attitude change abruptness at each time point;

[0013] the signal variability at each time point is obtained according to the signal interference degree and the attitude change abruptness.

[0014] Further, the signal interference degree at each time point is obtained by:

[0015] the radar signal is subjected to time-frequency decomposition to obtain component signals of different frequencies;

[0016] the variance of the frequencies of all component signals of the radar signal is calculated and denoted as a frequency dispersion degree, and the absolute value of the difference between the frequency dispersion degrees at each time point and the previous time point thereof is taken as the frequency domain dispersion difference at each time point;

[0017] the frequencies of all component signals of the radar signal are sequentially arranged to obtain a frequency sequence, and the DTW value of the frequency sequence of the radar signal at each time point and the previous time point thereof is obtained and denoted as the frequency spectrum pattern change degree at each time point;

[0018] the signal interference degree at each time point is obtained according to the frequency domain dispersion difference and the frequency spectrum pattern change degree.

[0019] Further, the model confidence weight at the current time point is obtained by:

[0020] control data at each time point in the flight process of the unmanned aerial vehicle is obtained in real time, and a control vector is formed by all control data at the same time point;

[0021] The actual state data at the first time in the flight process of the unmanned aerial vehicle is taken as the optimal state data; for the time after the first time in the flight process of the unmanned aerial vehicle, the control vector of each time and the optimal state data of the previous time are input into the optimization prediction model of each time to obtain the predicted state data of each time;

[0022] The absolute value of the difference between the predicted state data of each time and the actual state data of the previous time is taken as the prediction error of each time; the mean of the prediction errors of all times in the current analysis period is calculated, and the ratio of the prediction error of the current time to the mean is normalized to obtain the model confidence weight of the current time.

[0023] Further, the optimization prediction model of each time is formed by weighting the state prediction model by the sum of the signal variability of each time and a constant 1.

[0024] Further, the determination of the optimal state data of the current time comprises:

[0025] The model confidence weight of the current time and the actual state data of the previous time are input into the Kalman filtering algorithm to obtain the optimal state data of the current time.

[0026] Further, the signal interference degree and the attitude change abruptness degree are positively correlated with the signal variability.

[0027] Further, the state prediction model is a nonlinear state transition function.

[0028] Further, the last time in the current analysis period is the previous time of the current time.

[0029] Further, the control data comprises linear velocity and angular velocity of the unmanned aerial vehicle.

[0030] The present application has the following advantages:

[0031] In the embodiment of the present application, the signal variability is obtained by fusing the radar signal features and the unmanned aerial vehicle attitude features to analyze the variation of the radar signal caused by the abnormal flight attitude of the unmanned aerial vehicle and the movement of the obstacle; the state prediction model in the Kalman filtering algorithm is adjusted based on the signal variability to compensate and correct the untrustworthy state observation value, thereby effectively suppressing the misjudgment and improving the robustness of the system under complex working conditions; by monitoring the dynamic change of the prediction error in real time, the model confidence weight is adaptively adjusted, the trust degree of the model prediction and sensor measurement is dynamically adjusted, so that the prediction algorithm can correct the sensor artifacts caused by the abnormal attitude of the unmanned aerial vehicle and the movement of the obstacle, thereby effectively avoiding the misjudgment of the perception system, and fundamentally improving the stability and safety of the unmanned aerial vehicle under complex flight conditions. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 The system structure diagram of the multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception provided by an embodiment of the present application;

[0034] Figure 2 The structure diagram of the signal interference evaluation module provided by an embodiment of the present application;

[0035] Figure 3 The computer device schematic diagram of the multi-modal unmanned aerial vehicle autonomous obstacle avoidance device based on space-time perception provided by an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the specific implementation, structure, features and effects of the multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception according to the present application are described in detail as follows. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0038] The specific scheme of the multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception provided by the present application is specifically described below with reference to the drawings.

[0039] Embodiment 1:

[0040] Please refer to Figure 1 , which shows the system block diagram of the multi-modal unmanned aerial vehicle autonomous obstacle avoidance system based on space-time perception provided by an embodiment of the present application. The system comprises: a data acquisition module 110, a signal variation evaluation module 120, a data correction module 130, and an unmanned aerial vehicle obstacle avoidance module 140.

[0041] The data acquisition module 110 is used to acquire the actual state data and radar signals at each moment in the flight process of the unmanned aerial vehicle in real time.

[0042] During the flight of the unmanned aerial vehicle, the laser radar carried thereby emits an electromagnetic wave signal of a specific modulation form at each time, takes the echo signal emitted back at the time as the radar signal at the corresponding time, and meanwhile, the actual state data of the unmanned aerial vehicle at each time. The actual state data is used to understand the position and motion of the unmanned aerial vehicle itself, and includes the three-dimensional coordinates, the pitch angle, the roll angle, the yaw angle, the linear velocity and the angular velocity of the unmanned aerial vehicle; wherein the pitch angle, the roll angle and the yaw angle are called attitude data, and are the basis for planning the motion of the unmanned aerial vehicle, and the linear velocity and the angular velocity represent the motion state of the unmanned aerial vehicle. Among them, the data of each dimension in the three-dimensional coordinates is a kind of actual state data, and the global positioning system is used to determine the three-dimensional coordinates of the unmanned aerial vehicle, and the inertial sensor is used to collect other kinds of state data.

[0043] In the embodiment of the application, the time interval between the two adjacent times is set to 1 second.

[0044] It should be noted that the horizontal axis of the radar signal represents the distance, and the vertical coordinate represents the echo amplitude.

[0045] The signal variation evaluation module 120 is configured to obtain the signal variation degree of each time according to the frequency domain dispersion difference and the spectrum mode change degree of the radar signals of each time and the previous time, and the degree of change of the attitude of the unmanned aerial vehicle.

[0046] The energy of the radar echo without interference is usually concentrated on certain specific and stable frequency components, such as the Doppler frequency of a specific motion speed, and the motion of the radar target in a short time is usually continuous and smooth, and the spectrum mode of the echo signal is also continuous and gradual, so that the frequency domain dispersion of the radar signals of adjacent times is relatively stable and the spectrum mode is relatively similar. However, the radar echo affected by interference will introduce many new and random frequency components, or cause the original main frequency to jitter and diffuse, resulting in a dispersed frequency distribution, and strong interference will change the composition of the frequency components, so that the frequency domain dispersion of the radar signals of adjacent times changes and the spectrum mode mutates. At the same time, the flight attitude of the unmanned aerial vehicle when it is abnormally jittered is different from the normal flight attitude, so that the received radar signal also has abnormal interference. Therefore, when analyzing the degree of interference of the radar signal at each time according to the frequency domain dispersion difference and the spectrum mode change degree of the radar signals of adjacent times, the degree of change of the attitude of the unmanned aerial vehicle can be combined to determine the signal interference degree, which greatly improves the perception robustness and decision correctness of the flight control system of the unmanned aerial vehicle in a complex environment.

[0047] The data correction module 130 is configured to adjust a state prediction model based on signal variability to generate an optimized prediction model at each time point; obtain a model confidence weight at the current time point according to a deviation degree of a prediction error between predicted state data generated by the optimized prediction model at the current time point and actual state data at a previous time point relative to corresponding prediction errors at adjacent time points in a current analysis period, and determine optimal state data at the current time point.

[0048] The signal variability is obtained by fusing radar signal features and unmanned aerial vehicle attitude features, analyzing abnormal conditions of the unmanned aerial vehicle abnormal flight attitude and the obstacle movement on the radar signal, adjusting a state prediction model in a Kalman filtering algorithm based on the signal variability, and compensating and correcting untrusted state observation values, so as to effectively suppress misjudgment and improve the robustness of the system under complex working conditions.

[0049] The Kalman gain of the traditional Kalman filtering algorithm is calculated based on fixed process noise covariance and measurement noise covariance, and the reliability of the model and the sensor is assumed to be constant. The abnormal flight attitude of the unmanned aerial vehicle and the obstacle movement will make the kinematic model very inaccurate, and the traditional fixed gain Kalman filtering cannot cope with dynamic changes, which will lead to divergence or serious deviation of the filtering result from the true value. The prediction error measures the prediction performance of the system model at the previous time point, the mean value of the prediction errors at all time points in the current analysis period represents the normal level of the model prediction performance, the deviation degree of the prediction error at the current time point relative to the normal performance level of the model is analyzed to determine the accuracy of the model prediction, and the model confidence weight is determined. The scheme can monitor the dynamic changes of the prediction error in real time, and adaptively determine the model confidence weight. The weight is equivalent to the Kalman gain, so that the filtering algorithm can automatically identify the sensor artifacts caused by the abnormal attitude of the unmanned aerial vehicle and the movement of the obstacle, thereby effectively avoiding the misjudgment of the perception system, and fundamentally improving the stability and safety of the unmanned aerial vehicle under complex flight conditions.

[0050] In one implementation manner of the embodiment of the present application, the state prediction model is a nonlinear state transition function.

[0051] In one implementation manner of the embodiment of the present application, the last time point in the current analysis period is the previous time point of the current time point, and the adjacent 10 minutes before the current time point are taken as the current analysis period of the current time point.

[0052] The unmanned aerial vehicle obstacle avoidance module 140 is configured to perform unmanned aerial vehicle obstacle avoidance based on the optimal state data.

[0053] The data collected by the autonomous obstacle avoidance system of the UAV can be divided into three categories: in addition to state data, environmental perception data and task and global data, wherein the environmental perception data includes the accurate distance from the UAV to the surrounding objects and the like, which is used to identify and locate obstacles; the task and global data includes the real-time position of the UAV in the global map, the predetermined flight path and the target point coordinates.

[0054] Firstly, the PPS pulse or hardware trigger signal is used to ensure that the timestamps of all sensor data are highly synchronized, laying the foundation for spatiotemporal consistency. Then, the system performs multi-layer sensor fusion, data-level fusion: the raw data collected by different sensors are fused, such as projecting the point cloud data obtained by the laser radar onto the two-dimensional image collected by the camera, giving each pixel depth information, and generating an RGB-D image. Decision-level fusion: each sensor first processes the data independently and generates preliminary perception results, such as the camera identifying a "pedestrian" and the laser radar detecting a "cylindrical obstacle", the decision layer synthesizes these results to make advanced inferences, such as "there is a pedestrian moving to the left 5 meters in front of the left". Feature-level fusion: using a deep learning model to perform end-to-end joint processing of raw data from different sensors such as images and point cloud data, and directly outputting the fused environmental perception results. Finally, the fused data is used to construct an environmental map, mainly including: local map: such as 2D grid map, 3D elevation map or Euclidean signed distance field, mainly used for real-time obstacle avoidance decision; global map: such as 3D point cloud map or semantic map, used for global path planning and advanced task understanding.

[0055] To illustrate the working process of the autonomous obstacle avoidance system of the UAV, the sudden encounter with a tree branch during the flight of the UAV is taken as an example:

[0056] The perception link: the binocular camera and the laser radar carried by the UAV detect the obstacle in front and generate point cloud data containing the tree branch in the sensor coordinate system. The visual algorithm identifies the image data and preliminarily judges it as "tree branch". The system immediately calls the optimal state data at the current time, such as attitude and position information, and converts the point cloud data of the tree branch from the sensor coordinate system to the unified global world coordinate system through coordinate transformation.

[0057] The environmental modeling link: the simultaneous localization and mapping algorithm updates the position information of the tree branch perceived at different times to the local voxel map or the Euclidean signed distance field in real time, and marks the corresponding voxel as "occupied". In the process of updating the map, the system uses the optimal state data of the sequence of consecutive time to calculate the displacement and rotation of the UAV between adjacent time, so as to ensure that the perceived tree branch point cloud is seamlessly and accurately registered to the correct position of the map.

[0058] Planning and decision-making link: the local path planner such as A* search algorithm detects that there is an obstacle in the predetermined flight path, that is, the voxel is marked as occupied, and the planner takes the various optimal state data at the current time as the initial state of trajectory generation, and calculates a new trajectory from the current state, bypassing the branches and rejoining the predetermined flight path on the updated map; at the same time, the Euclidean signed distance field provides the distance information of each position to the nearest obstacle, ensuring that the new trajectory planned always maintains a safe distance from the tree branches.

[0059] Control execution link: the flight control system receives the new trajectory instruction from the planner, and the controller will continuously compare the various optimal state data obtained at each time with the error signals such as the expected state at the corresponding time on the trajectory, and based on these errors, the PID control system or other advanced controllers in the flight control system calculate the required motor speed command to drive the UAV to accurately track the planned obstacle avoidance trajectory.

[0060] Loop iteration link: the entire obstacle avoidance process constitutes a high-frequency running closed-loop system. Each link of perception, planning and control relies on the various optimal state data at the current time or the previous time, and after a single processing period is completed, the system immediately enters the next processing period. In the next moment, the latest sensor data and the optimal state data of the previous moment will be jointly input into the Kalman filtering algorithm for a new round of prediction and update to generate new optimal state data, thereby starting a new round of obstacle avoidance cycle. This continuous self-iteration optimization mechanism ensures the accuracy, real-time performance and reliability of the decision-making and control of the UAV in a complex dynamic environment.

[0061] It should be noted that the algorithms and technologies involved in the above content are well-known to those skilled in the art, and will not be described here.

[0062] Please refer to Figure 2 , which shows a structure diagram of a signal interference evaluation module provided by an embodiment of the application, and the signal interference evaluation module comprises: a signal interference analysis unit 121, a posture change analysis unit 122, and a signal variation analysis unit 123.

[0063] The signal interference analysis unit 121 is used for obtaining the signal interference degree at each moment according to the frequency domain stability difference and the time domain continuity difference of the radar signal at each moment and the radar signal at the previous moment of each moment.

[0064] Preferably, in some possible implementation manners of the embodiment of the present application, the method for obtaining the signal interference degree comprises: performing time-frequency decomposition on the radar signal to obtain component signals of different frequencies; calculating the variance of the frequencies of all the component signals of the radar signal, denoted as the frequency dispersion degree; taking the absolute value of the difference between the frequency dispersion degree at each time and that at the previous time as the frequency domain dispersion difference degree at each time; arranging the frequencies of all the component signals of the radar signal in sequence to obtain a frequency sequence; obtaining the DTW value of the frequency sequence of the radar signal at each time and that at the previous time, denoted as the spectral pattern change degree at each time; and obtaining the signal interference degree at each time according to the frequency domain dispersion difference degree and the spectral pattern change degree. The Dynamic Time Warping (DTW) algorithm is a known technology to those skilled in the art, and will not be described here.

[0065] It should be noted that the frequency fluctuation degree represents the dispersion degree of the frequency distribution of the radar signal; if both the frequency domain dispersion difference degree and the spectral pattern change degree are greater, it means that the spectrum of the radar signal at adjacent times becomes more concentrated or dispersed, i.e., the change in the spectral dispersion is more dramatic, and the difference in the composition of the frequency components is greater, i.e., the degree of change in the spectral pattern is greater, so the radar signal at each time is more severely interfered by the abnormal flight attitude and the movement of the obstacle, resulting in a greater signal interference degree. Therefore, both the frequency domain dispersion difference degree and the spectral pattern change degree are positively correlated with the signal interference degree. In the embodiment of the present application, the product of the frequency domain dispersion difference degree and the spectral pattern change degree of the radar signal at each time is normalized to obtain the signal interference degree.

[0066] In the embodiment of the present application, the Sigmoid function is used for normalization processing, and other normalization methods such as function transformation and maximum-minimum normalization can also be selected, which are not limited here.

[0067] In one implementation manner of the embodiment of the present application, the empirical mode decomposition algorithm is selected, and other algorithms such as the integrated empirical mode decomposition algorithm and the wavelet decomposition algorithm can also be selected.

[0068] It should be noted that the frequency sorting manner in the frequency sequence corresponding to the radar signal at different times is the same, i.e., the frequencies in the sequence are arranged in descending order or ascending order.

[0069] The attitude change analysis unit 122 is configured to obtain the attitude data of the UAV at each time, perform standardization processing on the attitude data to obtain standard attitude data, calculate the absolute value of the difference between the same kind of standard attitude data at each time and that at the previous time, and take the cumulative sum of all the absolute values as the attitude change abruptness at each time.

[0070] It should be noted that the dimensions of the attitude data of different dimensions are different, and the attitude data needs to be standardized before analysis. In the embodiment of the present application, the range standardization method is selected for standardization processing, and the Z-score standardization method, the decimal standardization method and the sum normalization method can also be selected. If the difference between the same standard attitude data at each time and the previous time is greater, it indicates that the flight attitude change degree of the unmanned aerial vehicle at adjacent time is greater, and the attitude change abruptness is greater.

[0071] The signal variation analysis unit 123 is configured to obtain the signal variation degree of each time according to the signal interference degree and the attitude change abruptness.

[0072] It should be noted that if the signal interference degree and the attitude change abruptness are greater, it means that the interference degree of the radar signal by the abnormal flight attitude and the obstacle movement is greater, so that the sensor measurement value appears the characteristics such as artifacts, and the variation degree of the radar signal is greater. Therefore, the signal interference degree and the attitude change abruptness are positively correlated with the variation degree. In the embodiment of the present application, the product of the signal interference degree and the attitude change abruptness at each time is normalized to obtain the signal variation degree. In the embodiment, the maximum-minimum normalization is selected for normalization processing.

[0073] Preferably, in some possible implementation manners of the embodiment of the present application, the obtaining method of the optimization prediction model comprises: the optimization prediction model at each time is formed by weighting the state prediction model with the sum value of the signal variation degree at each time and the constant 1 as the weight. It should be noted that if the signal variation degree is greater, it indicates that the unmanned aerial vehicle changes dramatically, and the original state prediction model may be less accurate, so the uncertainty of the model needs to be increased, and then the signal variation degree is added to the state prediction model.

[0074] Preferably, in some possible implementation manners of the embodiment of the present application, the obtaining method of the model confidence weight comprises: real-time obtaining the control data at each time during the flight process of the unmanned aerial vehicle, and the control vector is constituted by all the control data at the same time; the actual state data at the first time during the flight process of the unmanned aerial vehicle is taken as the optimal state data; for the time after the first time during the flight process of the unmanned aerial vehicle, the control vector at each time and the optimal state data at the previous time are input into the optimization prediction model at each time to obtain the predicted state data at each time; the absolute value of the difference between the predicted state data at each time and the actual state data at the previous time is taken as the prediction error at each time; the mean value of the prediction errors of all times in the current analysis period is calculated, and the ratio of the prediction error at the current time to the mean value is normalized to obtain the model confidence weight at the current time. The Kalman filtering algorithm is well known to those skilled in the art, and will not be described here.

[0075] It should be noted that the prediction error measures the prediction performance of the system model at the last time, and the greater the prediction error, the more unreliable the model is in the last step of prediction. The mean of the prediction error of all times in the current analysis period is the normal level of the prediction performance of the model, and if the prediction error of the current time is greater than the mean, it means that the prediction performance of the model at the current time is much worse than the normal level, and the more unreliable the model is, the more the final result should believe the sensor measurement, and the greater the model confidence weight; on the contrary, it means that the prediction performance of the model at the current time is better, and the unmanned aerial vehicle is in an abnormal jitter state, indicating that the measurement value may have a false anomaly, and the sensor measurement value is extremely unreliable, and the gain should be reduced, so that the final result believes the model prediction more and ignores the unreliable measurement false anomaly.

[0076] In the embodiment of the application, the control data determines the position and attitude change of the unmanned aerial vehicle, including the linear velocity and angular velocity of the unmanned aerial vehicle, and other embodiments can also include the heading of the unmanned aerial vehicle, which is not limited here.

[0077] In the embodiment of the application, the Sigmoid function is used for normalization processing, and other normalization methods such as function transformation, maximum and minimum normalization can also be selected, which are not limited here.

[0078] In other embodiments of the application, before the system is formally run, a period of actual state data is collected, and the mean of the actual state data of all times in the period is taken as the optimal state data of the first time in the flight process of the unmanned aerial vehicle.

[0079] Preferably, in some possible implementation manners of the embodiment of the application, the optimal state data acquisition method comprises: inputting the model confidence weight of the current time and the actual state data of the previous time into the Kalman filtering algorithm to obtain the optimal state data of the current time. It should be noted that in the process of obtaining the optimal state data of each time by using the Kalman filtering algorithm, the prediction step uses the optimized prediction model for prediction, and the model confidence weight of each time is used to replace the Kalman gain in the Kalman filtering algorithm, and other contents remain unchanged. Since the length of the current analysis period is 10 minutes, the optimal state data can be obtained only after the unmanned aerial vehicle has flown for 10 minutes.

[0080] Thus, the application is completed.

[0081] Embodiment 2:

[0082] Figure 3 A computer device schematic diagram of a multi-modal unmanned aerial vehicle autonomous obstacle avoidance equipment based on space-time perception provided by an embodiment of the application. For example, Figure 3As shown, the computer device includes: a memory 201, a processor 202, and a computer program 203 stored in the memory 201 and running on the processor 202, wherein when the processor 202 executes the computer program 203, the computer device can execute any of the aforementioned spatiotemporal perception-based multimodal unmanned aerial vehicle autonomous obstacle avoidance systems.

[0083] Furthermore, this application also protects an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to execute the spatiotemporal awareness-based multimodal unmanned aerial vehicle autonomous obstacle avoidance system provided in this application.

[0084] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0085] It should be understood that the device provided in this embodiment is used to execute the above-described spatiotemporal awareness-based multimodal unmanned aerial vehicle autonomous obstacle avoidance system, and therefore can achieve the same effect as the above-described implementation method.

[0086] When using integrated units, the device may include a processing module and a storage module. When applied to a workpiece, the processing module can be used to control and manage the workpiece's operations. The storage module can be used to support the execution of program code by the workpiece.

[0087] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits contained in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module may be a memory.

[0088] Example 3:

[0089] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the spatiotemporal awareness-based multimodal unmanned aerial vehicle autonomous obstacle avoidance system provided in the above embodiment.

[0090] Among them, the device and the computer readable storage medium provided by the embodiments are used to execute the corresponding system provided above, so the beneficial effects that can be achieved refer to the beneficial effects of the corresponding system provided above, which will not be described here.

[0091] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0092] Each of the embodiments in the specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments.

[0093] The above only describes the preferred embodiments of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A spatio-temporal perception based multi-modal UAV autonomous obstacle avoidance system, characterized in that, The system comprises: a data acquisition module, configured to acquire actual state data and radar signals at each moment in real time during flight of the unmanned aerial vehicle; a signal variation evaluation module, configured to acquire signal variation degrees at each moment according to a frequency domain dispersion difference degree and a spectrum mode change degree of the radar signals at each moment and a previous moment and a degree of change in attitude of the unmanned aerial vehicle; a data correction module, configured to adjust a state prediction model based on the signal variation degrees, to generate an optimized prediction model at each moment, to acquire a model confidence weight at a current moment according to a deviation degree of a prediction error between predicted state data generated by the optimized prediction model at the current moment and actual state data at a previous moment relative to prediction errors corresponding to adjacent moments within a current analysis period, and to determine optimal state data at the current moment; an unmanned aerial vehicle obstacle avoidance module, configured to perform unmanned aerial vehicle obstacle avoidance based on the optimal state data. The signal variation degrees at each moment are acquired by: acquiring signal interference degrees at each moment according to the frequency domain dispersion difference degrees and the spectrum mode change degrees of the radar signals at each moment and a previous moment; acquiring attitude data of the unmanned aerial vehicle at each moment, performing standardization processing on the attitude data to obtain standard attitude data, calculating absolute values of differences between the same kind of standard attitude data at each moment and a previous moment, and taking a sum of all the absolute values as a degree of abrupt change in attitude at each moment; acquiring the signal variation degrees at each moment according to the signal interference degrees and the degree of abrupt change in attitude. The signal interference degrees at each moment are acquired by: performing time-frequency decomposition on the radar signals to obtain component signals of different frequencies; calculating variances of frequencies of all the component signals of the radar signals, denoted as frequency dispersion degrees, and taking absolute values of differences between the frequency dispersion degrees at each moment and a previous moment as frequency domain dispersion difference degrees at each moment; sequentially arranging frequencies of all the component signals of the radar signals to obtain a frequency sequence, acquiring a DTW value of the frequency sequence of the radar signals at each moment and a previous moment, denoted as a spectrum mode change degree at each moment; and acquiring the signal interference degrees at each moment according to the frequency domain dispersion difference degrees and the spectrum mode change degrees.

2. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 1, wherein, The model confidence weight at the current moment is acquired by: acquiring control data at each moment in real time during flight of the unmanned aerial vehicle, and constructing a control vector from all the control data at the same moment; taking actual state data at a first moment during flight of the unmanned aerial vehicle as optimal state data, inputting the control vector at each moment and the optimal state data at a previous moment into the optimized prediction model at each moment to obtain predicted state data at each moment for moments after the first moment during flight of the unmanned aerial vehicle; taking absolute values of differences between the predicted state data at each moment and the actual state data at a previous moment as prediction errors at each moment, calculating a mean value of the prediction errors at all the moments within a current analysis period, and performing normalization processing on a ratio of the prediction error at the current moment to the mean value to obtain the model confidence weight at the current moment.

3. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 1, wherein, The optimization prediction model of each time instant is weighted by a state prediction model with a sum value of the signal variation degree of each time instant and a constant 1 as a weight.

4. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 1, wherein, The determination of the optimal state data of the current time instant comprises: The model confidence weight of the current time instant is input into a Kalman filtering algorithm with the actual state data of the previous time instant to obtain the optimal state data of the current time instant.

5. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 1, wherein, The signal interference degree and the attitude change abruptness degree are positively correlated with the signal variation degree.

6. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 1, wherein, The state prediction model is a nonlinear state transition function.

7. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 1, wherein, The last time instant in the current analysis period is the previous time instant of the current time instant.

8. The multi-modal autonomous obstacle avoidance system for UAVs based on spatio-temporal perception according to claim 2, wherein, The control data comprises linear velocity and angular velocity of the unmanned aerial vehicle.

Images