A flight robot dynamic obstacle avoidance control method based on multi-modal fusion perception

Abstract

The application discloses a kind of dynamic obstacle avoidance control methods of flying robot based on multi-modal fusion perception, comprising: through multi-modal sensor being carried on flying robot, the multi-modal perception data of the environment around flying robot is collected;The multi-modal perception data is input into multi-modal feature fusion model, and hierarchical feature extraction and depth fusion are carried out, and multi-modal fusion perception result is output;The motion trajectory of dynamic obstacle is predicted by time series prediction model to multi-modal fusion perception result, and the predicted trajectory information of dynamic obstacle in future preset time length is output;According to the predicted trajectory information, the current flight state of flying robot is combined, and PID control algorithm is used to dynamically adjust control parameters in real time, generates obstacle avoidance control instruction and drives flying robot to execute dynamic obstacle avoidance flight.The application significantly improves the obstacle avoidance safety and flight stability of flying robot in complex dynamic environment.

Description

Technical Field

[0001] This invention relates to the field of flight robot navigation technology, and more specifically, to a dynamic obstacle avoidance control method for flight robots based on multimodal fusion perception. Background Technology

[0002] With the rapid development of drone technology, flying robots are increasingly being used in agricultural plant protection, logistics and distribution, disaster relief, and urban security. In these application scenarios, flying robots often need to perform autonomous flight missions in complex and dynamic environments, which places extremely high demands on their real-time obstacle avoidance capabilities. Especially in complex environments such as low-altitude urban areas, densely populated high-rise buildings, and severe weather, flying robots face multiple challenges, including the interplay of static and dynamic obstacles, sensor performance being affected by environmental interference, and the high real-time requirements for obstacle avoidance decisions.

[0003] Currently, existing obstacle avoidance technologies for flying robots have the following main shortcomings:

[0004] First, at the perception level, existing solutions mostly employ single-modal sensors or simple sensor data overlay, lacking deep feature-level fusion of multi-source heterogeneous information such as vision, LiDAR, and millimeter-wave radar. Visual sensors perform well in textured areas, but their performance degrades under low light and strong reflective conditions; LiDAR can provide accurate 3D geometric information, but its detection range is limited in rainy and foggy weather; millimeter-wave radar has advantages in adverse weather conditions, but its resolution is low. Existing technologies fail to fully leverage the complementary advantages of different sensor modalities, resulting in insufficient robustness in perception under complex environments.

[0005] Second, at the decision-making level, existing solutions mostly adopt reactive obstacle avoidance strategies based on the current position of obstacles, lacking the ability to predict the future trajectory of dynamic obstacles. When facing high-speed moving obstacles (such as birds or other aircraft), making obstacle avoidance decisions based solely on the current position often leads to response lag, making it difficult to achieve safe and smooth obstacle avoidance flight.

[0006] Third, at the control level, existing solutions generally adopt an architecture that separates offline path planning from online PID tracking control. The reference trajectory generated by offline planning does not take into account the nonlinear dynamic constraints of the flying robot, resulting in a large tracking error; the PID controller parameters are fixed, making it difficult to adapt to the differentiated requirements of control response speed and stability in different obstacle avoidance scenarios.

[0007] The above shortcomings urgently need to be improved. Summary of the Invention

[0008] To overcome the technical problems of insufficient fusion of multi-source sensors, lack of dynamic obstacle prediction, and separation of planning and control in existing obstacle avoidance technologies for flying robots, this invention provides a dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception.

[0009] The technical solution of this invention is as follows:

[0010] This invention provides a dynamic obstacle avoidance control method for a flying robot based on multimodal fusion perception, comprising the following steps:

[0011] Step S1, Multimodal Data Acquisition: Collect multimodal perception data of the environment surrounding the flying robot using multimodal sensors mounted on the flying robot;

[0012] Step S2, Multimodal Feature Fusion Perception: Input the multimodal perception data into the multimodal feature fusion model, perform hierarchical feature extraction and deep fusion, and output the multimodal fusion perception result;

[0013] Step S3, Dynamic obstacle trajectory prediction: The multimodal fusion perception results are used to predict the motion trajectory of the dynamic obstacle through a time-series prediction model, and the predicted trajectory information of the dynamic obstacle within a preset time period is output.

[0014] Step S4, Adaptive Parameter Control: Based on the predicted trajectory information and the current flight state of the flying robot, the control parameters are dynamically adjusted in real time using a PID control algorithm to generate obstacle avoidance control commands and drive the flying robot to perform dynamic obstacle avoidance flight.

[0015] As a preferred embodiment of the present invention, the multimodal sensor includes a visual sensor, a lidar, and a millimeter-wave radar; the multimodal sensing data includes visual image data, lidar point cloud data, and millimeter-wave radar echo data.

[0016] As a preferred embodiment of the present invention, the multimodal feature fusion model adopts a hierarchical fusion architecture, including:

[0017] The first layer is used to perform noise reduction and enhancement processing on the visual image data, filtering and downsampling processing on the lidar point cloud data, and clutter suppression processing on the millimeter-wave radar echo data.

[0018] The second layer is used to extract semantic features of the visual image data through a convolutional neural network, extract geometric features of the lidar point cloud data through a point cloud feature extraction network, and extract motion features of the millimeter-wave radar echo data through a time-frequency feature extraction network.

[0019] The third layer is used to employ an attention-based feature fusion network to adaptively weight and fuse the semantic features of the visual image data, the geometric features of the lidar point cloud data, and the motion features of the millimeter-wave radar echo data to generate a unified environmental perception representation.

[0020] As a preferred embodiment of the present invention, the multimodal feature fusion model adaptively adjusts the feature weights according to environmental conditions:

[0021] The current lighting conditions are determined by the illumination detection and analysis module. When the illumination intensity is lower than the threshold, the weight of the semantic features of the visual image data is reduced, while the weight of the geometric features of the lidar point cloud data and the motion features of the millimeter-wave radar echo data is increased.

[0022] The meteorological sensing and analysis module determines the current weather conditions. When rain or fog is detected, the weight of the motion characteristics of the millimeter-wave radar echo data is increased, while the weight of the geometric characteristics of the lidar point cloud data is decreased.

[0023] The texture analysis module determines the richness of scene textures. When the texture features are rich, the weight of the semantic features of the visual image data is increased, while the weight of the geometric features of the lidar point cloud data and the motion features of the millimeter-wave radar echo data is decreased.

[0024] As a preferred embodiment of the present invention, the predicted trajectory information includes the motion path, speed change trend and position prediction information.

[0025] As a preferred embodiment of the present invention, the time series prediction model adopts a sequence prediction model based on a long short-term memory network or a Transformer architecture.

[0026] As a preferred embodiment of the present invention, the PID control algorithm is an adaptive parameter control method, which dynamically adjusts the proportional coefficient Kp, integral coefficient Ki, and derivative coefficient Kd in real time according to the following operating conditions:

[0027] The proportional coefficient Kp and the differential coefficient Kd are increased in high-speed obstacle avoidance conditions;

[0028] In close-range obstacle avoidance conditions, the proportional coefficient Kp is reduced and the integral coefficient Ki is increased.

[0029] As a preferred embodiment of the present invention, when the relative speed of the dynamic obstacle is greater than 5 m / s and the relative distance between the dynamic obstacles is greater than 10 m, it is determined to be the high-speed obstacle avoidance condition; when the relative distance between the dynamic obstacles is not greater than 5 m, it is determined to be the close-range obstacle avoidance condition.

[0030] As a preferred embodiment of the present invention, step S4 further includes: generating a local obstacle avoidance trajectory considering dynamic constraints online based on the predicted trajectory information and the nonlinear dynamic model of the flying robot, and performing trajectory tracking control through an adaptive PID controller.

[0031] As a preferred embodiment of the present invention, the flying robot is a multi-rotor drone, a fixed-wing drone, or a compound-wing drone.

[0032] According to the above-described solution, the beneficial effects of this invention are as follows:

[0033] This invention leverages a hierarchical multimodal feature fusion architecture and an adaptive weight adjustment mechanism to fully utilize the complementary advantages of various sensor modalities, including vision, lidar, and millimeter-wave radar. This effectively overcomes the performance degradation problem of single sensors in complex environments, significantly improving the environmental perception robustness of flying robots under harsh conditions such as low light, strong reflection, rain, and fog. Simultaneously, by introducing a temporal prediction model to accurately predict the future trajectory of dynamic obstacles in the short term, the flying robot acquires predictive obstacle avoidance capabilities, effectively solving the problem of lag in traditional reactive obstacle avoidance strategies and significantly improving obstacle avoidance safety and flight stability when facing high-speed moving obstacles. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating a dynamic obstacle avoidance control method for a flying robot based on multimodal fusion perception in one embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the structure of a dynamic obstacle avoidance control device for a flying robot based on multimodal fusion perception in one embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0037] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] It should be noted that the terms "comprising" and "having," and any variations thereof, in the specification and claims of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0039] like Figure 1 As shown, the present invention provides a dynamic obstacle avoidance control method for a flying robot based on multimodal fusion perception, comprising the following steps:

[0040] Step S1, Multimodal Data Acquisition: Multimodal perception data of the surrounding environment of the flying robot is acquired through multimodal sensors mounted on the flying robot.

[0041] Specifically, multimodal sensors include visual sensors, LiDAR, and millimeter-wave radar; multimodal sensing data includes visual image data, LiDAR point cloud data, and millimeter-wave radar echo data. Through a hardware synchronization triggering mechanism, spatiotemporally aligned data acquisition from visual sensors, LiDAR, and millimeter-wave radar is achieved, eliminating fundamental errors in multi-source information fusion at the data source level and providing reliable raw data support for subsequent high-precision feature fusion.

[0042] Step S2, Multimodal Feature Fusion Perception: Input the multimodal perception data into the multimodal feature fusion model, perform hierarchical feature extraction and deep fusion, and output the multimodal fusion perception result;

[0043] Specifically, the multimodal feature fusion model adopts a layered fusion architecture, including:

[0044] The first layer performs noise reduction preprocessing on the raw data. For visual image data, Gaussian filtering and histogram equalization are used for noise reduction and enhancement to solve the problems of image blurring and insufficient contrast in low-light and highly reflective scenes. For LiDAR point cloud data, statistical filtering is used to remove outliers and voxel grid downsampling is performed to remove noisy outliers, simplify redundant point clouds, and retain accurate three-dimensional geometric information. For millimeter-wave radar echo data, a constant false alarm rate (CFAR) detection algorithm is used for clutter suppression to effectively suppress rain, fog, and clutter interference. The first layer specifically compensates for the original defects of each sensor and significantly improves the quality of single-source data.

[0045] The second layer independently extracts features from each modality. Semantic features from visual image data are extracted using a convolutional neural network (such as the ResNet-50 backbone network); geometric features from LiDAR point cloud data are extracted using a point cloud feature extraction network (such as the PointNet++ network); and motion features from millimeter-wave radar echo data are extracted using a time-frequency feature extraction network (such as a one-dimensional convolutional network). This second layer employs corresponding networks adapted to visual semantics, point cloud geometry, and radar motion feature extraction. Unlike traditional general feature extraction methods, it accurately mines the core effective features of each modality, preserving key information such as image semantic details, obstacle 3D contour dimensions, and target motion speed. This maximizes the utilization of single-modality features and avoids problems such as ineffective feature extraction and information loss.

[0046] The third layer completes the deep fusion of multiple features and the removal of redundant information. It adopts a feature fusion network based on the attention mechanism (Multi-HeadSelf-Attention) to adaptively weight and fuse the semantic features of visual image data, the geometric features of LiDAR point cloud data, and the motion features of millimeter-wave radar echo data to generate a unified environmental perception representation.

[0047] In the third layer of feature fusion, the feature weights are adaptively adjusted according to the environmental conditions:

[0048] The current lighting conditions are determined by the illumination detection and analysis module. When the illumination intensity is below the threshold, the weight of the semantic features of the visual image data is reduced, while the weight of the geometric features of the lidar point cloud data and the motion features of the millimeter-wave radar echo data are increased. For example, the weight of the semantic features of the visual image data is reduced to 0.3, the weight of the geometric features of the lidar point cloud data is increased to 0.4, and the weight of the motion features of the millimeter-wave radar echo data is increased to 0.3.

[0049] The meteorological perception and analysis module determines the current weather conditions. When severe weather such as rain and fog is detected, the weight of motion features in millimeter-wave radar echo data is increased, while the weight of geometric features in lidar point cloud data is decreased. For example, the weight of motion features in millimeter-wave radar echo data is increased to 0.6, the weight of geometric features in lidar point cloud data is decreased to 0.2, and the weight of semantic features in visual image data is decreased to 0.2.

[0050] The texture analysis module determines the richness of scene textures. When the texture features are rich, the weight of the semantic features of the visual image data is increased, while the weight of the geometric features of the LiDAR point cloud data and the motion features of the millimeter-wave radar echo data is decreased, thereby improving the ability to understand the semantics of the environment. For example, the weight of the semantic features of the visual image data is increased to 0.5, the weight of the geometric features of the LiDAR point cloud data is decreased to 0.3, and the weight of the motion features of the millimeter-wave radar echo data is decreased to 0.2.

[0051] In this embodiment, the texture analysis module incorporates a texture richness evaluation model to quantitatively determine scene textures, providing accurate data support for adaptive adjustment of visual feature weights. The texture richness evaluation model extracts four core texture feature parameters based on the image gray-level co-occurrence matrix (GLCM): contrast, correlation, entropy, and uniformity. A comprehensive texture richness scoring formula is constructed through weighted fusion: Texture richness score S = α × contrast + β × entropy + γ × correlation - δ × uniformity, where α, β, γ, and δ are fixed weighting coefficients, and their optimal values ​​are determined through training on a massive scene dataset. The model's workflow is as follows: First, the preprocessed visual image is grayscaled and segmented for sampling, extracting texture feature parameters for each image block; then, the parameters are substituted into the scoring formula to calculate the global texture richness normalized score, with a score range of 0 to 1; two-level judgment thresholds are set: when S ≥ 0.6, the scene is judged as texture-rich; when S < 0.6, the scene is judged as texture-sparse. This texture richness evaluation model can accurately identify texture-rich scenes such as buildings, vegetation, and traffic signs, as well as texture-sparse scenes such as the sky, solid-color walls, and foggy blank scenes. It has high accuracy and strong real-time performance, and can be adapted to the perception needs of high-speed dynamic flight of flying robots.

[0052] Multi-type detection module result conflict handling mechanism: In the process of multi-modal feature fusion detection, the illumination detection and analysis module, meteorological perception analysis module, and texture analysis module may encounter conflicting judgment results and overlapping operating conditions. This invention sets up a hierarchical priority and conflict adaptive resolution mechanism to solve the problem of contradictory outputs from multiple modules and ensure the rationality of weight adjustment. The module priorities from high to low are: meteorological perception analysis module > illumination detection and analysis module > texture analysis module, with severe weather conditions having the highest priority. When multiple module results conflict, the judgment result of the high-priority module is responded to first, while the auxiliary correction of the low-priority module is also taken into account. For example, when severe weather conditions such as rain and fog (high priority) overlap with low illumination scenes (low priority), the high weight configuration of millimeter-wave radar is prioritized, while the weight of lidar is slightly increased and the visual weight is finely adjusted to avoid extreme judgments in a single operating condition; when sunny, high-brightness, sparse-texture scenes conflict with normal illumination scenes, the illumination judgment result is the core, the weight of visual features is weakened, and the basic perception weights of lidar and millimeter-wave radar are retained. At the same time, a conflict detection threshold is set. When the difference between the weight adjustment instructions output by multiple modules exceeds the preset threshold, the attention weight smoothing correction strategy is activated. Extreme weight assignments are abandoned, and the optimal fusion weight is output through a multi-feature weighting compromise scheme, so as to prevent the problem of sudden changes in perception output and detection failure caused by module judgment conflicts.

[0053] Environmental Condition Switching Output Continuity Guarantee Mechanism: Addressing the issues of frequent switching between lighting, weather, and scene textures during flight, which can lead to abrupt changes in modal weights, jumps in perception output, and flight attitude jitter in flying robots, this invention adds a smooth transition mechanism to ensure the continuity and stability of perception output and obstacle avoidance decisions when switching between different environmental conditions. Abandoning the traditional hard threshold switching method, a sliding window gradual adjustment strategy is adopted. A 5-frame perception data sliding window is set. When an environmental condition is detected to cross a threshold, instead of directly replacing the modal weight parameters, the feature weights of vision, LiDAR, and millimeter-wave radar are finely adjusted frame by frame according to a linear inter-frame transition method. The transition between the old and new condition weights is completed within the 5-frame window, with a single frame weight adjustment amplitude not exceeding 0.05. Simultaneously, a condition-locking anti-shake mechanism is set up. For instantaneous environmental disturbances (flickering light and shadow, short-term fog interference), the same condition feature must be detected for 3 consecutive frames to determine a valid condition switch, avoiding perception output oscillations caused by frequent erroneous switches. This mechanism can effectively solve the problems of lag during working condition switching and abrupt changes in perception in complex dynamic scenarios, ensuring the smoothness of the obstacle avoidance process of the flying robot.

[0054] In this embodiment, the third layer abandons the traditional simple splicing and fixed-weight fusion method. Instead, it automatically filters effective features and eliminates redundant and conflicting information through an attention mechanism to generate a unified, high-dimensional global environmental representation. Simultaneously, it dynamically adjusts the weights of each modality based on environmental conditions: in low-light scenarios, it weakens visual dependence and strengthens radar and laser perception; in rainy and foggy weather, it prioritizes the all-weather advantage of millimeter-wave radar; and in scenes with rich textures, it enhances visual semantic understanding. This effectively solves the industry pain points of poor environmental adaptability of single sensors and insufficient depth of multi-source information fusion, achieving accurate perception across all scenarios and weather conditions without blind spots, and significantly reducing the rate of missed and false detections of obstacles in complex and harsh environments.

[0055] Step S3, Dynamic obstacle trajectory prediction: The multimodal fusion perception results are used to make short-term accurate predictions of the motion trajectory of dynamic obstacles through a time-series prediction model, and the predicted trajectory information of dynamic obstacles within a preset time period is output, including motion path, speed change trend and position prediction information.

[0056] Specifically, the temporal prediction model employs a sequence prediction model based on Long Short-Term Memory (LSTM) networks, containing two layers of LSTM units (128 hidden nodes per layer). Through an encoder-decoder structure, it accurately captures the nonlinear motion patterns of dynamic obstacles such as birds and low-altitude aircraft, outputting predicted trajectory information for 40 frames within the next two seconds, thus achieving a technological upgrade from passive obstacle avoidance to active predictive obstacle avoidance. In other embodiments, the temporal prediction model also employs a sequence prediction model based on the Transformer (Temporal Fusion Transformer, TFT) architecture. This model automatically learns the importance of different input features through a variable selection network, captures long-distance dependencies in the trajectory sequence through a multi-head self-attention mechanism, and processes multi-scale temporal information through a gated residual network, adapting to complex motion conditions such as obstacle speed changes and direction changes.

[0057] Step S4, Adaptive Parameter Control: Based on the predicted trajectory information and the current flight state of the flying robot (position, speed, attitude angle, angular velocity), the control parameters are dynamically adjusted in real time using a PID control algorithm to generate obstacle avoidance control commands and drive the flying robot to perform dynamic obstacle avoidance flight. At the same time, based on the predicted trajectory information and the nonlinear dynamic model of the flying robot, a local obstacle avoidance trajectory considering dynamic constraints is generated online, and trajectory tracking control is performed through an adaptive PID controller to achieve tight coupling and coordination between planning and control.

[0058] Specifically, the PID control algorithm is an adaptive parameter control method that dynamically adjusts the proportional coefficient Kp, integral coefficient Ki, and derivative coefficient Kd in real time according to the following operating conditions:

[0059] When the relative speed of a dynamic obstacle is greater than 5 m / s and the relative distance between dynamic obstacles is greater than 10 m, it is determined to be a high-speed obstacle avoidance condition, and the proportional coefficient Kp and the derivative coefficient Kd are increased; for example, the proportional coefficient Kp is set to 2.5, the integral coefficient Ki is 0.3, and the derivative coefficient Kd is 1.2 to enhance the system's response speed and anticipatory control capability.

[0060] When the relative distance to a dynamic obstacle is no more than 5m, it is determined to be a close-range obstacle avoidance condition. The proportional coefficient Kp is reduced and the integral coefficient Ki is increased. For example, the proportional coefficient Kp is set to 1.2, the integral coefficient Ki is set to 0.8, and the derivative coefficient Kd is set to 0.6 to enhance control stability and suppress overshoot.

[0061] When the pitch angle change rate |dθ / dt| of the flying robot is greater than 15° / s, the proportional coefficient Kp is dynamically reduced to 80% of the original value to prevent attitude oscillation.

[0062] In this embodiment, the tightly coupled collaborative architecture of planning and control takes multimodal fusion perception results and obstacle prediction trajectories as inputs. It embeds hardware constraints such as nonlinear dynamic constraints, attitude limits, velocity limits, and acceleration limits of the flying robot into the local trajectory planning stage in real time, generating feasible obstacle avoidance trajectories that conform to the robot's flight characteristics from the source. This avoids the problem in traditional architectures where planned trajectories exceed the robot's dynamic limits and cannot be accurately tracked. The characteristics of this tightly coupled collaborative architecture include: first, real-time linkage, with trajectory planning frequency synchronized with control frequency. The control end provides real-time feedback on trajectory tracking errors and attitude fluctuation data, correcting the trajectory curvature and flight speed parameters of the planning end; second, constraint adaptability, binding the robot's nonlinear dynamic model throughout the process to adapt to the dynamic constraints of extreme conditions such as high-speed flight, sharp turns, and close-range obstacle avoidance; and third, closed-loop optimality, forming a closed-loop control link of perception prediction, trajectory planning, tracking control, error feedback, and trajectory correction, significantly reducing trajectory tracking errors and improving obstacle avoidance accuracy and flight stability.

[0063] This embodiment presents a dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception. Through a hierarchical multimodal feature fusion architecture and an adaptive weight adjustment mechanism, it fully leverages the complementary advantages of vision, LiDAR, and millimeter-wave radar sensors, effectively overcoming the performance degradation problem of single sensors in complex environments. This significantly improves the environmental perception robustness of the flying robot under harsh conditions such as low light, strong reflection, rain, and fog. By adding a continuity guarantee mechanism for working condition switching and a multi-module conflict resolution mechanism, it solves the problems of perception output jumps and contradictory working condition judgments in complex dynamic scenes, greatly improving environmental adaptability and perception stability. Finally, by constructing a quantized texture evaluation model, it achieves accurate scene texture recognition, providing a basis for adaptive adjustment of modal weights. The system provides comprehensive data support; by introducing a time-series prediction model to accurately predict the future trajectory of dynamic obstacles in the short term, the flying robot possesses predictive obstacle avoidance capabilities, effectively solving the problem of lag in response of traditional reactive obstacle avoidance strategies, and significantly improving obstacle avoidance safety and flight stability when facing high-speed moving obstacles; through a tightly coupled planning and control architecture, binding robot dynamic constraints, closed-loop collaboration between trajectory planning and motion control is achieved, coupled with a complete adaptive PID parameter dynamic adjustment process, which dynamically optimizes control parameters in real time according to the obstacle avoidance scenario, effectively reducing tracking errors, and achieving adaptive adaptation of control response speed and stability under different obstacle avoidance scenarios, comprehensively improving the overall dynamic obstacle avoidance performance of the flying robot in complex environments.

[0064] In one embodiment, the flying robot is a multi-rotor drone, a fixed-wing drone, or a compound-wing drone.

[0065] like Figure 2As shown, one embodiment of the present invention provides a dynamic obstacle avoidance control device for a flying robot based on multimodal fusion perception, comprising:

[0066] The multimodal sensor module 1 includes a vision sensor, a lidar and a millimeter-wave radar, used to collect multimodal perception data of the environment around the flying robot;

[0067] The multimodal feature fusion perception module 2 is used to perform hierarchical feature extraction and deep fusion on multimodal perception data and output multimodal fusion perception results.

[0068] The dynamic obstacle trajectory prediction module 3 is used to make short-term accurate predictions of the motion trajectory of dynamic obstacles based on the multimodal fusion perception results and through a time-series prediction model. It outputs the predicted trajectory information of the dynamic obstacle within a preset time period in the future, including the motion path, speed change trend and position prediction information.

[0069] The adaptive parameter control module 4 is used to dynamically adjust the control parameters in real time based on the predicted trajectory information and the current flight state of the flying robot (position, speed, attitude angle, angular velocity), and generate obstacle avoidance control commands by using a PID control algorithm.

[0070] Flight execution module 5 is used to receive obstacle avoidance control commands and drive the flying robot to perform dynamic obstacle avoidance flight.

[0071] The dynamic obstacle avoidance control device for flying robots based on multimodal fusion perception provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so they will not be described again here.

[0072] Figure 3 This is a schematic diagram of the electronic device that implements the dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to the present invention.

[0073] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a dynamic obstacle avoidance control program for a flying robot based on multimodal fusion perception.

[0074] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a dynamic obstacle avoidance control program for a flight robot based on multimodal fusion perception) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0075] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a dynamic obstacle avoidance control program for a flight robot based on multimodal fusion perception, but also to temporarily store data that has been output or will be output.

[0076] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0077] Communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), and optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0078] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0079] For example, a power supply, although not shown, may also include a power source (such as a battery) to power various components. Preferably, the power supply can be logically connected to at least one processor 10 via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power sources, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0080] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0081] The memory 11 in the electronic device stores a dynamic obstacle avoidance control program for a flying robot based on multimodal fusion perception, which is a combination of multiple computer programs.

[0082] For the specific implementation method of the above computer program by processor 10, please refer to Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0083] Furthermore, if the modules / units integrated into an electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, a computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0084] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following functions: Figure 1 The flowchart illustrates the dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception.

[0085] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0086] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0088] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0089] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0090] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0091] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0092] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0093] The present invention has been described above with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A dynamic obstacle avoidance control method for a flying robot based on multimodal fusion perception, characterized in that, Includes the following steps: Step S1, Multimodal Data Acquisition: Collect multimodal perception data of the environment surrounding the flying robot using multimodal sensors mounted on the flying robot; Step S2, Multimodal Feature Fusion Perception: Input the multimodal perception data into the multimodal feature fusion model, perform hierarchical feature extraction and deep fusion, and output the multimodal fusion perception result; Step S3, Dynamic obstacle trajectory prediction: The multimodal fusion perception results are used to predict the motion trajectory of the dynamic obstacle through a time-series prediction model, and the predicted trajectory information of the dynamic obstacle within a preset time period is output. Step S4, Adaptive Parameter Control: Based on the predicted trajectory information and the current flight state of the flying robot, the control parameters are dynamically adjusted in real time using a PID control algorithm to generate obstacle avoidance control commands and drive the flying robot to perform dynamic obstacle avoidance flight.

2. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 1, characterized in that, The multimodal sensors include visual sensors, lidar, and millimeter-wave radar; the multimodal sensing data includes visual image data, lidar point cloud data, and millimeter-wave radar echo data.

3. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 2, characterized in that, The multimodal feature fusion model adopts a hierarchical fusion architecture, including: The first layer is used to perform noise reduction and enhancement processing on the visual image data, filtering and downsampling processing on the lidar point cloud data, and clutter suppression processing on the millimeter-wave radar echo data. The second layer is used to extract semantic features of the visual image data through a convolutional neural network, extract geometric features of the lidar point cloud data through a point cloud feature extraction network, and extract motion features of the millimeter-wave radar echo data through a time-frequency feature extraction network. The third layer is used to employ an attention-based feature fusion network to adaptively weight and fuse the semantic features of the visual image data, the geometric features of the lidar point cloud data, and the motion features of the millimeter-wave radar echo data to generate a unified environmental perception representation.

4. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 3, characterized in that, The multimodal feature fusion model adaptively adjusts feature weights according to environmental conditions: The current lighting conditions are determined by the illumination detection and analysis module. When the illumination intensity is lower than the threshold, the weight of the semantic features of the visual image data is reduced, while the weight of the geometric features of the lidar point cloud data and the motion features of the millimeter-wave radar echo data is increased. The meteorological sensing and analysis module determines the current weather conditions. When rain or fog is detected, the weight of the motion characteristics of the millimeter-wave radar echo data is increased, while the weight of the geometric characteristics of the lidar point cloud data is decreased. The texture analysis module determines the richness of scene textures. When the texture features are rich, the weight of the semantic features of the visual image data is increased, while the weight of the geometric features of the lidar point cloud data and the motion features of the millimeter-wave radar echo data is decreased.

5. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 1, characterized in that, The predicted trajectory information includes the motion path, speed change trend, and position prediction information.

6. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 1, characterized in that, The time series prediction model adopts a sequence prediction model based on long short-term memory network or Transformer architecture.

7. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 1, characterized in that, The PID control algorithm is an adaptive parameter control method that dynamically adjusts the proportional coefficient Kp, integral coefficient Ki, and derivative coefficient Kd in real time according to the following operating conditions: The proportional coefficient Kp and the differential coefficient Kd are increased in high-speed obstacle avoidance conditions; In close-range obstacle avoidance conditions, the proportional coefficient Kp is reduced and the integral coefficient Ki is increased.

8. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 7, characterized in that, When the relative speed of the dynamic obstacle is greater than 5 m / s and the relative distance between the dynamic obstacles is greater than 10 m, it is determined to be the high-speed obstacle avoidance condition; when the relative distance between the dynamic obstacles is not greater than 5 m, it is determined to be the close-range obstacle avoidance condition.

9. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 1, characterized in that, Step S4 also includes: generating a local obstacle avoidance trajectory that takes into account dynamic constraints online based on the predicted trajectory information and the nonlinear dynamic model of the flying robot, and performing trajectory tracking control through an adaptive PID controller.

10. The dynamic obstacle avoidance control method for flying robots based on multimodal fusion perception according to claim 1, characterized in that, The flying robot is a multi-rotor drone, a fixed-wing drone, or a compound-wing drone.

Images