An unmanned aerial vehicle immersive operation simulation gimbal

Abstract

The application discloses a kind of unmanned plane immersive operation simulation simulation head, it is related to unmanned plane technical field.The head includes: the environment building module of constructing simulation simulation environment;The head control module of simulating the movement mode of head and carrying out head control;The flight control module of simulating the flight state of unmanned plane and carrying out flight control;Immersion feedback module is analyzed operating data and environmental data to obtain feedback information and adaptively adjusts simulation operating environment;Multi-user collaborative module supports multiple operators to participate in simulation training;Personalized training module generates personalized training tasks.The application combines virtual reality, augmented reality, sensor fusion and the synergistic effect of high-precision control, can provide real and accurate immersive simulation experience, to improve the operation level of operator and reduce training risk, also can improve the head control ability and stability, to achieve more efficient training effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicles, and particularly relates to an immersive operation simulation gimbal for unmanned aerial vehicles. BACKGROUND

[0002] In recent years, unmanned aerial vehicle technology has been widely applied in civil, military, agricultural, photography, logistics and other fields. The flight performance, stability and application scenarios of unmanned aerial vehicles are constantly improving, which increases the technical requirements for flight operators. In order to ensure that the pilot can operate the unmanned aerial vehicle skillfully, especially in complex environments, it is necessary to develop efficient and low-risk training tools. Flight simulation technology, as an important means of flight training, is widely used in civil aviation, military and unmanned aerial vehicle fields. Traditional flight simulators mainly provide virtual environment simulation through computer graphics display and controllers, but often lack immersion in the operation experience. With the maturity of virtual reality (VR), augmented reality (AR) and other technologies, flight simulation technology gradually develops towards higher immersion and interactivity, providing a more realistic control experience.

[0003] Traditional unmanned aerial vehicle operation training relies mainly on simulators and actual flights. However, in actual flights, operators face high costs and high risks. In order to overcome these challenges, immersive training systems have emerged. Through virtual reality technology, operators can simulate real flight scenarios, conduct gimbal operation and flight control training in a safe environment, avoiding the risks of actual flights and improving training efficiency. The gimbal is an important part of the unmanned aerial vehicle, used to stabilize the camera and ensure the stability of the shooting picture during flight. With the increasing demand for unmanned aerial vehicle photography, gimbal technology has been continuously improved, especially in terms of stability and control accuracy. Modern unmanned aerial vehicle gimbals not only support gimbal angle adjustment, but also have automation functions such as target tracking and object recognition. Currently, there are some unmanned aerial vehicle simulation systems on the market, which mainly focus on flight paths, control systems and environmental simulation, but few involve precise simulation of gimbal operation. As a core part of unmanned aerial vehicle applications, gimbal operation requires operators to have excellent control skills. Therefore, it is particularly important to develop a system with immersive simulation function specifically for unmanned aerial vehicle gimbal operation. SUMMARY

[0004] The purpose of the present application is to provide an immersive operation simulation gimbal for unmanned aerial vehicles, which can be realized by the following technical solutions:

[0005] The present application provides an immersive operation simulation gimbal for unmanned aerial vehicles, which comprises:

[0006] An environment construction module is used to construct a simulated simulation environment and obtain environment data of the simulation environment.

[0007] gimbal control module: for simulating the motion mode of the gimbal and performing gimbal control, as well as processing gimbal shooting images and gimbal control data;

[0008] flight control module: for simulating the flight state of the unmanned aerial vehicle and performing flight control;

[0009] immersive feedback module: for obtaining operation data of the operator in the simulation operation environment, obtaining feedback information by analyzing the operation data and the environment data, and adaptively adjusting the simulation operation environment according to the feedback information;

[0010] multi-user collaboration module: for establishing a multi-user collaboration mechanism to support multiple operators to participate in simulation training together;

[0011] personalized training module: for real-time collection and recording of various data in the process of flight control and gimbal control, comprehensive analysis and evaluation combined with the training performance of the operator, and generation of personalized training tasks for the operator according to the analysis and evaluation results;

[0012] The gimbal control module includes a gimbal motion unit, a gimbal control unit, an optical anti-shake unit, an electronic anti-shake unit, an image processing unit and a data processing unit.

[0013] The gimbal motion unit is configured to simulate the motion mode of the gimbal in the simulation operation environment, and the motion mode includes gimbal pitch motion, gimbal yaw motion and gimbal roll motion.

[0014] The gimbal control unit is configured to perform multi-dimensional control of the gimbal in the simulation operation environment.

[0015] The optical anti-shake unit is configured to perform optical anti-shake processing when the gimbal is shooting.

[0016] The electronic anti-shake unit is configured to perform electronic anti-shake processing on the video images shot by the gimbal.

[0017] The image processing unit is configured to perform subsequent processing on the video images to generate gimbal output images, including image adjustment, image compression and image optimization.

[0018] The data processing unit is configured to perform data processing on the gimbal data obtained by the gimbal, including data compression and data response.

[0019] Preferably, the environment construction module includes a digital twin module, a virtual reality module, an augmented reality module and a sensor fusion module.

[0020] The digital twin module adopts a digital twin technology to construct a digital twin system for immersive operation of the unmanned aerial vehicle.

[0021] The virtual reality module adopts virtual reality technology to build a three-dimensional virtual environment for immersive operation of the unmanned aerial vehicle.

[0022] The augmented reality module adopts augmented reality technology to build a three-dimensional real environment for immersive operation of the unmanned aerial vehicle.

[0023] The sensor fusion module adopts sensor fusion technology to fuse data of the three-dimensional virtual environment and the three-dimensional real environment based on the digital twin system, so as to form the simulation operation environment for immersive operation of the unmanned aerial vehicle.

[0024] The digital twin system is built, including:

[0025] A construction request is obtained, and a system construction project list is obtained by analyzing the construction request. The construction request includes application scenario requirements, function requirements, data management requirements, and user interaction requirements.

[0026] An initial twin system is established according to the system construction project list.

[0027] Multiple running tests are performed based on the initial twin system to obtain running sample data.

[0028] The initial twin system is optimized and adjusted using the running sample data to generate a final digital twin system.

[0029] Preferably, the initial twin system is established according to the system construction project list, including:

[0030] The system project module is obtained by analyzing the system construction project list, specifically:

[0031] An immersive scenario project module is obtained according to the application scenario requirements.

[0032] An aerodynamic analysis project module is obtained according to the function requirements.

[0033] A data management project module is obtained according to the data management requirements.

[0034] A human-computer interaction project module is obtained according to the user interaction requirements.

[0035] Preferably, the initial twin system is established according to the system construction project list, further including:

[0036] The system project module is analyzed to establish the initial twin system, specifically:

[0037] determine a plurality of device types of the UAV and the UAV support device based on the aerodynamic analysis module, and establish a device model corresponding to each device type;

[0038] obtain structural components corresponding to each device type, and construct a structural component model based on the structural components;

[0039] obtain interaction data of the UAV operation based on the human-computer interaction module, and establish a fault model corresponding to the device model according to the interaction data;

[0040] obtain environmental attributes of the UAV operation based on the immersive scene module;

[0041] obtain navigation attributes of the UAV operation based on the data management module;

[0042] establish a monitoring point model corresponding to each device type according to the environmental attributes and the navigation attributes;

[0043] establish the initial twin system according to the device model, the structural component model, the monitoring point model, and the fault model.

[0044] Preferably, the data fusion of the three-dimensional virtual environment and the three-dimensional real environment by using the sensor fusion technology comprises:

[0045] Data acquisition: collecting virtual environment data in the three-dimensional virtual environment; collecting real environment data in the three-dimensional real environment;

[0046] Data processing: data preprocessing of the virtual environment data and the real environment data; the data preprocessing comprises data cleaning, data denoising, time synchronization, and space registration;

[0047] Data fusion: data fusion of the preprocessed data to generate a fusion data set, specifically comprising:

[0048] First-level data fusion by using a Kalman filtering algorithm;

[0049] Second-level data fusion by using a particle filtering algorithm;

[0050] Generating the fusion data set by using the output data after two-level data fusion;

[0051] Updating rendering: updating the state of the three-dimensional virtual environment in real time based on the fusion data set, and rendering the updated three-dimensional virtual environment to generate a visual effect consistent with the three-dimensional real environment.

[0052] Preferably, the first-level data fusion by using the Kalman filtering algorithm comprises:

[0053] Defining system model: establishing state equation, observation equation and noise model;

[0054] Initialization setting: setting initial state estimation and initial state covariance matrix;

[0055] State prediction: performing state prediction according to the state equation to obtain predicted current state estimation and predicted current state covariance matrix;

[0056] State update: obtaining observation data according to the observation equation and updating the current state estimation and the current state covariance matrix using the observation data;

[0057] Iterative process: continuously repeating the state prediction step and the state update step by obtaining new observation data;

[0058] Output result: outputting the final optimal state estimation and optimal state covariance matrix.

[0059] Preferably, the second data fusion is performed by using a particle filter algorithm, comprising:

[0060] Initializing particle filter: taking the optimal state estimation and the optimal state covariance matrix output by the Kalman filter algorithm as the input of the particle filter algorithm and initializing, including particle state initialization and particle weight initialization;

[0061] Particle prediction: predicting the current particle state of each particle according to the state transition equation;

[0062] Particle update: updating the weight of the particle according to the new observation data;

[0063] Particle resampling: resampling according to the weight of the updated particle;

[0064] Particle state estimation: calculating the final particle state estimation according to the weighted average value of all particles;

[0065] Particle iteration and output: continuously iterating the particle prediction, particle update and particle resampling steps according to the new observation data until the final particle state estimation is output.

[0066] Preferably, the optical anti-shake processing is performed when the gimbal is shooting, specifically:

[0067] Monitoring the motion change of the camera equipment and the lens by using the gyroscope and the accelerometer to obtain camera motion data;

[0068] Analyzing the camera motion data and identifying the to-be-compensated jitter exceeding the preset jitter threshold;

[0069] Compensate the shake by adjusting the optical elements in the camera and lens.

[0070] Continuously adjust the camera and lens by real-time feedback data, detect the definition of each frame of image obtained by shooting, and continuously adjust according to the detection result.

[0071] Preferably, the electronic anti-shake processing of the video image shot by the holder is specifically:

[0072] Classify the video image based on the histogram distribution method, and divide it into a first type of video image containing active motion of the camera and a second type of video image not containing active motion of the camera.

[0073] The first type of video image is subjected to a first type of electronic anti-shake processing.

[0074] The second type of video image is subjected to a second type of electronic anti-shake processing.

[0075] Preferably, the flight control module comprises a flight mode unit, a flight attitude unit, a path planning and navigation unit and a flight controller.

[0076] The flight mode unit is used to simulate the flight mode of the unmanned aerial vehicle and perform corresponding flight operations; the flight mode includes manual control, automatic flight and target tracking.

[0077] The flight attitude unit is used to simulate the flight attitude of the unmanned aerial vehicle and perform corresponding flight operations; the flight attitude includes pitch flight, roll flight and yaw flight.

[0078] The path planning and navigation unit is used to plan the flight path of the unmanned aerial vehicle and perform flight navigation and flight obstacle avoidance.

[0079] The flight controller is used to generate flight control instructions for flight control of the unmanned aerial vehicle according to the current flight state of the unmanned aerial vehicle.

[0080] The beneficial effects of the application are: the application combines the synergistic effect of virtual reality, augmented reality, sensor fusion and high-precision control, can provide real and accurate immersive simulation experience, thereby improving the operation level of the operator and reducing the training risk, and can also improve the holder control ability and stability, and further achieve more efficient training effect. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to better understand and implement, the technical solutions of the application are described in detail below with reference to the drawings.

[0082] Figure 1A structure schematic diagram of an unmanned aerial vehicle immersive operation simulation gimbal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0083] To further clarify the technical means and effects taken by the present application to achieve the predetermined inventive purpose, exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0084] The terms used in the present application are merely for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a," "an," and "the" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0085] The specific embodiments, features and effects according to the present application are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0086] Please refer to Figure 1 The embodiment of the present application provides an unmanned aerial vehicle immersive operation simulation gimbal, which comprises:

[0087] An environment construction module is configured to construct a simulated simulation operation environment and acquire environment data of the simulation operation environment.

[0088] A gimbal control module is configured to simulate a motion mode of the gimbal and perform gimbal control, and to process gimbal shooting images and gimbal control data.

[0089] A flight control module is configured to simulate a flight state of the unmanned aerial vehicle and perform flight control.

[0090] An immersive feedback module is configured to acquire operation data of an operator in the simulation operation environment, to acquire feedback information by analyzing the operation data and the environment data, and to adaptively adjust the simulation operation environment according to the feedback information.

[0091] A multi-user collaboration module is configured to establish a multi-user collaboration mechanism to support multiple operators to participate in simulation training together.

[0092] The personalized training module is used for collecting and recording various data in the flight control and gimbal control process in real time, and comprehensively analyzing and evaluating the training performance of the operator, and generating personalized training tasks for the operator according to the analysis and evaluation results.

[0093] Specifically, since the existing unmanned aerial vehicle simulation system focuses on flight path, control system and environment simulation, but less involves precise simulation of gimbal operation, and gimbal operation as a core link in unmanned aerial vehicle application requires the operator to have superb control skills, and has a great influence on the simulation experience of the operator, in order to improve the operation level of the operator and reduce the training risk, and also improve the gimbal control ability and stability, the application focuses on improving the unmanned aerial vehicle operation gimbal, the simulation environment and the flight control, so as to provide a real and precise immersive simulation experience, and achieve better training effect, which specifically includes the following contents: first, an environment construction module is used to construct a simulated simulation operation environment and obtain environment data of the simulated simulation operation environment; then a gimbal control module is used to simulate the motion mode of the gimbal and control the gimbal; then a flight control module is used to simulate the flight state of the unmanned aerial vehicle and control the flight; then an immersive feedback module is used to obtain operation data of the operator in the simulated simulation operation environment, and the operation data and the environment data are analyzed to obtain feedback information, and the simulated simulation operation environment is adaptively adjusted according to the feedback information; then a multi-user cooperation module is used to establish a multi-user cooperation mechanism to support multiple operators to participate in simulation training together; finally, a personalized training module is used to collect and record various data in the flight control and gimbal control process in real time, and comprehensively analyze and evaluate the training performance of the operator, and generate personalized training tasks for the operator according to the analysis and evaluation results. The application improves the stability, response speed and operation precision of the gimbal, improves the gimbal control algorithm, optimizes the sensor fusion technology and image stabilization technology, so that the motion of the gimbal in the simulation environment is more stable and realistic, thereby improving the immersion and operation experience of the operator. In addition, through the cooperation of the above several modules, by combining virtual reality technology, augmented reality technology, sensor fusion technology and high-precision control technology, a real and precise immersive simulation experience is provided for the operator, thereby improving the operation level of the operator and reducing the training risk, and also improving the gimbal control ability and stability, thereby achieving more efficient training effect.

[0094] In an embodiment provided by the application, the environment construction module includes a digital twin module, a virtual reality module, an augmented reality module and a sensor fusion module.

[0095] The digital twin module constructs a digital twin system of unmanned aerial vehicle immersive operation by using digital twin technology.

[0096] The virtual reality module adopts virtual reality technology to build a three-dimensional virtual environment for immersive operation of the unmanned aerial vehicle.

[0097] The augmented reality module adopts augmented reality technology to build a three-dimensional real environment for immersive operation of the unmanned aerial vehicle.

[0098] The sensor fusion module adopts sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment based on the digital twin system, so as to form the simulation operation environment for immersive operation of the unmanned aerial vehicle.

[0099] The digital twin system is built, including:

[0100] S11, a construction request is obtained, and a system construction project list is obtained by analyzing the construction request, the construction request including application scenario requirements, function requirements, data management requirements, and user interaction requirements;

[0101] S12, an initial twin system is established according to the system construction project list;

[0102] S13, a plurality of running tests are performed based on the initial twin system to obtain running sample data;

[0103] S14, the initial twin system is optimized and adjusted using the running sample data to generate a final digital twin system;

[0104] The initial twin system is established according to the system construction project list, specifically including:

[0105] S121, the system construction project list is analyzed to obtain a system project module, specifically:

[0106] S1211, an immersive scenario project module is obtained according to the application scenario requirements;

[0107] S1212, a pneumatic analysis project module is obtained according to the function requirements;

[0108] S1213, a data management project module is obtained according to the data management requirements;

[0109] S1214, a human-computer interaction project module is obtained according to the user interaction requirements;

[0110] S122, the system project module is modularly analyzed to establish an initial twin system, specifically:

[0111] S1221, a plurality of device types of unmanned aerial vehicles and unmanned aerial vehicle support devices are determined based on the pneumatic analysis project module, and a device model corresponding to each device type is established;

[0112] S1222, obtain structural components corresponding to each device type and construct a structural component model based on the structural components;

[0113] S1223, obtain interaction data of unmanned aerial vehicle operation based on the human-computer interaction project module, and establish a fault model corresponding to the device model according to the interaction data;

[0114] S1224, obtain environmental attributes of unmanned aerial vehicle operation based on the immersive scene project module;

[0115] S1225, obtain navigation attributes of unmanned aerial vehicle operation based on the data management project module;

[0116] S1226, establish a monitoring point model corresponding to each device type according to the environmental attributes and the navigation attributes;

[0117] S1227, establish the initial twin system according to the device model, the structural component model, the monitoring point model, and the fault model;

[0118] The immersive scene project module comprises a scene rendering sub-module and an environment perception sub-module;

[0119] The aerodynamic analysis project module comprises a device support sub-module, a trajectory analysis sub-module, a structural component sub-module, an aerodynamic heat map sub-module, a power system sub-module, a wing system sub-module, and a tail system sub-module;

[0120] The data management project module comprises a digital asset sub-module and a trajectory data sub-module; the digital asset sub-module comprises an unmanned aerial vehicle model and a simulated airport;

[0121] The human-computer interaction project module comprises a flight control sub-module, a task scheduling sub-module, an operation feedback sub-module, and a somatosensory interaction sub-module.

[0122] Specifically, since the premise of realizing the unmanned aerial vehicle immersive operation simulation is to build a simulation environment that is as real as possible, the application adopts digital twinning technology and combines virtual reality, augmented reality and sensor fusion technology to build the simulation operation environment required by the application. Among them, the digital twinning technology creates a virtual copy of the real world entity (such as the unmanned aerial vehicle and its gimbal, sensors, environment, etc.), realizes real-time simulation, monitoring and optimization of the entity, and can provide an accurate, efficient and immersive simulation environment for the operation of the unmanned aerial vehicle; this technology not only reflects the performance of the unmanned aerial vehicle in various environments and tasks in real time, but also provides a highly simulated virtual operation experience for the operator, enhancing the effectiveness and safety of flight training; and on the basis of the digital twinning system, combined with virtual reality and augmented reality technology, the virtual and real can be closely combined, not only can accurately simulate the visual, auditory and tactile effects in flight, but also can dynamically reflect the changes in the environment, helping operators to train and operate in a more realistic and complex virtual environment, thereby greatly improving the immersion and realism of the unmanned aerial vehicle simulation environment. In addition, the sensor fusion technology can fuse the data of multiple sensors to provide data support for the subsequent generation of three-dimensional maps and environment models, and is conducive to updating the virtual environment, such as adjusting the position of the unmanned aerial vehicle, the flight path and the control feedback in the virtual environment according to the real-time changes of the sensor data. Therefore, in summary, based on the digital twinning system, combined with the above virtual reality, augmented reality and sensor fusion technology, a highly realistic immersive simulation operation environment for unmanned aerial vehicles can be created. This environment not only synchronizes the information of the virtual and real worlds, but also updates and feeds back in real time, providing more accurate flight simulation and operation experience.

[0123] On the other hand, since the construction of the digital twinning system has adaptability and non-universality, that is, for different service or application requirements, a digital twinning system that is suitable for it needs to be constructed, which will lead to low efficiency and precision of the construction of the twinning system. Therefore, when constructing the above-mentioned digital twinning system, the application considers the construction request in the construction process, obtains the twinning system construction project list by analyzing the construction request, establishes the initial twinning system by analyzing the above-mentioned list, obtains sample data by running the initial system, and continuously optimizes and adjusts the initial system until the final digital twinning system is generated.

[0124] In addition, since there are many different types of unmanned aerial vehicles and supporting devices matched with the unmanned aerial vehicles, and the differences in structure, operation process, real-time feedback and operating environment of different types of unmanned aerial vehicles and supporting devices and different components of the devices are not considered in the related art, in order to avoid the problem that only a model is established for the physical structure of a certain type of unmanned aerial vehicle and matched device, and a complete system is not formed, the system project module is obtained based on the above system project list, and modular analysis is performed according to the system project module, and the device model, the structure component model, the monitoring point model and the fault model are respectively established, and the initial twin system is constructed through the above multiple models, so that the operation state of the unmanned aerial vehicle and the device entity is more accurately reflected, and since the fault model is used as a construction factor of the twin system, the initial twin system can more accurately and comprehensively analyze and predict the fault state of the unmanned aerial vehicle and the supporting device, and can be adjusted in real time according to the analysis and prediction, so that a more accurate and immersive operation experience is achieved.

[0125] In an embodiment provided by the application, the data fusion of the three-dimensional virtual environment and the three-dimensional real environment by using the sensor fusion technology comprises:

[0126] Data acquisition: collecting virtual environment data in the three-dimensional virtual environment; collecting real environment data in the three-dimensional real environment;

[0127] Data processing: data preprocessing of the virtual environment data and the real environment data; the data preprocessing comprises data cleaning, data denoising, time synchronization and space registration;

[0128] Data fusion: data fusion of the preprocessed data to generate a fusion data set, specifically comprising:

[0129] First data fusion by using a Kalman filtering algorithm;

[0130] Second data fusion by using a particle filtering algorithm;

[0131] Output data after two data fusions is used to generate the fusion data set;

[0132] Update rendering: real-time updating of the state of the three-dimensional virtual environment based on the fusion data set, and environment rendering of the updated three-dimensional virtual environment to generate a visual effect consistent with the three-dimensional real environment.

[0133] Specifically, in order to further enhance the reality and interactivity of the unmanned aerial vehicle simulation operation environment, the embodiment adopts sensor fusion technology to fuse the data of the three-dimensional virtual environment and the three-dimensional real environment on the basis of the digital twin system. The purpose of this fusion is to combine the data of the real world and the virtual environment in real time and accurately, thereby providing the operator with a more realistic and immersive flight experience. In the data fusion process, first, relevant data needs to be collected and preprocessed from the real environment and the virtual environment. The types of data include sensor data, environment models, unmanned aerial vehicle state information, etc. Among them, the real environment data mainly includes unmanned aerial vehicle flight data and ground physical environment data. The unmanned aerial vehicle flight data includes position, attitude, and speed. The ground physical environment data includes obstacles in the environment and meteorological data. Since the three-dimensional virtual environment is similar to the three-dimensional real environment, the virtual environment data also includes physical environment data mainly composed of terrain, buildings, plants, weather, etc., as well as meteorological data of factors affecting flight such as simulated weather changes, light changes, and wind speed changes. In addition, there are flight control data simulated by the physical engine, including speed, acceleration, attitude change, etc. The data preprocessing of the embodiment includes data cleaning, data denoising, time synchronization, and space registration. Through noise filtering of sensor data, unreliable data points are removed. Time synchronization is performed by using timestamps to ensure that the data in the virtual environment and the real environment are synchronized in time, and the data of each sensor and the state information of the virtual environment can be updated at the same time point. The coordinate transformation algorithm (such as the coordinate conversion matrix) is used to convert the GPS coordinates, IMU coordinates, and ground sensor coordinates in the real environment into a unified coordinate system, so that the data can be fused in the same three-dimensional space, thereby preparing for subsequent fusion. Finally, Kalman filtering algorithm and particle filtering algorithm are used for double data fusion of the preprocessed data. Through multiple fusion, noise and clutter in the data are removed, and filtering effect and filtering precision are improved, thereby effectively improving the perception accuracy and robustness of the multi-sensor system, and ensuring the stable operation and accurate control of the simulation operation environment.

[0134] It should be noted that GPS (Global Positioning System) is a global positioning system, and IMU (Inertial Measurement Unit) is an inertial measurement unit.

[0135] In an embodiment provided in the application, the first data fusion using the Kalman filtering algorithm comprises:

[0136] Defining a system model: establishing a state equation, an observation equation, and a noise model;

[0137] The state equation is represented as: x k = Axk-1 +Bu k +β k ;

[0138] wherein, x k represents a state vector at k moment, x k-1 represents a state vector at k-1 moment; A represents a state transition matrix, used for describing the state change rule from k moment to k-1 moment; B represents a control input matrix, used for describing the influence of external control input on system state; u k represents a control input vector; β k represents a Kalman process noise;

[0139] The observation equation is represented as: z k =Hx k +v k ;

[0140] wherein, z k represents observation data at k moment; v k represents a Kalman observation noise; H represents a linear observation matrix, used for mapping the state vector x k to the observation data z k , describing the linear relationship between the observation data and the internal state vector of the system;

[0141] In the noise model, the Kalman process noise β k and the Kalman observation noise v k are both zero-mean Gaussian noise with Gaussian distribution, and have a covariance matrix Q of the Kalman process noise and a covariance matrix R of the Kalman observation noise, respectively;

[0142] Initialization setting: set initial state estimation and initial state covariance matrix;

[0143] State prediction: perform state prediction according to the state equation to obtain predicted current state estimation and predicted current state covariance matrix;

[0144] The predicted current state estimation is represented as:

[0145] wherein, represents predicted current state estimation; represents predicted state estimation at previous moment;

[0146] The predicted current state covariance matrix is represented as:

[0147] wherein, represents predicted current state covariance matrix; P k-1 represents predicted state covariance matrix at previous moment; AT denotes the transpose of the state transition matrix;

[0148] State Update: Obtain the observation data according to the observation equation and update the current state estimate and the current state covariance matrix using the observation data, denoted as:

[0149]

[0150] where, denotes the updated state estimate; K k denotes the Kalman gain, which measures the reliability of the current observation;

[0151] The expression of the Kalman gain K k is: H T denotes the transpose of the linear observation matrix;

[0152] Iterative Process: Obtain new observation data and repeatedly perform the state prediction step and the state update step;

[0153] Output Result: Output the final optimal state estimate and the optimal state covariance matrix.

[0154] Specifically, the Kalman filtering algorithm is used for first-level data fusion on the preprocessed data. This process is used to extract state information from multiple sensors or data sources and optimize the estimation. The state equation (also known as the dynamic model) is used to describe how the system (in this embodiment, the system represents the digital twin system described above) evolves from one time step to the next time step; while the observation equation (also known as the measurement model) is used to describe how to observe the state at the current time through sensors. The core of Kalman filtering is to estimate the system state through prediction and update steps, and to combine multiple sensor data for fusion. The state prediction step is to predict the state at the current time based on the dynamic model of the system and the state estimation at the last time, and to reflect the prediction uncertainty of the current state through the prediction state covariance matrix, which includes the influence of system noise. In the update step, the Kalman gain is calculated to measure the credibility of the current measurement. When the measurement noise is large, the Kalman gain is small, and more depends on the prediction; on the contrary, if the measurement is more accurate, the Kalman gain is larger, and more depends on the measurement data. Then the updated state estimation is obtained by weighted combination of the predicted state and the observation value, and the weight is determined by the Kalman gain; and the updated state covariance matrix reflects the uncertainty of the current state estimation. Then in the iteration process step, the prediction step and the update step are repeated every time new observation data arrives. With the passage of time, the Kalman filtering algorithm gradually converges to the true system state; finally, the final optimal state estimation and the corresponding optimal covariance matrix are output by the output result step, and the covariance matrix provides a quantitative measure of the estimation accuracy. The smaller the value, the more accurate the estimation. In general, the Kalman filtering algorithm performs well in handling linear systems in three-dimensional virtual environments and three-dimensional real environments based on digital twin systems, especially in the case of Gaussian noise. It can effectively fuse data from different sensors and provide optimal state estimation.

[0155] It should be noted that in this application, k time can refer to the current time, and k-1 time is the previous time of the current time. Observation data refers to raw measurement data, which can be a scalar or a vector (or other forms), describing the data information obtained from the environment through sensors and other means, and the observation vector is a specific representation of the observation data, representing a vector containing multiple measurement data.

[0156] In an embodiment provided in the present application, the second-level data fusion using the particle filtering algorithm comprises:

[0157] Initialize the particle filter: the optimal state estimation and the optimal state covariance matrix output by the Kalman filtering algorithm are used as the input of the particle filtering algorithm and are initialized, including particle state initialization and particle weight initialization;

[0158] Particle Prediction: Predict the current particle state of each particle according to the state transition equation, which is expressed as:

[0159]

[0160] where, denotes the particle state of the i-th particle at time k; denotes the particle state of the i-th particle at time k-1; u k denotes the control input vector; denotes the process noise of the i-th particle; denotes the state transition equation of the i-th particle;

[0161] Particle Update: Update the weight of the particle according to the new observation data, which is expressed as:

[0162]

[0163] where, denotes the weight of the i-th particle at time k; denotes the weight of the i-th particle at time k-1; is the particle likelihood, which denotes the conditional probability density of the observation data z k when given the particle state ;

[0164] The particle likelihood is calculated based on the particle observation model, which is calculated as:

[0165]

[0166] where, is the predicted data based on the particle state ; R' denotes the covariance matrix of the particle observation noise; denotes the difference between the observation data and the predicted data;

[0167] The particle observation model is z k = h(x k ) + v' k ; v' k denotes the particle observation noise;

[0168] h(x k ) denotes a nonlinear observation function that maps the current particle state to the observation data z k , describing the nonlinear relationship between the observation data and the internal state vector of the system;

[0169] Particle Resampling: Resample according to the weight of the updated particle, which is:

[0170] Normalizing particle weights: normalize the particle weights so that the sum of all particle weights is 1, then the weight of each particle becomes a probability, representing the probability of this particle being selected;

[0171] The expression for normalizing particle weights is:

[0172] where N is the total number of particles;

[0173] Select particles: reselect particles according to the normalized particle weights and form a new particle set, specifically:

[0174] Generate an equidistant cumulative weight sequence is expressed as: j is the jth particle;

[0175] Randomly generate a uniformly distributed random number r, and r∈[0,1];

[0176] For each particle, the particle state Find the first particle that satisfies and copy this particle to the new particle set;

[0177] Resample the particles until the size of the new particle set reaches the size of the original particle set;

[0178] Particle state estimation: calculate the final particle state estimate according to the weighted average of all particles, then the calculation formula is: represents the final particle state estimate;

[0179] Particle iteration and output: continue to iterate the particle prediction, particle update and particle resampling steps according to the new observation data, until the final particle state estimate is output.

[0180] Specifically, the particle filtering algorithm is used for second data fusion in the present application, which can effectively combine multiple data sources of different types or sources, thereby more accurately estimating the state of the system. On the other hand, the particle filtering algorithm can effectively process data with different levels of noise, especially when the sensor noise is not simple Gaussian noise. Particle filtering can effectively suppress the influence of noise and enhance the robustness of the system to noise through resampling and weight updating mechanism, which makes the system still maintain high estimation accuracy when the data quality is inconsistent or there are serious errors.

[0181] In addition, the Kalman filtering algorithm is combined with the particle filtering algorithm in the application, the output obtained by the Kalman filtering algorithm is taken as the input of the particle filtering algorithm, and the combination of the two is used to process the multi-level data fusion problem and form more compact data fusion. The Kalman filtering is used as the first heavy data fusion tool to process linear and Gaussian noise sensor data, and the particle filtering is used as the second heavy data fusion tool to process more complex conditions of nonlinearity and non-Gaussian noise. Therefore, in general, the combination of the Kalman filtering and the particle filtering applied to the data fusion of the application can improve the accuracy and robustness of the overall state estimation of the digital twin system; more diverse system models (including linear, nonlinear and complex noise models) can be adapted; the calculation burden can be reduced, and the complexity can be reduced by combining the advantages of the two. Therefore, both linear problems and nonlinear problems can be solved.

[0182] In an embodiment provided by the application, the gimbal control module comprises a gimbal movement unit and a gimbal control unit.

[0183] The gimbal movement unit is configured to simulate a movement mode of the gimbal in the simulation operation environment, and the movement mode comprises a gimbal pitching movement, a gimbal yawing movement and a gimbal rolling movement, and specifically comprises:

[0184] The gimbal pitching movement: the camera device and / or the sensor device of the gimbal perform a front-leaning and back-tilting movement, and is configured to adjust the vertical direction of the camera device and / or the sensor device.

[0185] The gimbal yawing movement: the camera device and / or the sensor device of the gimbal perform a left-right translation rotation, and is configured to control the horizontal orientation of the camera device and / or the sensor device.

[0186] The gimbal rolling movement: the camera device and / or the sensor device of the gimbal perform a left-right rotation movement, and is configured to change the horizontal angle of the camera device and / or the sensor device.

[0187] The gimbal control unit is configured to control the gimbal in multiple dimensions in the simulation operation environment, and specifically comprises:

[0188] Position control: a PID control algorithm is used to control the gimbal to rotate to a specific angle position and move to a position.

[0189] Speed control: the rotation speed of the gimbal is controlled to ensure that the gimbal runs smoothly at a set speed.

[0190] Angle control: an incremental control method is used to control the gimbal to rotate to a target angle within a predetermined time.

[0191] Attitude control: the target attitude of the gimbal is adjusted according to preset target attitude parameters.

[0192] Specifically, in the gimbal control module of the embodiment, the movement modes of the gimbal are controlled by the gimbal movement unit, which can be combined into compound movements, and the posture and movement trajectory of the gimbal are accurately described by simulating different combinations and relative position changes of the movement modes; the specific movement form of the gimbal is controlled by the gimbal control unit, different control algorithms are adopted and the target position, speed, acceleration and other parameters are set, and appropriate motor control instructions are generated to control the movement of the gimbal, so as to ensure the accuracy and responsiveness of the gimbal movement. It can be understood that the embodiment mainly expands the gimbal movement unit and the gimbal control unit, and does not specifically limit the control devices such as motor driving devices in the gimbal.

[0193] In an embodiment provided in the application, the gimbal control module further comprises an optical anti-shake unit, an electronic anti-shake unit, an image processing unit and a data processing unit.

[0194] The optical anti-shake unit is configured to perform optical anti-shake processing when the gimbal is shooting, and specifically comprises the following steps:

[0195] S21, monitoring the movement changes of the camera equipment and the lens by the gyroscope and the accelerometer to obtain camera movement data;

[0196] S22, analyzing the camera movement data and identifying the to-be-compensated shake exceeding the preset shake threshold;

[0197] S23, performing optical compensation on the to-be-compensated shake by adjusting the optical elements in the camera equipment and the lens;

[0198] S24, constantly adjusting the camera equipment and the lens by real-time feedback data, detecting the definition of each frame of image obtained by shooting, and continuously adjusting according to the detection result;

[0199] The electronic anti-shake unit is configured to perform electronic anti-shake processing on the video image shot by the gimbal, and specifically comprises the following steps:

[0200] S31, classifying the video image based on the histogram distribution method, and dividing it into a first type of video image containing active movement of the camera equipment and a second type of video image not containing active movement of the camera equipment, and specifically comprising the following steps:

[0201] S311, extracting image frames in the video image sequence at equal intervals and giving a preset similarity threshold;

[0202] S312, compare the hue and saturation histogram distribution of the image frames with each other, if the comparison result of one time is lower than the set similarity threshold, it indicates that the picture has changed due to non-camera random shaking, then the video image is divided into the first type of video image containing active motion of the camera equipment; if all comparison results are not lower than the set similarity threshold, the video image is divided into the second type of video image not containing active motion of the camera equipment;

[0203] It should be noted that the first type of video image containing active motion of the camera equipment refers to the change of the position (displacement) and angle of the camera equipment and lens generated by the operation instruction of the unmanned aerial vehicle, which is driven by the flight control system and operation instruction, and is therefore regarded as active motion of the camera equipment, and the second type of video image not containing active motion of the camera equipment is opposite to the first type.

[0204] S32, performing first type of electronic anti-shaking processing on the first type of video image, including:

[0205] S321, based on the first type of video image, obtaining the motion trajectory of the camera equipment by using a comprehensive corner detection algorithm and a random sample consensus algorithm; wherein the comprehensive corner detection algorithm is fused based on a FAST corner detection algorithm and a Shi-Tomasi corner detection algorithm;

[0206] The obtaining of the motion trajectory of the camera equipment is specifically:

[0207] S3211, performing preliminary corner detection by using the FAST corner detection algorithm in the fusion corner detection algorithm, and extracting candidate feature points of adjacent image frames in the first type of video image;

[0208] S3212, performing fine corner detection by using the Shi-Tomasi corner detection algorithm in the fusion corner detection algorithm, verifying the accuracy of the candidate feature points and screening out optimized feature points;

[0209] S3213, tracking the optimized feature points by using a bidirectional optical flow method and performing feature matching;

[0210] S3214, removing matching error feature points by using the random sample consensus algorithm, and obtaining matching feature points of adjacent image frames after removal;

[0211] S3215, based on the matching feature points, calculating affine transformation parameters by using a least square method;

[0212] S3216, determining the transformation relationship of adjacent image frames based on the affine transformation parameters, and accumulating the transformation relationship to calculate the motion trajectory of the camera equipment;

[0213] S322, then the motion trajectory of the camera is smoothed by using Gaussian filter to obtain the jitter motion estimation;

[0214] S323, then the adjacent image frames are subjected to affine transformation by using the jitter motion estimation to generate stable frame images;

[0215] S324, the stable frame images are compensated by using the interpolation algorithm;

[0216] S325, finally, the first type of video images are input into the multi-thread execution model for multi-thread processing until the video image frames are completely processed, and then the multi-thread is ended;

[0217] S33, the second type of electronic anti-shake processing is performed on the second type of video images, including:

[0218] S331, a plurality of image frames are extracted from the second type of video images, and the first frame and the subsequent frames other than the first frame are distinguished;

[0219] S332, the first frame is subjected to feature extraction by using the Harris matrix-based feature point extraction method, and the first frame feature points are obtained; wherein the Harris matrix-based feature point extraction method specifically divides the first frame into a grid, calculates the Harris matrix of each point in the grid, and selects the feature points, and this embodiment only extracts the feature points from the first frame. It should be noted that the specific calculation method of the Harris matrix and the selection of the feature points are not described in detail in this embodiment.

[0220] S333, the feature matching is performed by using the bidirectional optical flow method, the positions of the first frame feature points are tracked in the subsequent frames to obtain the coordinates of the current frame (the current frame in the subsequent frames) feature points, and the current frame feature points are tracked in the first frame in reverse;

[0221] S334, a confidence degree is set to constrain the first frame feature points; the purpose is to ensure that the current frame stable transformation estimation feature points participating in matching appear in all frames as much as possible, and to avoid that a feature point is estimated incorrectly due to the large value of its Harris matrix.

[0222] S335, the transformation matrix is obtained based on the first frame feature points and the current frame feature points;

[0223] S336, the transformation matrix is applied to the current frame to obtain the anti-shake frame, and the anti-shake frame is compensated by using the interpolation algorithm;

[0224] The image processing unit is configured to perform subsequent processing on the video images to generate gimbal output images, including image adjustment, image compression, and image optimization;

[0225] Image adjustment: automatically light compensation processing for video images acquired in weak light; also adaptive frame compensation processing for distorted images collected;

[0226] Image compression: using image spectrum compression technology to compress video images; for reducing the transmission delay of video images, indirectly increasing the wireless communication distance, and enhancing the stability of image transmission.

[0227] Image optimization: using image correction algorithms (such as distortion correction, geometric correction, etc.) to compensate for image distortion caused by the lens;

[0228] The data processing unit is used to process the gimbal data obtained by the gimbal, including data compression and data response;

[0229] Data compression: using a time domain compression algorithm based on SDT (Swivel Door Transformation) to compress the gimbal data, so as to reduce the transmission delay of real-time sensor data, indirectly increase the wireless communication distance, and enhance the stability of data transmission;

[0230] Data response: using a hard real-time scheduling algorithm and parallel computing to respond to the gimbal data, so as to improve the response speed of the gimbal control and the real-time performance of data interaction. The hard real-time scheduling algorithm can ensure that high-priority tasks are executed in time in task scheduling, avoiding delay and ensuring the real-time performance and accuracy of the gimbal operation.

[0231] Specifically, the gimbal control module of the embodiment mainly processes the shaking of the gimbal during shooting, performs optical anti-shake through the optical anti-shake unit, and accurately controls the real-time adjustment of the lens and other camera equipment components, so as to effectively eliminate the picture shaking caused by the movement of the camera and other camera equipment or external disturbance, thereby ensuring the picture stability and clarity during shooting. The electronic anti-shake unit is also used to process the images or video frames that have been shot, and the image stability is optimized to reduce the picture instability caused by camera shaking. The embodiment combines optical anti-shake and electronic anti-shake technologies to eliminate the shaking caused by unmanned aerial vehicle shooting, thereby ensuring the stability and accuracy of the gimbal control.

[0232] It can be understood that in the electronic anti-shake unit, the embodiment classifies video images and processes two types of images respectively, thereby solving the problem that the prior art can only process video of a single type of camera motion trajectory, and meeting the anti-shake needs of different motion trajectory videos, thereby effectively improving the stability of the picture.

[0233] It can also be understood that the embodiment adopts a comprehensive corner detection algorithm based on the fusion of the FAST corner detection algorithm and the Shi-Tomasi corner detection algorithm for feature point extraction. The FAST corner detection algorithm is used for preliminary fast detection, and the algorithm can be used to quickly scan the image during the flight of the unmanned aerial vehicle, thereby improving the overall processing speed. Then, the Shi-Tomasi corner detection algorithm is used to finely process the candidate corner points detected by the FAST corner detection algorithm, further verify the reliability and accuracy of these corner points, and eliminate unstable corner points that may be affected by noise or errors, so as to improve the reliability of the final detection result. Therefore, the embodiment can improve the detection accuracy while ensuring real-time by fusing the above two algorithms, thereby effectively improving the image processing performance of the unmanned aerial vehicle to meet the needs of various tasks.

[0234] In summary, the gimbal control module of the present application can make the gimbal smoothly and accurately perform various motion tasks and ensure the stability and reliability of the gimbal through the joint action of the gimbal motion unit, the gimbal control unit, the optical anti-shake unit, the electronic anti-shake unit, the image processing unit and the data processing unit.

[0235] In an embodiment provided by the present application, the flight control module includes a flight mode unit, a flight attitude unit, a path planning and navigation unit, and a flight controller.

[0236] The flight mode unit is configured to simulate the flight mode of the unmanned aerial vehicle and perform corresponding flight operations. The flight mode includes manual control, automatic flight and target tracking.

[0237] The flight attitude unit is configured to simulate the flight attitude of the unmanned aerial vehicle and perform corresponding flight operations. The flight attitude includes pitch flight, roll flight and yaw flight.

[0238] The path planning and navigation unit is configured to plan the flight path of the unmanned aerial vehicle and perform flight navigation and flight obstacle avoidance.

[0239] The flight controller is configured to generate flight control instructions for controlling the flight of the unmanned aerial vehicle according to the real-time state data of the current flight state of the unmanned aerial vehicle, specifically:

[0240] Obtain real-time state data of the current flight state of the unmanned aerial vehicle;

[0241] Input the real-time state data into the flight controller, and estimate the current flight state in combination with the aircraft dynamics model to obtain a flight state estimate.

[0242] Use a flight control algorithm to calculate the subsequent flight state of the unmanned aerial vehicle according to the flight state estimate and the expected flight target.

[0243] The environment data is acquired and the flight control instruction is obtained based on a subsequent flight state.

[0244] Specifically, in the simulation operation environment, the flight control module simulates the flight state of the unmanned aerial vehicle and performs flight control, involves the above-mentioned multiple interrelated modules, these modules jointly act on each other, simulate the motion mode and motion attitude of the unmanned aerial vehicle, plan the flight path and avoid obstacles, and calculate and generate the control instruction for the subsequent flight of the unmanned aerial vehicle according to the current motion state of the unmanned aerial vehicle, and perform subsequent flight control according to the control instruction. Thus, the overall test and control of the flight process of the unmanned aerial vehicle are realized.

[0245] On the other hand, the operation user can perform operation simulation of each stage such as take-off, flight and landing of the unmanned aerial vehicle through the simulation gimbal. This simulation training can help the user master the operation mode of the unmanned aerial vehicle, improve the flight skill, and be familiar with the processing method for various flight conditions. And in the simulation flight process, the simulation gimbal can also identify the operation error of the user and give corresponding prompt and correction suggestion, which helps the user to find and correct the operation problem in time, and improves the safety and stability of the flight.

[0246] It should be noted that the flight attitude referred to in the embodiment refers to various flight postures of the unmanned aerial vehicle, including but not limited to pitch, roll and yaw, which is different from the gimbal motion mode of the gimbal motion unit in the gimbal control module. The former refers to the flight attitude of the unmanned aerial vehicle, and the latter refers to the motion mode of the gimbal. Although both of them contain pitch, roll and yaw, there are essential differences in specific meanings and objects.

[0247] In an embodiment provided in the application, the establishment of the multi-user cooperation mechanism comprises:

[0248] Role definition: determining the user roles of different operators, and the cooperation tasks and cooperation targets between different user roles; for example, commander, operator, logistics support personnel, etc.

[0249] Information acquisition: acquiring the flight state information of multiple unmanned aerial vehicles and the environment data and performing information fusion to obtain global situation information;

[0250] Task allocation: based on the global situation information, dividing the priority of the cooperation task and allocating the flight task to different user roles according to the divided priority;

[0251] Network communication: based on different communication needs, establishing multiple cooperation communication units with different communication functions;

[0252] Multi-user cooperation: cooperative communication unit based on flight task selection and flight task matching, and multi-user cooperation operation according to the cooperation target and user role.

[0253] Specifically, the above-mentioned cooperative communication unit includes a real-time monitoring feedback subunit based on UDP and WebSocket: UDP (User Datagram Protocol) is used for real-time control instruction and data transmission; WebSocket is used for bidirectional communication with the ground control station to realize real-time monitoring and feedback of flight status, a task scheduling synchronization subunit based on MQTT and TCP / IP: MQTT (Message Queuing Telemetry Transport) is used for lightweight task scheduling, status updating, message broadcasting, etc., and TCP / IP (Transmission Control Protocol / Internet Protocol) is used for reliable task transmission and log synchronization; a low-power long-distance subunit based on LoRa (Long Range) and Zigbee: used for low-power communication between UAV groups, position synchronization, a group cooperation simulation subunit based on HLA (High-Level Architecture) and DIS (Distributed Interactive Simulation): used for complex tasks that require accurate simulation and high cooperation, such as military training or simulation and simulation of large-scale multi-UAV cooperation tasks. The multi-user cooperation operation of the UAV in this embodiment usually needs to combine the use of multiple protocols to meet different needs. And through the multi-user cooperation module, a multi-user cooperation mechanism can be established to support multiple operators to participate in simulation training together, improving.

[0254] In an embodiment provided in the present application, in the individualized training module, the individualized training tasks include power patrol tasks, geographic mapping tasks, and fire rescue tasks.

[0255] Specifically, the simulation gimbal can simulate different flight training tasks such as power inspection, geographic mapping, fire rescue, etc., so that users can plan and practice tasks in a virtual environment. This helps users better understand task requirements, optimize task execution plans, and improve their ability to handle actual tasks.

[0256] Power inspection: The UAV obtains detailed images of power lines through the camera carried and realizes the monitoring and anomaly detection of power facilities by combining image recognition technology. It also carries an infrared thermal imager to monitor the temperature changes of power equipment in real time, timely discovers abnormal hot spots of equipment and predicts potential failure risks, realizes preventive maintenance and reduces the risk of equipment damage.

[0257] Geographical mapping: Simulate various complex geographical environments such as mountains, plains, lakes, etc., and different weather conditions such as sunny, rainy, foggy, etc. These simulated environments can help users understand the flight performance of the UAV in different terrains and weather conditions, so as to make more reasonable flight plans and operation strategies. Users can simulate actual geographical mapping tasks in a virtual environment, evaluate the task execution ability and efficiency of the UAV by simulating different task scenarios, optimize task planning and improve the accuracy and integrity of mapping data.

[0258] Fire rescue: Simulate different fire scenarios and rescue tasks, allowing rescue personnel to conduct flight training without actual UAVs, which helps rescue personnel familiarize with the operation of UAVs, master flight skills, and familiarize with the operation characteristics in various flight environments. Through simulation of different situations, the ability to respond to emergencies is improved. Through simulation of the process of front and rear coordination command, the command efficiency in actual rescue is improved. Through the use of simulation training data, fire situation analysis and rescue decision learning are carried out.

[0259] In summary, the immersive operation simulation gimbal of the present application is used to simulate the flight environment and operation process of the UAV, providing users with a highly realistic immersive flight experience. By combining simulation software, physical devices, and virtual reality technology, augmented reality technology, etc., the operation of each stage of the UAV such as take-off, flight, and landing can be simulated, making users feel as if they are in a real flight scene. Users can learn and train flight operations in a safe and controllable environment without actually flying a UAV, which not only reduces training costs but also avoids potential safety risks. At the same time, the simulation function of the simulation gimbal can also help users better understand the performance characteristics and operation requirements of the UAV, providing strong support for actual flight operations.

[0260] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0261] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed with the preferred embodiments as above, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with equivalent changes, as long as the changes or modifications do not deviate from the technical solution of the present application. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application still belongs to the scope of the technical solution of the present application.

Claims

1. A drone immersive operation simulation gimbal, characterized in that: Comprise: An environment building module for building a simulated simulation operating environment and obtaining environment data of the simulated simulation operating environment; A gimbal control module for simulating a motion mode of a gimbal and performing gimbal control, and processing gimbal shooting images and gimbal control data; A flight control module for simulating a flight state of a drone and performing flight control; An immersive feedback module for obtaining operation data of an operator in the simulated simulation operating environment, obtaining feedback information by analyzing the operation data and the environment data, and adaptively adjusting the simulated simulation operating environment according to the feedback information; A multi-user collaboration module for establishing a multi-user collaboration mechanism to support multiple operators to participate in simulation training together; A personalized training module for real-time collection and recording of various data in the process of flight control and gimbal control, comprehensive analysis and evaluation of the training performance of the operator, and generation of personalized training tasks for the operator according to the analysis and evaluation results; The gimbal control module comprises an optical anti-shake unit, an electronic anti-shake unit, an image processing unit and a data processing unit; The optical anti-shake unit is used for optical anti-shake processing during gimbal shooting, specifically: Monitoring the motion changes of the camera equipment and the lens by using a gyroscope and an accelerometer to obtain camera motion data; Analyzing the camera motion data and identifying the to-be-compensated jitter that exceeds a preset jitter threshold; Adjusting the optical elements in the camera equipment and the lens to optically compensate for the to-be-compensated jitter; Continuously adjusting the camera equipment and the lens in real time and detecting the definition of each frame of image obtained by shooting, and continuously adjusting according to the detection result; The electronic anti-shake unit is used for electronic anti-shake processing of the video images shot by the gimbal, specifically: Classifying the video images based on a histogram distribution method, and dividing them into a first type of video images containing active motion of the camera equipment and a second type of video images not containing active motion of the camera equipment; Performing first-type electronic anti-shake processing on the first-type video images, including: Based on the first-type video images, using a comprehensive corner detection algorithm and a random sample consensus algorithm to obtain the motion trajectory of the camera equipment; wherein the comprehensive corner detection algorithm is fused based on a FAST corner detection algorithm and a Shi-Tomasi corner detection algorithm; Then, using Gaussian filtering to smooth the motion trajectory of the camera equipment to obtain a jitter motion estimator; Next, using the jitter motion estimator to perform affine transformation on adjacent image frames to generate stable frame images; Then, using an interpolation algorithm to compensate the stable frame images; Finally, inputting the first-type video images as complete video sequences into a multi-thread execution model for multi-thread processing, and ending the multi-threading when all video image frames are processed; Performing second-type electronic anti-shake processing on the second-type video images; The image processing unit is used for subsequent processing of the video images to generate gimbal output images, including image adjustment, image compression and image optimization; The data processing unit is configured to perform data processing on gimbal data obtained by the gimbal, including data compression and data response. 2.The unmanned aerial vehicle (UAV) immersive operation simulation gimbal of claim 1, wherein: The environment construction module includes a digital twin module, a virtual reality module, an augmented reality module, and a sensor fusion module. The digital twin module constructs a digital twin system for the immersive operation of the unmanned aerial vehicle by using a digital twin technology. The virtual reality module constructs a three-dimensional virtual environment for the immersive operation of the unmanned aerial vehicle by using a virtual reality technology. The augmented reality module constructs a three-dimensional real environment for the immersive operation of the unmanned aerial vehicle by using an augmented reality technology. The sensor fusion module performs data fusion on the three-dimensional virtual environment and the three-dimensional real environment by using a sensor fusion technology based on the digital twin system, so as to form the simulation operation environment for the immersive operation of the unmanned aerial vehicle. The digital twin system is constructed by: obtaining a construction request, analyzing the construction request to obtain a system construction project list, and the construction request including application scenario requirements, function requirements, data management requirements, and user interaction requirements; establishing an initial twin system according to the system construction project list; performing multiple running tests based on the initial twin system to obtain running sample data; optimizing and adjusting the initial twin system by using the running sample data to generate a final digital twin system.

3. The unmanned aerial vehicle immersive operation simulation gimbal of claim 2, wherein: The initial twin system is established according to the system construction project list, including: analyzing the system construction project list to obtain system project modules, specifically: obtaining an immersive scenario project module according to the application scenario requirements; obtaining a pneumatic analysis project module according to the function requirements; obtaining a data management project module according to the data management requirements; obtaining a human-computer interaction project module according to the user interaction requirements.

4. The unmanned aerial vehicle immersive operation simulation gimbal of claim 3, wherein: The initial twin system is established according to the system construction project list, further including: performing modular analysis on the system project modules to establish the initial twin system, specifically: determining multiple device types of the unmanned aerial vehicle and unmanned aerial vehicle support devices based on the pneumatic analysis project module, and establishing a device model corresponding to each device type; obtaining structural components corresponding to each device type, and constructing a structural component model therefrom; obtaining interaction data of the unmanned aerial vehicle operation based on the human-computer interaction project module, and establishing a fault model corresponding to the device model according to the interaction data; obtaining environmental attributes of the unmanned aerial vehicle operation based on the immersive scenario project module; obtaining navigation attributes of the unmanned aerial vehicle operation based on the data management project module; establishing a monitoring point model corresponding to each device type according to the environmental attributes and the navigation attributes; establishing the initial twin system according to the device model, the structural component model, the monitoring point model, and the fault model.

5. The unmanned aerial vehicle immersive operation simulation gimbal of claim 2, wherein: The data fusion on the three-dimensional virtual environment and the three-dimensional real environment by using the sensor fusion technology includes: data acquisition: collecting virtual environment data in the three-dimensional virtual environment; collecting real environment data in the three-dimensional real environment; Data processing: data preprocessing is performed on the virtual environment data and the real environment data; the data preprocessing includes data cleaning, data denoising, time synchronization and space registration; Data fusion: data fusion is performed on the preprocessed data to generate a fusion data set, specifically including: First-level data fusion is performed using a Kalman filter algorithm; Second-level data fusion is performed using a particle filter algorithm; The output data after two-level data fusion is used to generate the fusion data set; Update rendering: the state of the three-dimensional virtual environment is updated in real time based on the fusion data set, and the updated three-dimensional virtual environment is rendered to generate a visual effect consistent with the three-dimensional real environment.

6. The unmanned aerial vehicle immersive operation simulation gimbal of claim 5, wherein: The first-level data fusion using the Kalman filter algorithm includes: Defining system model: establishing state equation, observation equation and noise model; Initialization setting: setting initial state estimation and initial state covariance matrix; State prediction: state prediction is performed according to the state equation to obtain predicted current state estimation and predicted current state covariance matrix; State update: observation data is obtained according to the observation equation and used to update the current state estimation and the current state covariance matrix; Iterative process: new observation data is obtained and the state prediction step and the state update step are repeatedly performed; Output result: the final optimal state estimation and optimal state covariance matrix are output.

7. The unmanned aerial vehicle immersive operation simulation gimbal of claim 6, wherein: The second-level data fusion using the particle filter algorithm includes: Initialization of particle filter: the optimal state estimation and optimal state covariance matrix output by the Kalman filter algorithm are used as the input of the particle filter algorithm and initialized, including particle state initialization and particle weight initialization; Particle prediction: the current particle state of each particle is predicted according to the state transition equation; Particle update: the weight of the particle is updated according to the new observation data; Particle resampling: resampling is performed according to the weight of the updated particle; Particle state estimation: the final particle state estimation is calculated according to the weighted average value of all particles; Particle iteration and output: the particle prediction, particle update and particle resampling steps are continuously iterated and executed according to the new observation data until the final particle state estimation is output.

8. The unmanned aerial vehicle immersive operation simulation gimbal of claim 1, wherein: The flight control module includes a flight mode unit, a flight attitude unit, a path planning and navigation unit, and a flight controller; The flight mode unit is used to simulate the flight mode of the UAV and perform corresponding flight operations; the flight mode includes manual control, automatic flight and target tracking; The flight attitude unit is used to simulate the flight attitude of the UAV and perform corresponding flight operations; the flight attitude includes pitch flight, roll flight and yaw flight; The path planning and navigation unit is used to plan the flight path of the UAV and perform flight navigation and flight obstacle avoidance; The flight controller is used to generate flight control instructions for controlling the UAV according to the current flight state of the UAV.

Images