Immersive operation analog simulation holder for unmanned aerial vehicle

Abstract

The invention discloses an immersive operation analog simulation holder for an unmanned aerial vehicle, and relates to the technical field of unmanned aerial vehicles. The holder comprises an environment construction module for constructing a simulated simulation operation environment; the pan-tilt control module is used for simulating the motion mode of the pan-tilt and controlling the pan-tilt; the flight control module simulates the flight state of the unmanned aerial vehicle and performs flight control; the immersive feedback module is used for analyzing the operation data and the environment data to obtain feedback information and adaptively adjusting the simulation operation environment; the multi-user cooperation module supports a plurality of operators to participate in simulation training together; and the personalized training module is used for generating personalized training tasks. According to the invention, by combining the synergistic effect of virtual reality, augmented reality, sensor fusion and high-precision control, real and accurate immersive simulation experience can be provided, so that the operation level of an operator is improved, the training risk is reduced, the control capability and stability of the holder can be improved, and a more efficient training effect is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to an immersive operation simulation gimbal for unmanned aerial vehicles. Background Art

[0002] In recent years, drone technology has been widely used in civil, military, agricultural, photography, logistics and other fields. The flight performance, stability and application scenarios of drones are constantly improving, which has led to increasing technical requirements for flight operators. In order to ensure that pilots can operate drones proficiently, especially in complex environments, it is necessary to develop efficient and low-risk training tools. As an important means of flight training, flight simulation technology is widely used in civil aviation, military and drone fields. Traditional flight simulators mainly provide virtual environment simulation through computer graphical displays and controllers, but often lack immersion in the operating experience. With the maturity of technologies such as virtual reality (VR) and augmented reality (AR), flight simulation technology has gradually developed towards a higher sense of immersion and interactivity, providing a more realistic control experience.

[0003] Traditional drone operation training mostly relies on simulators and field flights. In actual flights, operators face problems such as high costs and high risks. In order to overcome these challenges, immersive training systems have emerged. Through virtual reality technology, operators can simulate real flight situations, conduct gimbal operation and flight control training in a safe environment, avoid risks in actual flights, and improve training efficiency. The gimbal is an important component of drones, used to stabilize the camera and ensure the stability of the shooting during flight. With the increase in the demand for drone shooting, gimbal technology continues to improve, especially its stability and control accuracy. Modern drone gimbals not only support gimbal angle adjustment, but also have automatic functions such as target tracking and object recognition. At present, there are already some drone simulation systems on the market, which mainly focus on flight paths, control systems and environmental simulation, but rarely involve accurate simulation of gimbal operation. As the core link in drone applications, gimbal operation requires operators to have superb control skills. In response to this demand, it is particularly important to develop a system with immersive simulation functions specifically for drone gimbal operation. Summary of the invention

[0004] The purpose of the present invention is to provide an immersive operation simulation pan-tilt platform for unmanned aerial vehicles, which can be achieved through the following technical solutions:

[0005] The embodiment of the present application provides a drone immersive operation simulation gimbal, including:

[0006] Environment construction module: used to construct a simulated operation environment and obtain environmental data of the simulated operation environment;

[0007] PTZ control module: used to simulate the motion mode of the PTZ and control the PTZ, as well as process the PTZ shooting images and PTZ control data;

[0008] Flight control module: used to simulate the flight status of the drone and perform flight control;

[0009] Immersive feedback module: used for obtaining the operation data of the operator in the simulated operation environment, obtaining feedback information by analyzing the operation data and the environment data, and adaptively adjusting the simulated operation environment according to the feedback information;

[0010] Multi-user collaboration module: used to establish a multi-user collaboration mechanism to support multiple operators to participate in simulation training;

[0011] Personalized training module: used to collect and record various data during flight control and gimbal control in real time, and conduct comprehensive analysis and evaluation based on the operator's training performance, and generate personalized training tasks for the operator based on the analysis and evaluation results;

[0012] The gimbal control module includes a gimbal motion unit, a gimbal control unit, an optical image stabilization unit, an electronic image stabilization unit, an image processing unit and a data processing unit;

[0013] The pan / tilt motion unit is used to simulate the motion mode of the pan / tilt in the simulated operation environment, wherein the motion mode includes the pan / tilt pitch motion, the pan / tilt yaw motion and the pan / tilt tumble motion;

[0014] The pan / tilt control unit is used to perform multi-dimensional control on the pan / tilt in the simulated operating environment;

[0015] The optical image stabilization unit is used to perform optical image stabilization processing when the gimbal is shooting;

[0016] The electronic anti-shake unit is used to perform electronic anti-shake processing on the video image taken by the PTZ;

[0017] The image processing unit is used to perform subsequent processing on the video image to generate a PTZ output image, including image adjustment, image compression and image optimization;

[0018] The data processing unit is used to process the PTZ data acquired by the PTZ, including data compression and data response.

[0019] Preferably, the environment construction module includes a digital twin module, a virtual reality module, an augmented reality module and a sensor fusion module;

[0020] The digital twin module uses digital twin technology to build a digital twin system for immersive operation of drones;

[0021] The virtual reality module uses virtual reality technology to build a three-dimensional virtual environment for immersive operation of the drone;

[0022] The augmented reality module uses augmented reality technology to build a three-dimensional real environment for immersive operation of the drone;

[0023] The sensor fusion module, based on the digital twin system, uses sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment to form the simulated operation environment for immersive operation of the drone;

[0024] Wherein, constructing the digital twin system includes:

[0025] Obtaining a construction request, and obtaining a system construction project list by analyzing the construction request, wherein the construction request includes application scenario requirements, function requirements, data management requirements, and user interaction requirements;

[0026] Establishing an initial twin system according to the system construction project list;

[0027] Conducting multiple operation tests based on the initial twin system to obtain operation sample data;

[0028] The initial twin system is optimized and adjusted using the running sample data to generate a final digital twin system.

[0029] Preferably, the establishing of the initial twin system according to the system construction project list comprises:

[0030] Analyze the system construction project list to obtain system project modules, specifically:

[0031] Acquire an immersive scene project module according to the application scenario requirements;

[0032] Acquire a pneumatic analysis project module according to the functional requirements;

[0033] Acquire a data management project module according to the data management requirements;

[0034] A human-computer interaction project module is obtained according to the user interaction requirement.

[0035] Preferably, the step of establishing an initial twin system according to the system construction project list further includes:

[0036] The system project modules are modularly analyzed to establish an initial twin system, specifically:

[0037] Determining multiple equipment types of UAVs and UAV support equipment based on the aerodynamic analysis project module, and establishing an equipment model corresponding to each equipment type;

[0038] Obtaining structural components corresponding to each equipment type and constructing a structural component model based on the structural components;

[0039] Acquire the interaction data of the UAV operation based on the human-computer interaction project module, and establish a fault model corresponding to the device model according to the interaction data;

[0040] Acquire the environmental attributes of the UAV operation based on the immersive scene project module;

[0041] Acquire the navigation attributes of the UAV operation based on the data management project module;

[0042] Establishing a monitoring point model corresponding to each equipment type according to the environmental attributes and the navigation attributes;

[0043] The initial twin system is established according to the equipment model, the structural component model, the monitoring point model and the fault model.

[0044] Preferably, the adopting of sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment includes:

[0045] Data collection: collecting virtual environment data in the three-dimensional virtual environment; collecting real environment data in the three-dimensional real environment;

[0046] Data processing: performing data preprocessing on the virtual environment data and the real environment data; the data preprocessing includes data cleaning, data denoising, time synchronization and space registration;

[0047] Data fusion: The preprocessed data is fused to generate a fused data set, including:

[0048] The Kalman filter algorithm is used for the first data fusion;

[0049] The particle filter algorithm is used for the second data fusion;

[0050] Generating the fused data set with the output data after the two data are fused;

[0051] Update rendering: update the state of the three-dimensional virtual environment in real time based on the fused data set, and perform environmental rendering on the updated three-dimensional virtual environment to generate a visual effect consistent with the three-dimensional real environment.

[0052] Preferably, the first data fusion is performed using a Kalman filter algorithm, including:

[0053] Define the system model: establish the state equation, observation equation and noise model;

[0054] Initialization settings: set the initial state estimate and initial state covariance matrix;

[0055] State prediction: performing state prediction according to the state equation to obtain a predicted current state estimate and a predicted current state covariance matrix;

[0056] State update: obtaining observation data according to the observation equation and using the observation data to update the current state estimate and the current state covariance matrix;

[0057] Iterative process: Get new observation data and repeat the state prediction step and state update step;

[0058] Output results: Output the final optimal state estimate and optimal state covariance matrix.

[0059] Preferably, the second data fusion using a particle filter algorithm includes:

[0060] Initialize particle filter: use the optimal state estimate and optimal state covariance matrix output by the Kalman filter algorithm as the input of the particle filter algorithm and initialize it, including particle state initialization and particle weight initialization;

[0061] Particle prediction: predict the current particle state of each particle based on the state transfer equation;

[0062] Particle update: Update the particle weight according to the new observation data;

[0063] Particle resampling: resampling according to the updated particle weights;

[0064] Particle state estimation: Calculate the final particle state estimation based on the weighted average of all particles;

[0065] Particle iteration and output: Continue to iterate the particle prediction, particle update, and particle resampling steps according to the new observation data until the final particle state estimate is output.

[0066] Preferably, the optical image stabilization process is performed during the pan / tilt shooting, specifically:

[0067] Monitor the movement changes of the camera device and the lens through the gyroscope and accelerometer to obtain the camera movement data;

[0068] Analyzing the camera motion data and identifying jitter to be compensated that exceeds a preset jitter threshold;

[0069] Optically compensating the jitter to be compensated by adjusting the camera device and the optical elements in the lens;

[0070] The camera equipment and lens are continuously adjusted through real-time feedback data, and the clarity of each frame of image captured is tested, and continuous adjustments are made based on the test results.

[0071] Preferably, the electronic anti-shake processing of the video image captured by the PTZ is specifically performed as follows:

[0072] Classifying the video images based on a histogram distribution method into a first type of video images that include active motion of a camera device and a second type of video images that do not include active motion of a camera device;

[0073] Performing a first type of electronic anti-shake processing on the first type of video images;

[0074] The second type of electronic anti-shake processing is performed on the second type of video images.

[0075] Preferably, the flight control module includes a flight mode unit, a flight attitude unit, a path planning and navigation unit and a flight controller;

[0076] The flight mode unit is used to simulate the flight mode of the UAV and perform corresponding flight operations; the flight modes include manual control, automatic flight and target tracking;

[0077] The flight attitude unit is used to simulate the flight attitude of the UAV and perform corresponding flight operations; the flight attitude includes pitch flight, roll flight and yaw flight;

[0078] The path planning and navigation unit is used to plan the flight path of the UAV and perform flight navigation and flight obstacle avoidance;

[0079] The flight controller is used to generate flight control instructions for controlling the UAV according to the current flight status of the UAV.

[0080] The beneficial effects of the present invention are as follows: the present application combines the synergy of virtual reality, augmented reality, sensor fusion and high-precision control to provide a real and accurate immersive simulation experience, thereby improving the operator's operating level and reducing training risks, and can also improve the pan-tilt control capability and stability, thereby achieving a more efficient training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0082] Figure 1A schematic diagram of the structure of a drone immersive operation simulation gimbal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of methods and systems consistent with some aspects of the present application as detailed in the attached claims.

[0084] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to any or all possible combinations of one or more associated listed items.

[0085] The specific implementation methods, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0086] See also Figure 1 The embodiment of the present application provides a drone immersive operation simulation gimbal, including:

[0087] Environment construction module: used to construct a simulated operation environment and obtain environmental data of the simulated operation environment;

[0088] PTZ control module: used to simulate the motion mode of the PTZ and control the PTZ, as well as process the PTZ shooting images and PTZ control data;

[0089] Flight control module: used to simulate the flight status of the drone and perform flight control;

[0090] Immersive feedback module: used for obtaining the operation data of the operator in the simulated operation environment, obtaining feedback information by analyzing the operation data and the environment data, and adaptively adjusting the simulated operation environment according to the feedback information;

[0091] Multi-user collaboration module: used to establish a multi-user collaboration mechanism to support multiple operators to participate in simulation training;

[0092] Personalized training module: used to collect and record various data during flight control and gimbal control in real time, conduct comprehensive analysis and evaluation based on the operator's training performance, and generate personalized training tasks for the operator based on the analysis and evaluation results.

[0093] Specifically, since existing drone simulation systems focus on flight paths, control systems and environmental simulations, but rarely involve accurate simulation of gimbal operations, and gimbal operations, as a core link in drone applications, require operators to have superb control skills and will have a great impact on the operator's simulation experience, in order to improve the operator's operating level and reduce training risks, as well as to improve the gimbal control capabilities and stability, this application focuses on improving drone operation gimbals, simulation environments and flight control, so as to provide a realistic and accurate immersive simulation experience and achieve better training results, specifically including the following contents: first, a simulated operating environment is constructed through an environment construction module and environmental data of the simulated operating environment is obtained; then, The gimbal control module simulates the movement mode of the gimbal and performs gimbal control; then the flight control module simulates the flight state of the drone and performs flight control; then the immersive feedback module obtains the operator's operation data in the simulated operation environment, analyzes the operation data and the environmental data to obtain feedback information and adaptively adjusts the simulated operation environment based on the feedback information; then the multi-user collaboration module is used to establish a multi-user collaboration mechanism to support multiple operators to participate in simulation training; finally, the personalized training module collects and records various types of data during flight control and gimbal control in real time, and conducts comprehensive analysis and evaluation based on the operator's training performance, and generates personalized training tasks for the operator based on the analysis and evaluation results. This application improves the stability, response speed and operation accuracy of the gimbal, and by improving the gimbal control algorithm, optimizing the sensor fusion technology and the image stabilization technology, the movement of the gimbal in the simulation environment is smoother and more realistic, thereby enhancing the operator's sense of immersion and operating experience. In addition, this application, through the joint effect of the above-mentioned modules, combines the synergy of virtual reality technology, augmented reality technology, sensor fusion technology and high-precision control technology to provide operators with a real and accurate immersive simulation experience, thereby improving the operator's operating level and reducing training risks. It can also improve the gimbal control ability and stability, thereby achieving more efficient training results.

[0094] In one embodiment provided in the present application, the environment construction module includes a digital twin module, a virtual reality module, an augmented reality module and a sensor fusion module;

[0095] The digital twin module uses digital twin technology to build a digital twin system for immersive operation of drones;

[0096] The virtual reality module uses virtual reality technology to build a three-dimensional virtual environment for immersive operation of the drone;

[0097] The augmented reality module uses augmented reality technology to build a three-dimensional real environment for immersive operation of the drone;

[0098] The sensor fusion module, based on the digital twin system, uses sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment to form the simulated operation environment for immersive operation of the drone;

[0099] Wherein, constructing the digital twin system includes:

[0100] S11, obtaining a construction request, and obtaining a system construction project list by analyzing the construction request, wherein the construction request includes application scenario requirements, function requirements, data management requirements, and user interaction requirements;

[0101] S12, establishing an initial twin system according to the system construction project list;

[0102] S13, performing multiple operation tests based on the initial twin system to obtain operation sample data;

[0103] S14, optimizing and adjusting the initial twin system using the running sample data to generate a final digital twin system;

[0104] The step of establishing an initial twin system according to the system construction project list specifically includes:

[0105] S121, analyzing the system construction project list to obtain system project modules, specifically:

[0106] S1211, obtaining an immersive scene project module according to the application scenario requirements;

[0107] S1212, obtaining an aerodynamic analysis project module according to the functional requirement;

[0108] S1213, obtaining a data management project module according to the data management requirement;

[0109] S1214, obtaining a human-computer interaction project module according to the user interaction requirement;

[0110] S122, performing modular analysis on the system project modules to establish an initial twin system, specifically:

[0111] S1221, determining multiple equipment types of the UAV and the UAV support equipment based on the aerodynamic analysis project module, and establishing an equipment model corresponding to each equipment type;

[0112] S1222, obtaining structural components corresponding to each device type, and constructing a structural component model based on the structural components;

[0113] S1223, acquiring interaction data of the operation of the drone based on the human-computer interaction project module, and establishing a fault model corresponding to the device model according to the interaction data;

[0114] S1224, obtaining the environmental attributes of the drone operation based on the immersive scene project module;

[0115] S1225, obtaining the navigation attributes of the UAV operation based on the data management project module;

[0116] S1226, establishing a monitoring point model corresponding to each equipment type according to the environmental attributes and the navigation attributes;

[0117] S1227, establishing the initial twin system according to the equipment model, the structural component model, the monitoring point model and the fault model;

[0118] The immersive scene project module includes a scene rendering submodule and an environment perception submodule;

[0119] The aerodynamic analysis project module includes an equipment support submodule, a trajectory analysis submodule, a structural component submodule, an aerodynamic heat map submodule, a power system submodule, a wing system submodule and a tail system submodule;

[0120] The data management project module includes a digital asset submodule and a trajectory data submodule; the digital asset submodule includes a drone model and a simulated airport;

[0121] The human-computer interaction project module includes a flight control submodule, a task scheduling submodule, an operation feedback submodule and a somatosensory interaction submodule.

[0122] Specifically, since the premise of realizing immersive operation simulation of drones is to build a sufficiently real simulation environment, this application adopts digital twin technology and combines multiple technologies such as virtual reality, augmented reality and sensor fusion to jointly construct the simulation operation environment required by this application. Among them, digital twin technology can provide an accurate, efficient and immersive simulation environment for the operation of drones by creating virtual copies of real-world entities (such as drones and their gimbals, sensors, environments, etc.); this technology can not only reflect the performance of drones in various environments and tasks in real time, but also provide operators with a highly simulated virtual operation experience, enhancing the effectiveness and safety of flight training; and on the basis of the digital twin system, combined with virtual reality and augmented reality technology, it is possible to achieve a close combination of virtual and reality, which can not only accurately simulate the visual, auditory and tactile effects in flight, but also dynamically reflect environmental changes, helping operators to train and operate in a more realistic and complex virtual environment, thereby greatly enhancing the immersion and authenticity of the drone simulation environment. In addition, sensor fusion technology can fuse data from multiple sensors to provide data support for the subsequently generated three-dimensional maps and environmental models, and is conducive to updating the virtual environment, such as adjusting the position, flight path and control feedback of the drone in the virtual environment according to the real-time changes in sensor data. Therefore, in summary, this application is based on a digital twin system, combined with the above-mentioned virtual reality, augmented reality and sensor fusion technology, to create a highly realistic drone immersive simulation operation environment, which can not only synchronize information between the virtual and real worlds, but also update and feedback in real time, providing a more accurate flight simulation and operation experience.

[0123] On the other hand, since the construction of the digital twin system is adaptable and non-universal, that is, for different service or application requirements, it is necessary to build a digital twin system that is compatible with it, which will lead to low construction efficiency and accuracy of the twin system. Therefore, when constructing the above-mentioned digital twin system, this application takes the construction request into consideration in the construction process, obtains the list of twin system construction projects by analyzing the construction request, and establishes the initial twin system by analyzing the above list, and then obtains sample data by running the initial system, and continuously optimizes and adjusts the initial system until the final digital twin system is generated.

[0124] In addition, since there are many different types of drones and supporting equipment that cooperate with drones, and there is currently no relevant technology that takes into account the differences in structure, operation process, real-time feedback and operating environment between different types of drones and supporting equipment and different components of the equipment, in order to avoid only establishing a model for the physical structure of a certain type of drone and its matching equipment without forming a complete system, this application obtains a system project module based on the above-mentioned system construction project list, and performs modular analysis based on the system project module, and establishes equipment models, structural component models, monitoring point models, and fault models respectively, and then constructs the above-mentioned initial twin system through the above-mentioned multiple models to more accurately reflect the operating status of the drone and equipment entities. And since this embodiment uses the fault model as the construction factor of the twin system, the initial twin system can more accurately and comprehensively analyze and predict the fault status of the drone and supporting equipment, and can make real-time adjustments based on the analysis and prediction, so as to achieve a more accurate and immersive operation experience.

[0125] In an embodiment provided in the present application, the adopting of sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment includes:

[0126] Data collection: collecting virtual environment data in the three-dimensional virtual environment; collecting real environment data in the three-dimensional real environment;

[0127] Data processing: performing data preprocessing on the virtual environment data and the real environment data; the data preprocessing includes data cleaning, data denoising, time synchronization and space registration;

[0128] Data fusion: The preprocessed data is fused to generate a fused data set, including:

[0129] The Kalman filter algorithm is used for the first data fusion;

[0130] The particle filter algorithm is used for the second data fusion;

[0131] Generating the fused data set with the output data after the two data are fused;

[0132] Update rendering: update the state of the three-dimensional virtual environment in real time based on the fused data set, and perform environmental rendering on the updated three-dimensional virtual environment to generate a visual effect consistent with the three-dimensional real environment.

[0133] Specifically, in order to further enhance the authenticity and interactivity of the UAV simulation operation environment, this embodiment uses sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment on the basis of the digital twin system. The purpose of this fusion is to combine the data of the real world with the data in the virtual environment in real time and accurately, so as to provide the operator with a more realistic and immersive flight experience. In the process of data fusion, it is first necessary to collect and preprocess relevant data from the real environment and the virtual environment. The types of data include sensor data, environmental models, UAV status information, etc. Among them, the real environment data mainly includes UAV flight data and ground physical environment data. UAV flight data includes position, attitude, and speed; ground physical environment data includes obstacles in the environment, meteorological data, etc.; and because the three-dimensional virtual environment is similar to the three-dimensional real environment, the virtual environment data also includes physical environment data mainly based on environmental elements such as terrain, buildings, plants, and weather, as well as meteorological data that simulates factors affecting flight such as weather changes, light changes, and wind speed changes, and flight control data simulated according to the physical engine, including speed, acceleration, attitude changes, etc. The data preprocessing of this embodiment includes data cleaning, data denoising, time synchronization and spatial registration. Unreliable data points are removed by filtering the sensor data for noise, and the data in the virtual environment and the real environment are synchronized in time by using timestamps for time synchronization, ensuring that the data of each sensor and the state information of the virtual environment can be updated at the same time point; the GPS coordinates, IMU coordinates, ground sensor coordinates, etc. in the real environment are converted into a unified coordinate system by using a coordinate transformation algorithm (such as a coordinate transformation matrix), so that the data can be fused in the same three-dimensional space, so as to prepare for subsequent fusion. Finally, the Kalman filter algorithm and the particle filter algorithm are respectively used to perform double data fusion on the preprocessed data, and the noise and clutter in the data are removed through multiple fusions, and the filtering effect and filtering accuracy are improved, which effectively improves the perception accuracy and robustness of the multi-sensor system, thereby ensuring the stable operation and precise control of the simulation operation environment.

[0134] It should be noted that GPS (Global Positioning System) is the global positioning system and IMU (Inertial Measurement Unit) is the inertial measurement unit.

[0135] In an embodiment provided in the present application, the first data fusion is performed using a Kalman filter algorithm, including:

[0136] Define the system model: establish the state equation, observation equation and noise model;

[0137] The state equation is expressed as: k =Axk-1 +Bu k +β k ;

[0138] Among them, x k represents the state vector at time k, x k-1 represents the state vector at time k-1; A represents the state transfer matrix, which is used to describe the state change law from time k to time k-1; B represents the control input matrix, which is used to describe the influence of external control input on the system state; u k represents the control input vector; β k represents the Kalman process noise;

[0139] The observation equation is expressed as: k =Hx k +v k ;

[0140] Among them, z k represents the observed data at time k; v k represents the Kalman observation noise; H represents the linear observation matrix, which is used to transform the state vector x k Mapping to observation data z k , describes the linear relationship between the observed data and the internal state vector of the system;

[0141] In the noise model, the Kalman process noise β k and Kalman observation noise v k They are all zero-mean Gaussian noises with Gaussian distribution, and have the covariance matrix Q of Kalman process noise and the covariance matrix R of Kalman observation noise respectively;

[0142] Initialization settings: set the initial state estimate and initial state covariance matrix;

[0143] State prediction: performing state prediction according to the state equation to obtain a predicted current state estimate and a predicted current state covariance matrix;

[0144] The predicted current state estimate is expressed as:

[0145] in, represents the predicted current state estimate; represents the forecast state estimate at the previous moment;

[0146] The predicted current state covariance matrix is ​​expressed as:

[0147] in, Represents the predicted current state covariance matrix; P k-1 A represents the predicted state covariance matrix of the previous moment;T represents the transpose of the state transfer matrix;

[0148] State update: Obtain observation data according to the observation equation and use the observation data to update the current state estimate and the current state covariance matrix, expressed as:

[0149]

[0150] in, represents the updated state estimate; K k represents the Kalman gain, which is used to measure the credibility of the current observation;

[0151] Kalman gain K k The expression is: H T represents the transpose of the linear observation matrix;

[0152] Iterative process: Get new observation data and repeat the state prediction step and state update step;

[0153] Output results: Output the final optimal state estimate and optimal state covariance matrix.

[0154] Specifically, the Kalman filter algorithm is used to perform the first data fusion on the preprocessed data. This process is used to extract state information from multiple sensors or data sources and optimize the estimation. The state equation (also called the dynamic model) is used to describe how the system (in this embodiment, the system represents the above-mentioned digital twin system) evolves from the state of one time step to the state of the next time step; and the observation equation (also called the measurement model) is used to describe how the current state is observed by the sensor. The core of Kalman filtering is to estimate the system state through the two steps of prediction and update, and to fuse multiple sensor data. The state prediction step is to predict the state at the current moment based on the dynamic model of the system and the state estimate of the previous moment, and to reflect the uncertainty of the prediction of the current state by predicting the state covariance matrix, including the influence of system noise. In the update step, the credibility of the current measurement is measured by calculating the Kalman gain. When the measurement noise is large, the Kalman gain will be small and more dependent on the prediction; on the contrary, if the measurement is more accurate and the Kalman gain is large, it depends more on the measurement data. Then the updated state estimate is obtained by weighted combination of predicted state and observed value, and its weight is determined by Kalman gain; and the updated state covariance matrix reflects the uncertainty of the current state estimate. Then in the iterative process step, the above prediction step and update step are repeated each time new observation data arrives. As time goes by, the Kalman filter algorithm gradually converges to the real system state; finally, the output result step outputs the final optimal state estimate and the corresponding optimal covariance matrix. The covariance matrix provides quantification of estimation accuracy. The smaller the value, the more accurate the estimation. In general, the Kalman filter algorithm performs well in processing linear system parts in three-dimensional virtual environment and three-dimensional real environment based on digital twin system, especially in the case of Gaussian noise. It can effectively fuse data from different sensors and provide optimal state estimation.

[0155] It should be noted that in this application, the k moment may refer to the current moment, and the k-1 moment may refer to the moment before the current moment. Observation data refers to the original measurement data, which may be a scalar or vector (or other form), describing the data information obtained from the environment by means of sensors, etc., and the observation vector is a specific representation of the observation data, representing a vector containing multiple measurement data.

[0156] In an embodiment provided in the present application, the second data fusion is performed using a particle filter algorithm, including:

[0157] Initialize particle filter: use the optimal state estimate and optimal state covariance matrix output by the Kalman filter algorithm as the input of the particle filter algorithm and initialize it, including particle state initialization and particle weight initialization;

[0158] Particle prediction: predict the current particle state of each particle according to the state transfer equation, the expression is:

[0159]

[0160] in, represents the particle state of the i-th particle at time k; represents the particle state of the i-th particle at time k-1; u k represents the control input vector; represents the process noise of the i-th particle; represents the state transfer equation of the i-th particle;

[0161] Particle update: Update the particle weight according to the new observation data. The expression is:

[0162]

[0163] in, represents the weight of the i-th particle at time k; represents the weight of the i-th particle at time k-1; is the particle likelihood, which means that in a given particle state When the observed data z k The conditional probability density of ;

[0164] The particle likelihood is calculated based on the particle observation model, and its calculation formula is:

[0165]

[0166] in, Based on the particle state The predicted data obtained; R' represents the covariance matrix of particle observation noise; Represents the difference between observed data and predicted data;

[0167] The particle observation model is z k =h(x k )+v' k ;v' k represents the particle observation noise;

[0168] h(x k ) represents the nonlinear observation function, which is used to transform the current particle state Mapping to observation data z k , describes the nonlinear relationship between the observed data and the internal state vector of the system;

[0169] Particle resampling: Resample according to the updated particle weights, specifically:

[0170] Normalized particle weights: Normalize the particle weights so that the sum of the weights of all particles is 1. Then the weight of each particle becomes a probability, indicating the probability of this particle being selected.

[0171] The expression for normalizing particle weight is:

[0172] Where N is the total number of particles;

[0173] Select particles: reselect particles according to the normalized particle weights and form a new particle set, specifically:

[0174] Generate an equally spaced sequence of cumulative weights It is expressed as: j is the jth particle;

[0175] Randomly generate a uniformly distributed random number r, and r∈[0,1];

[0176] For each particle the particle state Find the first satisfaction The particle and copy this particle to the new particle set;

[0177] The particles are sampled repeatedly until the size of the new particle set reaches the size of the original particle set;

[0178] Particle state estimation: The final particle state estimation is calculated based on the weighted average of all particles. The calculation formula is: represents the final particle state estimate;

[0179] Particle iteration and output: Continue to iterate the particle prediction, particle update, and particle resampling steps according to the new observation data until the final particle state estimate is output.

[0180] Specifically, the present application adopts a particle filter algorithm for the second data fusion, and performs comprehensive processing on multiple data of different types or sources, which can effectively combine these data sources to more accurately estimate the state of the system. On the other hand, the particle filter algorithm can effectively process data with different degrees of noise, especially when the sensor noise is not a simple Gaussian noise. The particle filter can effectively suppress the influence of noise through resampling and weight update mechanisms, and enhance the robustness of the system to noise, which enables the system to maintain a high estimation accuracy when the data quality is inconsistent or there are serious errors.

[0181] In addition, the present application combines the Kalman filter algorithm with the particle filter algorithm, and uses the output obtained by the Kalman filter algorithm as the input of the particle filter algorithm. The combination of the two is used to handle multi-level data fusion problems and form a tighter data fusion. Kalman filtering is used as the first data fusion tool to process linear and Gaussian noise sensor data, while particle filtering is used as the second data fusion tool to process more complex situations of nonlinear and non-Gaussian noise. Therefore, in general, the combination of Kalman filtering and particle filtering for data fusion in the present application can improve the accuracy and robustness of the overall state estimation of the digital twin system; it can also adapt to more diverse system models (including linear, nonlinear and complex noise models); it can also reduce the computational burden, combining the advantages of both to reduce complexity. Therefore, it can solve both linear and nonlinear problems.

[0182] In an embodiment provided in the present application, the pan-tilt control module includes a pan-tilt motion unit and a pan-tilt control unit;

[0183] The gimbal motion unit is used to simulate the motion mode of the gimbal in the simulated operation environment, and the motion mode includes the gimbal pitch motion, the gimbal yaw motion and the gimbal tumbling motion, specifically:

[0184] The pan-tilt motion: the camera device and / or sensor device of the pan-tilt performs forward and backward tilt motion to adjust the vertical direction of the camera device and / or sensor device;

[0185] The pan-tilt yaw motion: the camera device and / or sensor device of the pan-tilt performs left and right translation and rotation to control the horizontal orientation of the camera device and / or sensor device;

[0186] The pan / tilt tumbling motion: the camera device and / or sensor device of the pan / tilt performs left-right rotation motion to change the horizontal angle of the camera device and / or sensor device;

[0187] The pan / tilt control unit is used to perform multi-dimensional control on the pan / tilt in the simulated operation environment, specifically including:

[0188] Position control: Use PID control algorithm to control the pan / tilt to rotate to a specific angle position and move position;

[0189] Speed ​​control: Control the rotation speed of the gimbal to ensure that the gimbal runs smoothly at the set speed.

[0190] Angle control: Use incremental control method to control the gimbal to rotate to the target angle within a predetermined time;

[0191] Attitude control: adjust the gimbal's target attitude according to preset target attitude parameters.

[0192] Specifically, in the gimbal control module of this embodiment, the motion mode of the gimbal is controlled by the gimbal motion unit, and these motion modes can be combined into compound motions. By simulating different combinations of these motion modes and relative position changes, the posture and motion trajectory of the gimbal can be accurately described; the specific motion form of the gimbal is controlled by the gimbal control unit, and different control algorithms are used to generate appropriate motor control instructions to control the motion of the gimbal according to the set target position, speed, acceleration and other parameters, so as to ensure the accuracy and responsiveness of the gimbal motion. It can be understood that this embodiment is mainly specifically developed for the gimbal motion unit and the gimbal control unit, and does not specifically limit the control devices such as the motor drive device in the gimbal.

[0193] In an embodiment provided in the present application, the gimbal control module further includes an optical image stabilization unit, an electronic image stabilization unit, an image processing unit, and a data processing unit;

[0194] The optical image stabilization unit is used to perform optical image stabilization processing when the gimbal is shooting, specifically:

[0195] S21, monitoring the movement changes of the camera device and the lens through the gyroscope and the accelerometer to obtain camera movement data;

[0196] S22, analyzing the camera motion data and identifying a jitter to be compensated that exceeds a preset jitter threshold;

[0197] S23, optically compensating the jitter to be compensated by adjusting the camera device and the optical elements in the lens;

[0198] S24, continuously adjusting the camera equipment and lens through real-time feedback data, and detecting the clarity of each frame of image captured, and then making continuous adjustments based on the detection results;

[0199] The electronic anti-shake unit is used to perform electronic anti-shake processing on the video image taken by the PTZ, specifically:

[0200] S31, classifying the video images based on a histogram distribution method into a first type of video images containing active motion of the camera device and a second type of video images not containing active motion of the camera device, specifically:

[0201] S311, extracting image frames in the video image sequence at equal intervals and giving a preset similarity threshold;

[0202] S312, comparing the hue and saturation histogram distributions of the image frames in pairs. If a comparison result is lower than a set similarity threshold, it means that the image has undergone a change that is not caused by random camera jitter, and the video image is classified as a first type of video image containing active motion of the camera device; if all comparison results are not lower than the set similarity threshold, the video image is classified as a second type of video image not containing active motion of the camera device;

[0203] It should be noted that the above-mentioned first type of video images that include the active movement of the camera device refers to the changes in the position (displacement) and angle of the camera device and lens caused by the flight of the UAV under operating instructions. This change is driven by the flight control system and operating instructions, and is therefore regarded as the active movement of the camera device. The second type of video images that do not include the active movement of the camera device are the opposite.

[0204] S32, performing a first type of electronic anti-shake processing on the first type of video images, including:

[0205] S321, based on the first type of video images, using a comprehensive corner detection algorithm and a random sampling consistency algorithm to obtain a motion trajectory of the camera device; wherein the comprehensive corner detection algorithm is a fusion of a FAST corner detection algorithm and a Shi-Tomasi corner detection algorithm;

[0206] The obtaining of the motion trajectory of the camera device is specifically as follows:

[0207] S3211, performing preliminary corner point detection by using the FAST corner point detection algorithm in the fusion corner point detection algorithm to extract candidate feature points of adjacent image frames in the first type of video image;

[0208] S3212, performing refined corner point detection by using the Shi-Tomasi corner point detection algorithm in the fusion corner point detection algorithm, verifying the accuracy of the candidate feature points and selecting optimized feature points;

[0209] S3213, using a bidirectional optical flow method to track the optimized feature points and perform feature matching;

[0210] S3214, using the random sampling consistency algorithm to eliminate feature points with incorrect matching, and obtaining matching feature points of adjacent image frames after elimination;

[0211] S3215, calculating affine transformation parameters using a least squares method based on the matching feature points;

[0212] S3216, determining a transformation relationship between adjacent image frames based on the affine transformation parameters, and accumulating the transformation relationship to calculate a motion trajectory of the camera device;

[0213] S322, then using Gaussian filtering to smooth the motion trajectory of the camera device to obtain a jitter motion estimation;

[0214] S323, then using the jitter motion estimation amount to perform affine transformation on adjacent image frames to generate a stable frame image;

[0215] S324, performing compensation processing on the stabilized frame image using an interpolation algorithm;

[0216] S325, finally inputting the first type of video images as a complete video sequence into a multi-thread execution model for multi-thread processing, and terminating the multi-threading until all video image frames are processed;

[0217] S33, performing a second type of electronic anti-shake processing on the second type of video images, including:

[0218] S331, extracting a plurality of image frames from the second type of video images and distinguishing a first frame and subsequent frames other than the first frame;

[0219] S332, use the feature point extraction method based on the Harris matrix to extract features from the first frame and obtain the feature points of the first frame; wherein, the feature point extraction method based on the Harris matrix specifically divides the first frame into a grid, and calculates the Harris matrix of each point in the grid to select feature points. This embodiment only extracts feature points from the first frame. It should be noted that this embodiment does not specifically describe the specific calculation method of the Harris matrix and the selection of feature points.

[0220] S333, using a bidirectional optical flow method to perform feature matching, tracking the position of the feature point of the first frame in the subsequent frames to obtain the coordinates of the feature point of the current frame (the current frame in the subsequent frames), and reversely tracking the feature point of the current frame in the first frame;

[0221] S334, setting a confidence level to constrain the feature points of the first frame; its purpose is to ensure that the feature points of the current frame stable transformation estimation participating in the matching appear in all frames as much as possible, and to avoid feature point estimation errors caused by a large value of the Harris matrix of a certain feature point.

[0222] S335, obtaining a transformation matrix based on the feature points of the first frame and the feature points of the current frame;

[0223] S336, applying the transformation matrix to the current frame to obtain an anti-shake frame, and performing compensation processing on the anti-shake frame by using an interpolation algorithm;

[0224] The image processing unit is used to perform subsequent processing on the video image to generate a PTZ output image, including image adjustment, image compression and image optimization;

[0225] Image adjustment: Automatically fill in the light of the video image acquired when the light is weak; also perform adaptive frame filling on the distorted image;

[0226] Image compression: Use image spectrum compression technology to compress video images; used to reduce the transmission delay of video images, indirectly increase the wireless communication distance, and enhance the stability of image transmission.

[0227] Image optimization: Use image correction algorithms (such as distortion correction, geometric correction, etc.) to compensate for image distortion caused by the lens;

[0228] The data processing unit is used to process the PTZ data acquired by the PTZ, including data compression and data response;

[0229] Data compression: The PTZ data is compressed using a time domain compression algorithm based on SDT (Swivel Door Transformation), thereby reducing the transmission delay of real-time sensor data, indirectly increasing the wireless communication distance, and enhancing the stability of data transmission;

[0230] Data response: Use hard real-time scheduling algorithm and parallel computing to respond to the PTZ data, so as to improve the response speed of PTZ control and the real-time performance of data interaction. Among them, the hard real-time scheduling algorithm can ensure that high-priority tasks are executed in a timely manner in task scheduling, avoid delays, and ensure the real-time and accuracy of PTZ operations.

[0231] Specifically, the gimbal control module of this embodiment mainly processes the jitter generated by the gimbal during shooting, and performs optical image stabilization through the optical image stabilization unit, so as to accurately control the real-time adjustment of the camera device components such as the lens, so as to effectively eliminate the picture jitter caused by the movement of the camera and other camera devices or external disturbances, thereby ensuring the stability and clarity of the picture during shooting. The electronic image stabilization unit is also used to process the already captured images or video frames, and the image instability caused by camera shake is reduced by optimizing the image stability. This embodiment combines the two technologies of optical image stabilization and electronic image stabilization to jointly eliminate the jitter caused by drone shooting, thereby ensuring the stability and accuracy of the gimbal control.

[0232] It can be understood that, in the electronic image stabilization unit, this embodiment classifies the video images and performs anti-shake processing on the two types of images separately, thereby solving the problem in the prior art that only videos with a single category of camera motion trajectory can be anti-shake processed. It can meet the anti-shake requirements of videos with different motion trajectories, thereby effectively improving the stability of the picture.

[0233] It can also be understood that this embodiment uses a comprehensive corner detection algorithm based on the fusion of the FAST corner detection algorithm and the Shi-Tomasi corner detection algorithm to extract feature points, and uses the FAST corner detection algorithm for preliminary rapid detection. The fast detection speed of the algorithm can be used to quickly scan the image during the flight of the drone, thereby improving the overall processing speed; the Shi-Tomasi corner detection algorithm is then used to refine the candidate corner points detected by the FAST corner detection algorithm, further verify the reliability and accuracy of these corner points, and eliminate those unstable corner points that may be affected by noise or errors, so as to improve the reliability of the final detection result. Therefore, this embodiment can improve the detection accuracy while ensuring real-time performance by fusing the above two algorithms, thereby effectively improving the image processing performance of the drone, thereby meeting the needs of various different tasks.

[0234] In summary, the gimbal control module of the present application, through the joint action of multiple units such as the gimbal motion unit, the gimbal control unit, the optical image stabilization unit, the electronic image stabilization unit, the image processing unit and the data processing unit, enables the gimbal to smoothly and accurately perform various motion tasks and ensure the stability and reliability of the gimbal.

[0235] In one embodiment provided in the present application, the flight control module includes a flight mode unit, a flight attitude unit, a path planning and navigation unit, and a flight controller;

[0236] The flight mode unit is used to simulate the flight mode of the UAV and perform corresponding flight operations; the flight modes include manual control, automatic flight and target tracking;

[0237] The flight attitude unit is used to simulate the flight attitude of the UAV and perform corresponding flight operations; the flight attitude includes pitch flight, roll flight and yaw flight;

[0238] The path planning and navigation unit is used to plan the flight path of the UAV and perform flight navigation and flight obstacle avoidance;

[0239] The flight controller is used to generate flight control instructions for controlling the UAV according to the current flight status of the UAV, specifically:

[0240] Get the real-time status data of the drone’s current flight status;

[0241] Inputting the real-time state data into the flight controller, and estimating the current flight state in combination with the aircraft dynamics model to obtain a flight state estimate;

[0242] Using a flight control algorithm, calculate a subsequent flight state of the UAV based on the flight state estimate and the desired flight target;

[0243] The environmental data is acquired and the flight control instruction is obtained based on a subsequent flight status.

[0244] Specifically, in the simulation operation environment, the present embodiment simulates the flight state of the UAV and performs flight control through the flight control module, involving the above-mentioned multiple interrelated modules, which work together to simulate the UAV's motion mode, motion posture, plan the flight path and navigate to avoid obstacles, and calculate and generate control instructions for the UAV's subsequent flight according to the UAV's current motion state, and perform subsequent flight control according to the control instructions. This achieves comprehensive testing and control of the UAV's flight process.

[0245] On the other hand, the user can use the simulation gimbal to simulate the operation of the drone at various stages, such as take-off, flight, and landing. This simulation training can help users master the control method of the drone, improve flight skills, and become familiar with the handling methods for various flight situations. In addition, during the simulated flight, the simulation gimbal can also identify the user's operational errors and give corresponding prompts and correction suggestions, which helps users to promptly discover and correct their own operational problems and improve the safety and stability of the flight.

[0246] It should be noted that the flight attitude mentioned in this embodiment refers to various flight attitudes of the UAV, including but not limited to pitch, roll and yaw. This flight attitude is different from the gimbal motion mode mentioned by the gimbal motion unit in the gimbal control module. The former refers to the flight attitude of the UAV, and the latter refers to the motion mode of the gimbal. Although both include pitch, roll and yaw, there are essential differences in their specific meanings and the objects they target.

[0247] In an embodiment provided in the present application, the establishing of a multi-user collaboration mechanism includes:

[0248] Role definition: Determine the user roles of different operators, as well as the collaborative tasks and collaborative goals between different user roles; for example, commander, operator, logistics support personnel, etc.

[0249] Information acquisition: acquiring the flight status information and environmental data of multiple UAVs and performing information fusion to obtain global situation information;

[0250] Task allocation: based on the global situation information, prioritize the collaborative tasks and allocate flight tasks to different user roles according to the prioritized tasks;

[0251] Network communication: Establish multiple collaborative communication units with different communication functions based on different communication needs;

[0252] Multi-user collaboration: Select a collaborative communication unit that matches the flight mission based on the flight mission, and perform multi-user collaborative operations according to the collaborative goals and user roles.

[0253] Specifically, the above-mentioned collaborative communication unit includes a real-time monitoring and feedback sub-unit based on UDP and WebSocket: UDP (User Datagram Protocol) is used for real-time control instructions and data transmission; WebSocket is used for two-way communication with the ground control station to realize real-time monitoring and feedback of the flight status, and a task scheduling and synchronization sub-unit based on MQTT and TCP / IP: MQTT (Message Queuing Telemetry Transport) is used for lightweight task scheduling, status update, message broadcasting, etc., and TCP / IP (Transmission Control Protocol / Internet Protocol) is used for reliable task transmission and log synchronization; a low-power long-distance sub-unit based on LoRa (Long Range) and Zigbee: used for low-power communication and position synchronization between drone groups, and a group collaborative simulation sub-unit based on HLA (High-Level Architecture) and DIS (Distributed Interactive Simulation): used for complex tasks that require precise simulation and high coordination, such as simulation and simulation of military training or large-scale multi-drone collaborative tasks. The multi-user cooperative operation of the drone of this embodiment usually requires the use of a combination of multiple protocols to meet different needs. And through the multi-user cooperative module to establish a multi-user cooperative mechanism, it can support multiple operators to participate in simulation training and improve.

[0254] In an embodiment provided in the present application, in the personalized training module, the personalized training tasks include electric cruise tasks, geographic surveying and mapping tasks, and fire rescue tasks.

[0255] Specifically, the simulation gimbal can simulate different flight training tasks, such as power inspection, geographic surveying and mapping, fire rescue, etc., allowing users to plan and practice tasks in a virtual environment. This helps users better understand task requirements, optimize task execution plans, and improve their ability to cope with actual tasks.

[0256] Power inspection: The drone uses the camera on board to obtain detailed images of power lines and combines image recognition technology to monitor power facilities and detect anomalies. It is also equipped with an infrared thermal imager to monitor the temperature changes of power equipment in real time, promptly detect abnormal hot spots of equipment and predict potential failure risks, thereby achieving preventive maintenance and reducing the risk of equipment damage.

[0257] Geographical surveying and mapping: Simulate various complex geographical environments, such as mountains, plains, lakes, etc., as well as different climatic conditions, such as sunny days, rainy days, foggy days, etc. These simulated environments can help users understand the flight performance of drones under different terrain and weather conditions, so as to formulate more reasonable flight plans and operation strategies. Users can simulate actual geographic surveying and mapping tasks in a virtual environment, evaluate the mission execution capability and efficiency of drones by simulating different mission scenarios, optimize mission planning, and improve the accuracy and integrity of surveying and mapping data.

[0258] Firefighting and rescue: Simulate different fire scenes and rescue missions, so that rescuers can conduct flight training without actual drones, which helps rescuers become familiar with the control methods of drones, master flight skills, and become familiar with the operating characteristics in various flight environments. Improve the ability to respond to emergencies by simulating different situations. Also improve the command efficiency in actual rescue by simulating the process of coordinated command between the front and rear. Also use the data of simulation training to analyze the fire situation and learn rescue decisions.

[0259] In summary, the immersive operation simulation gimbal of the present application is used to simulate the flight environment and operation process of the drone, providing users with a highly realistic immersive flight experience. By combining simulation software, physical equipment, virtual reality technology, augmented reality technology and other technologies, it is possible to simulate the operations of various stages such as take-off, flight, and landing of the drone, making the user feel as if they are in a real flight scene. Users can learn and train flight operations in a safe and controllable environment without actually flying the drone, which not only reduces training costs but also avoids potential safety risks. At the same time, the simulation function of the simulation gimbal can also help users better understand the performance characteristics and operating requirements of the drone, providing strong support for actual flight operations.

[0260] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0261] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A drone immersive operation simulation gimbal, characterized by: include: Environment construction module: used to construct a simulated operation environment and obtain environmental data of the simulated operation environment; PTZ control module: used to simulate the motion mode of the PTZ and control the PTZ, as well as process the PTZ shooting images and PTZ control data; Flight control module: used to simulate the flight status of the drone and perform flight control; Immersive feedback module: used for obtaining the operation data of the operator in the simulated operation environment, obtaining feedback information by analyzing the operation data and the environment data, and adaptively adjusting the simulated operation environment according to the feedback information; Multi-user collaboration module: used to establish a multi-user collaboration mechanism to support multiple operators to participate in simulation training; Personalized training module: used to collect and record various data during flight control and gimbal control in real time, and conduct comprehensive analysis and evaluation based on the operator's training performance, and generate personalized training tasks for the operator based on the analysis and evaluation results; The gimbal control module includes a gimbal motion unit, a gimbal control unit, an optical image stabilization unit, an electronic image stabilization unit, an image processing unit and a data processing unit; The pan / tilt motion unit is used to simulate the motion mode of the pan / tilt in the simulated operation environment, wherein the motion mode includes the pan / tilt pitch motion, the pan / tilt yaw motion and the pan / tilt tumble motion; The pan / tilt control unit is used to perform multi-dimensional control on the pan / tilt in the simulated operating environment; The optical image stabilization unit is used to perform optical image stabilization processing when the gimbal is shooting; The electronic anti-shake unit is used to perform electronic anti-shake processing on the video image taken by the PTZ; The image processing unit is used to perform subsequent processing on the video image to generate a PTZ output image, including image adjustment, image compression and image optimization; The data processing unit is used to process the PTZ data acquired by the PTZ, including data compression and data response.

2. The drone immersive operation simulation gimbal according to claim 1, characterized in that: The environment construction module includes a digital twin module, a virtual reality module, an augmented reality module and a sensor fusion module; The digital twin module uses digital twin technology to build a digital twin system for immersive operation of drones; The virtual reality module uses virtual reality technology to build a three-dimensional virtual environment for immersive operation of the drone; The augmented reality module uses augmented reality technology to build a three-dimensional real environment for immersive operation of the drone; The sensor fusion module, based on the digital twin system, uses sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment to form the simulated operation environment for immersive operation of the drone; Wherein, constructing the digital twin system includes: Obtaining a construction request, and obtaining a system construction project list by analyzing the construction request, wherein the construction request includes application scenario requirements, function requirements, data management requirements, and user interaction requirements; Establishing an initial twin system according to the system construction project list; Conducting multiple operation tests based on the initial twin system to obtain operation sample data; The initial twin system is optimized and adjusted using the running sample data to generate a final digital twin system.

3. The drone immersive operation simulation gimbal according to claim 2, characterized in that: The establishing of the initial twin system according to the system construction project list includes: Analyze the system construction project list to obtain system project modules, specifically: Acquire an immersive scene project module according to the application scenario requirements; Acquire a pneumatic analysis project module according to the functional requirements; Acquire a data management project module according to the data management requirements; A human-computer interaction project module is obtained according to the user interaction requirement.

4. The drone immersive operation simulation gimbal according to claim 3, characterized in that: The step of establishing an initial twin system according to the system construction project list further includes: The system project modules are modularly analyzed to establish an initial twin system, specifically: Determining multiple equipment types of UAVs and UAV support equipment based on the aerodynamic analysis project module, and establishing an equipment model corresponding to each equipment type; Obtaining structural components corresponding to each equipment type and constructing a structural component model based on the structural components; Acquire the interaction data of the UAV operation based on the human-computer interaction project module, and establish a fault model corresponding to the device model according to the interaction data; Acquire the environmental attributes of the UAV operation based on the immersive scene project module; Acquire the navigation attributes of the UAV operation based on the data management project module; Establishing a monitoring point model corresponding to each equipment type according to the environmental attributes and the navigation attributes; The initial twin system is established according to the equipment model, the structural component model, the monitoring point model and the fault model.

5. The drone immersive operation simulation gimbal according to claim 2, characterized in that: The adopting of sensor fusion technology to perform data fusion on the three-dimensional virtual environment and the three-dimensional real environment includes: Data collection: collecting virtual environment data in the three-dimensional virtual environment; collecting real environment data in the three-dimensional real environment; Data processing: performing data preprocessing on the virtual environment data and the real environment data; the data preprocessing includes data cleaning, data denoising, time synchronization and space registration; Data fusion: The preprocessed data is fused to generate a fused data set, including: The Kalman filter algorithm is used for the first data fusion; The particle filter algorithm is used for the second data fusion; Generating the fused data set with the output data after the two data are fused; Update rendering: update the state of the three-dimensional virtual environment in real time based on the fused data set, and perform environmental rendering on the updated three-dimensional virtual environment to generate a visual effect consistent with the three-dimensional real environment.

6. The drone immersive operation simulation gimbal according to claim 5, characterized in that: The first data fusion is performed using the Kalman filter algorithm, including: Define the system model: establish the state equation, observation equation and noise model; Initialization settings: set the initial state estimate and initial state covariance matrix; State prediction: performing state prediction according to the state equation to obtain a predicted current state estimate and a predicted current state covariance matrix; State update: obtaining observation data according to the observation equation and using the observation data to update the current state estimate and the current state covariance matrix; Iterative process: Get new observation data and repeat the state prediction step and state update step; Output results: Output the final optimal state estimate and optimal state covariance matrix.

7. The drone immersive operation simulation gimbal according to claim 6, characterized in that: The second data fusion is performed by using a particle filter algorithm, including: Initialize particle filter: use the optimal state estimate and optimal state covariance matrix output by the Kalman filter algorithm as the input of the particle filter algorithm and initialize it, including particle state initialization and particle weight initialization; Particle prediction: predict the current particle state of each particle based on the state transfer equation; Particle update: Update the particle weight according to the new observation data; Particle resampling: resampling according to the updated particle weights; Particle state estimation: Calculate the final particle state estimation based on the weighted average of all particles; Particle iteration and output: Continue to iterate the particle prediction, particle update, and particle resampling steps according to the new observation data until the final particle state estimate is output.

8. The drone immersive operation simulation gimbal according to claim 1, characterized in that: The optical image stabilization process is performed during the gimbal shooting, specifically: Monitor the movement changes of the camera device and the lens through the gyroscope and accelerometer to obtain the camera movement data; Analyzing the camera motion data and identifying jitter to be compensated that exceeds a preset jitter threshold; Optically compensating the jitter to be compensated by adjusting the camera device and the optical elements in the lens; The camera equipment and lens are continuously adjusted through real-time feedback data, and the clarity of each frame of image captured is tested, and continuous adjustments are made based on the test results.

9. The drone immersive operation simulation gimbal according to claim 1, characterized in that: The electronic anti-shake processing of the video image captured by the PTZ is specifically performed as follows: Classifying the video images based on a histogram distribution method into a first type of video images that include active motion of a camera device and a second type of video images that do not include active motion of a camera device; Performing a first type of electronic anti-shake processing on the first type of video images; The second type of electronic anti-shake processing is performed on the second type of video images.

10. The drone immersive operation simulation gimbal according to claim 1, characterized in that: The flight control module includes a flight mode unit, a flight attitude unit, a path planning and navigation unit and a flight controller; The flight mode unit is used to simulate the flight mode of the UAV and perform corresponding flight operations; the flight modes include manual control, automatic flight and target tracking; The flight attitude unit is used to simulate the flight attitude of the UAV and perform corresponding flight operations; the flight attitude includes pitch flight, roll flight and yaw flight; The path planning and navigation unit is used to plan the flight path of the UAV and perform flight navigation and flight obstacle avoidance; The flight controller is used to generate flight control instructions for controlling the UAV according to the current flight status of the UAV.

Images