Cross-medium control method and system based on unmanned aerial vehicle visual system and storage medium

Abstract

The invention relates to the field of unmanned aerial vehicle control, and discloses a cross-medium control method and system based on an unmanned aerial vehicle visual system and a storage medium. The method comprises the steps of obtaining a first prediction time when an unmanned aerial vehicle arrives at critical positions of a first area and a second area and a current time when the unmanned aerial vehicle arrives at a current position in the first area, and if a time difference is smaller than a first threshold, adjusting configuration parameters of the unmanned aerial vehicle; determining an optimal path of the unmanned aerial vehicle from the critical position to a target position in the second area, and setting a plurality of first positions on the optimal path; establishing a state control model, inputting a current image of a current position and a reference image of an adjacent first position, and outputting an estimated value of the state adjustment amount of the unmanned aerial vehicle; taking the adjacent first position as a new current position, and repeating the step until the optimal path is traversed; when the unmanned aerial vehicle returns to the first area, the return critical time for reaching the critical position is obtained, the configuration parameters are adjusted, and the stability of the unmanned aerial vehicle in different media is improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control, and particularly to a cross-media control method, system, and storage medium based on a UAV vision system. Background Art

[0002] With the rapid development of UAV technology, UAVs are increasingly widely used in various complex environments, including multiple media such as air, underwater, and land. However, UAVs face many challenges when crossing media (such as entering underwater from the air). When crossing media, the stability of the UAV will be affected. In the underwater environment, the GPS signal is usually very weak or completely unavailable, which makes traditional GPS-based positioning methods ineffective.

[0003] Existing underwater path planning methods, such as the Chinese patent application with publication number CN113342008A, disclose a path planning system and method for sea-air collaborative underwater target tracking. The method includes: an aerial monitoring device cluster obtains the target position information of the detection target, performs a first path planning along the sea surface monitoring device lane, constructs a sea surface navigation area map, and transmits the sea surface navigation area map to the sea surface monitoring device cluster; the sea surface monitoring device cluster performs a second path planning along the sea surface monitoring device lane according to the sea surface navigation area map and its own position information, reaches the vicinity of the detection target, detects the underwater environment in the vicinity, constructs an underwater obstacle environment map, and transmits the underwater obstacle environment map to the underwater monitoring device cluster; the underwater monitoring device cluster performs a third path planning according to the underwater obstacle environment map and tracks to the position of the detection target. The present invention adopts cluster collaborative optimization, reduces the number of iterations, improves the optimization efficiency, quickly tracks the target position, and improves the autonomous collaborative tracking ability. However, the above information needs to be transmitted at multiple levels from the air to the sea surface and then to the underwater, which may cause delays and poor real-time performance, and may affect the tracking efficiency and accuracy.

[0004] For another example, in the existing cross-medium control method, such as the Chinese patent application with the publication number CN115509246A, a longitudinal attitude control method for a water-air amphibious unmanned aerial vehicle (UAV) with a buoy during cross-medium takeoff is disclosed, including the following steps: S1. Divide the control system of the UAV into three parts: a power system, a controller, and a longitudinal motion dynamics model of the UAV; S2. The power system solution relies on a virtual force N to change the forward speed of the UAV and changes the flight attitude through an elevator; S3. The controller controls the longitudinal attitude during takeoff; S4. The dynamics model controls the forces and torques acting on the UAV according to the output of the power system and adjusts the longitudinal attitude of the UAV. The above existing technology mainly uses a physics model to achieve attitude control of the UAV during cross-medium, however, the above solution mainly uses a physics model to achieve attitude control of the UAV during cross-medium, and the physical model requires precise parameter settings, otherwise there will be severe attitude fluctuations, affecting flight stability.

[0005] Therefore, a cross-medium control method based on the UAV vision system is needed to ensure the stability of the UAV during cross-medium while improving the efficiency of the UAV working underwater. Summary of the Invention

[0006] To solve the above technical problems, the present application provides a cross-medium control method, system, and storage medium based on the UAV vision system for improving the stability of the UAV during cross-medium.

[0007] In the first aspect, the present application provides a cross-medium control method based on the UAV vision system, the method includes: Step S1: Set the departure position of the UAV in the first area, start timing from the departure of the UAV, obtain the first predicted time when the UAV reaches the critical position between the first area and the second area, and the current time of the current position of the UAV during flight. If the difference between the first predicted time and the current time is less than a first threshold, adjust the configuration parameters of the UAV based on an adaptive adjustment mechanism; Step S2: After reaching the second area, take the critical position as the starting point of the current position, obtain the current image of the UAV at the current position, and perform calibration processing on the current image; Step S3: Determine the optimal path of the UAV from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, and input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position into the state control model to generate an estimated value of the state adjustment amount of the UAV; Step S4: The drone moves from the current position to an adjacent first position based on the state adjustment amount, takes the adjacent first position as the new current position, and repeats this step until the optimal path is traversed; Step S5: When the drone returns from the second area to the first area, obtain the return critical time when the drone reaches the critical position, and adjust the configuration parameters of the drone based on the return critical time.

[0008] Combined with the first aspect, in the first implementation manner of the first aspect of this application, obtaining the first predicted time for the drone to reach the critical positions of the first area and the second area includes: Obtain the real-time flight data of the drone in the first area, establish a time prediction model, input the real-time flight data into the time prediction model, the time prediction model is a neural network model, and train the time prediction model based on the historical flight data of the drone in the first area. The historical flight data includes the flight path of the drone from the first area to the critical position of the second area, historical wind data, and flight time. Each flight path consists of multiple position points, and each position point has a corresponding flight time.

[0009] Combined with the first aspect, in the second implementation manner of the first aspect of this application, determining the optimal path of the drone from the critical position to the target position includes: Create a three-dimensional environment map of the second area, set multiple navigation positions in the three-dimensional environment map, establish a weighted topological map of the second area with each navigation position as a node, calculate the similarity between the reference image corresponding to each navigation position and the reference image of the adjacent navigation position, set the weight between adjacent navigation positions based on the similarity, and define the node sequence with the minimum sum of edge weights from the start node to the end node in the weighted topological map as the optimal path of the drone. The start node and the end node are the critical position and the target position respectively.

[0010] Combined with the first aspect, in the third implementation manner of the first aspect of this application, generating an estimated value of the state adjustment amount of the drone includes: The state control model is a neural network model. The neural network model extracts the feature quantities in the current image and the reference image, obtains the auxiliary data of the current image and the reference image at the first position. The auxiliary data includes the image shooting position, the acceleration and angular velocity of the drone, calculates the state difference of the drone from the current position to the adjacent first position based on the auxiliary data. The state difference includes position difference, direction difference, and distance difference, and obtains the labeled data by labeling the state difference; Taking the characteristic quantity and the annotation data as training data of the state control model, the state control model predicts and generates a state difference from the current position to an adjacent first position, and defines the state difference as an estimated value of each state adjustment amount of the UAV.

[0011] Combined with the first aspect, in the fourth implementation manner of the first aspect of the present application, obtaining the return critical time for the UAV to reach the critical position includes: Obtaining a target image at the target position and extracting feature information, comparing the feature information with the standard features of the target object in the database to lock the target object, defining the time point when the UAV reaches the second position as the starting time point, where the second position is a position at a first value away from the critical position, taking pictures of the second area every target time period starting from the starting time point to obtain area images, obtaining the detection range of the vision camera within each target time period, and extracting feature information from the area images, matching the feature information within each target time period with the first features in the database, where the first features are the feature information obtained from the area images of the critical position, matching the detection range with the standard detection range, and if both the feature information and the detection range match successfully, defining the current target time period as the return critical time.

[0012] Combined with the first aspect, in the fifth implementation manner of the first aspect of the present application, adjusting the configuration parameters of the UAV based on an adaptive adjustment mechanism includes: Based on different dynamic models of the UAV in the first area and the second area, calculating the difference value of the physical parameters of the UAV in the two areas, where the physical parameters include the central position, moment of inertia, and resistance information of the UAV, calculating the compensation parameters of the UAV based on the difference value of the physical parameters, adjusting the control signal of the UAV based on the compensation parameters, and adjusting the configuration parameters of the UAV based on the control signal, where the configuration parameters include flight parameters and sensor parameters.

[0013] In a second aspect, the present application provides a cross-media control system based on a UAV vision system, the system includes: A departure control module, configured to set the departure position of the UAV in the first area, start timing from the departure of the UAV, obtain the first predicted time for the UAV to reach the critical positions of the first area and the second area, and the current time of the current position of the UAV during flight, and if the difference between the first predicted time and the current time is less than a first threshold, adjust the configuration parameters of the UAV based on an adaptive adjustment mechanism; An image correction module, which is used to, after reaching the second area, take the critical position as the starting point of the current position, obtain the current image of the UAV at the current position, and perform calibration processing on the current image; A path formation module, which is used to determine the optimal path of the UAV from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, and input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position into the state control model to generate an estimated value of the state adjustment amount of the UAV; A state adjustment module, which is used to enable the UAV to move from the current position to the adjacent first position based on the state adjustment amount, take the adjacent first position as the new current position, and repeat this step until the optimal path is traversed; A return control module, when the UAV returns from the second area to the first area, obtains the return critical time when the UAV reaches the critical position, and adjusts the configuration parameters of the UAV based on the return critical time.

[0014] The third aspect of the present application provides a computer-readable storage medium, in which instructions are stored, and when they run on a computer, the computer is enabled to execute the above-mentioned cross-media control method based on the UAV vision system.

[0015] In the technical solution provided by the present application, by establishing a neural network model to predict the time when the UAV reaches the critical position, when the difference between the predicted time and the actual time is less than the set threshold, the configuration parameters of the UAV are adjusted to ensure the safety and stability of the UAV during the cross-media process; the adaptive adjustment mechanism can dynamically adjust the configuration parameters of the UAV according to real-time data, enabling the UAV to better adapt to different flight environments and mission requirements.

[0016] By establishing a weighted topological map of the second area and determining the optimal path from the critical position to the target position, the flight time and energy consumption can be reduced. By setting multiple first positions on the optimal path, the complex path can be decomposed into multiple small segments, and each small segment can be precisely controlled and adjusted, thereby improving the safety and reliability of the path; through the state control model, an accurate state adjustment amount can be generated, enabling the UAV to accurately move from the current position to the adjacent first position, ensuring the accuracy of path tracking, and at the same time solving the problem of weak GPS signals in the underwater environment and inability to navigate; when the UAV returns to the first area, by predicting the time to reach the critical position, the configuration parameters can be adjusted in advance; it can ensure the stable flight state of the UAV during the return process, reduce the return failure caused by navigation errors or environmental changes, and improve the safety of the return. Description of the Drawings

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of an embodiment of the cross-media control method based on the UAV vision system in the embodiments of the present application; Figure 2 It is a structural diagram of the UAV in the embodiments of the present application; Figure 3 It is a weighted topological graph of the second area in the embodiments of the present application; Figure 4 It is a schematic diagram of an embodiment of the cross-media control system based on the UAV vision system in the embodiments of the present application; In the figure: 1, propeller; 2, motor; 3, vision sensor; 4, ground station signal receiver; 5, flight control board; 6, electronic speed controller; 7, vision camera; 8, UAV bracket. Specific embodiments

[0019] The embodiments of the present application provide a cross-media control method, system and storage medium based on the UAV vision system. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above accompanying drawings of the present application are used to distinguish similar objects and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "comprising" or "having" and any variation thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0020] For ease of understanding, the following describes the specific process of the embodiments of the present application. Please refer to Figure 1 , an embodiment of the cross-media control method based on the UAV vision system in the embodiments of the present application includes: Step S1: Set the departure position of the UAV in the first area, start timing from the departure of the UAV, obtain the first predicted time when the UAV reaches the critical position between the first area and the second area, and the current time of the current position during the flight of the UAV. If the difference between the first predicted time and the current time is less than the first threshold, adjust the configuration parameters of the UAV based on the adaptive adjustment mechanism.

[0021] Specifically, as Figure 2 shown, it is the structural diagram of a drone, including: propeller 1, motor 2, vision sensor 3, ground station signal receiver 4, flight control board 5, electronic speed controller 6, vision camera 7, and drone bracket 8. Among them, propeller 1 provides the lift and thrust required for the drone to fly; motor 2 is the main power source of the drone and can achieve the motion control of the drone by adjusting the rotation speed; vision sensor 3 is used to capture environmental images and videos and supports the vision navigation, target detection, and obstacle avoidance functions of the drone; ground station signal receiver 4 receives the command signals from the ground control station to achieve remote control and data transmission; flight control board 5 is the core control unit of the drone, responsible for attitude stabilization, navigation control, and data processing; electronic speed controller 6 is an electronic speed governor that controls the rotation speed and direction of motor 2 and adjusts the power output of propeller 1 according to the commands of flight control board 5 to achieve stable flight; drone bracket 7 provides structural support and stability for the drone; drone bracket 8 is used to capture images or videos in real time, and can support the drone to perform monitoring, mapping, or target identification, and is applied to various scenarios such as mapping and search and rescue.

[0022] The vision system of the drone consists of vision camera 7 and flight control board 5. First, vision camera 7 is used to obtain the images of the environment around the drone, identify the environmental features in the images, and send the obtained image data to flight control board 5. After receiving the image data, flight control board 5 automatically analyzes the images and quickly identifies the type and features of the target object by comparing the image data with the information in the database. In addition, flight control board 5 uses image processing algorithms to calculate the specific azimuth and distance of the target, so as to provide accurate position information for the drone.

[0023] Suppose the drone receives a control command from a remote end through ground station signal receiver 4, requiring the drone to start from the first area (air area), enter the second area (water area) after passing through the critical position, search for the target object in the water area, confirm the target position of the target object, and return to the air area. The target object may be an enemy mine, an underwater spy device, etc.; and when the drone needs to transition from air to water, its dynamic characteristics, resistance, buoyancy, etc. will all change. In order to ensure that the drone can safely and stably transition from the air area to the water area, the present invention uses a neural network model to predict the first prediction time when the drone reaches the critical position between the first area and the second area. Starting from the time when the drone starts, the difference between the first prediction time and the current time is calculated regularly (every 3 seconds). If the difference is less than the first threshold, the first threshold (such as 5 seconds) is set, that is, when the drone is approaching the critical position, the configuration parameters can be adjusted in advance through electronic speed controller 6 to ensure a smooth entry from air to water and improve the stability and reliability of the cross-media mission.

[0024] Step S2: After reaching the second area, using the critical position as the starting point of the current position, obtain the current image of the drone at the current position and perform calibration processing on the current image.

[0025] Specifically, when the drone enters the underwater area from the air area, the underwater environment may introduce additional noise into the image, such as particulate matter or bubbles, and water will absorb and scatter light, resulting in color distortion. Therefore, it is necessary to perform calibration processing on the current image captured by the drone using a vision camera. For example, the Auto White Balance algorithm can be used to correct the color distortion caused by water absorbing different wavelengths of light.

[0026] Step S3: Determine the optimal path of the drone from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, and input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position in the optimal path into the state control model to generate an estimated value of the state adjustment amount of the drone.

[0027] Specifically, based on the environmental map of the underwater area, multiple representative navigation positions are preset in advance, which can be specific landmarks, marker points or known navigation points underwater. At each navigation position, a reference image is captured using the vision camera 7, and the reference image and its corresponding navigation position information are stored in the storage device of the drone. A weighted topological map from the critical position to the target position is established based on the reference images, and the optimal path is selected from the network topological map. The navigation positions on the optimal path except the target position are defined as the first positions, and these first positions will serve as relay points for the drone to navigate underwater to guide the drone to move in the underwater environment.

[0028] The state control model is a neural network model. This model takes the calibrated current image and the reference image corresponding to the first position adjacent to the critical position in the optimal path as inputs. The reference image is a standard image stored in advance in the database. The convolutional layer is used to extract the feature vectors of the current image and the reference image. By comparing the feature vectors of the current image and the reference image, the adjustment amount of the current state (such as position, direction, speed, etc.) of the drone relative to the adjacent state (the first position) is estimated.

[0029] Step S4: The drone moves from the current position to the adjacent first position based on the state adjustment amount, takes the adjacent first position as the new current position, and repeats this step until the optimal path is traversed.

[0030] Specifically, taking the adjacent first position as the new current position, the current image at the new current position and the reference image at the adjacent first position in the optimal path are used as the inputs of the state control model to generate the state adjustment value of the new current position. Through this method, the UAV can accurately identify its current position after entering the underwater environment, achieve high-precision positioning, and use the weighted topological map to realize the navigation of the UAV in the water area, enhancing the adaptability of the UAV to complex environmental changes.

[0031] Step S5: When the UAV returns from the second area to the first area, obtain the return critical time when the UAV reaches the critical position, and adjust the configuration parameters of the UAV based on the return critical time.

[0032] Specifically, when the UAV returns from the second area (underwater) to the first area (air), due to the influence of factors such as buoyancy in water, the calculation method of the return critical time from the target position to the critical position is different from the calculation method of the first prediction time. How to calculate it specifically will be elaborated later. When approaching the return critical time, adjust the configuration parameters of the UAV, including calibrating the GPS and IMU to ensure the accuracy of sensor data, and adjusting the angles of the elevator and aileron to ensure the flight stability of the UAV in the air.

[0033] In the embodiment of the present application, by establishing a neural network model to predict the time when the UAV reaches the critical position, when the difference between the predicted time and the actual time is less than the set threshold, adjust the configuration parameters of the UAV to ensure the safety and stability of the UAV during the cross-media process; the adaptive adjustment mechanism can dynamically adjust the configuration parameters of the UAV according to real-time data, enabling the UAV to better adapt to different flight environments and mission requirements.

[0034] By establishing a weighted topological map of the second area and determining the optimal path from the critical position to the target position, the flight time and energy consumption can be reduced. By setting multiple first positions on the optimal path, the complex path can be decomposed into multiple small segments, and each small segment can be precisely controlled and adjusted, thereby improving the safety and reliability of the path; through the state control model, an accurate state adjustment amount can be generated, enabling the UAV to accurately move from the current position to the adjacent first position, ensuring the accuracy of path tracking, and at the same time solving the problem of weak GPS signals in the underwater environment and inability to navigate; when the UAV returns to the first area, by predicting the time to reach the critical position, the configuration parameters can be adjusted in advance; it can ensure the stable flight state of the UAV during the return process, reduce the return failure caused by navigation errors or environmental changes, and improve the safety of the return.

[0035] In a specific embodiment, the steps for obtaining the first prediction time when the UAV reaches the critical positions of the first area and the second area are as follows: Obtain the real-time flight data of the drone in the first area, establish a time prediction model, input the real-time flight data into the time prediction model. The time prediction model is a neural network model, and train the time prediction model based on the historical flight data of the drone in the first area. The historical flight data includes the flight path of the drone from the first area to the critical position of the second area, historical wind data, and flight time. Each flight path consists of multiple position points, and each position point has a corresponding flight time.

[0036] Specifically, collect the historical flight data of the drone during multiple flights from the first area (such as in the air) to the second area (such as underwater), including the flight path, the time to reach each position point, the time point to reach the critical position, and wind data. Use the historical flight data as the training set of the neural network model. The neural network model extracts features from the historical flight data, such as the coordinates of the position points, time intervals, wind speed and direction, etc., and learns the relationship between the wind data and the time to reach the critical position. During the actual flight of the drone, obtain the current flight data of the drone, including position points, wind data, and the position information of the critical position, etc., and input the current flight data into the trained neural network model. The model outputs the first predicted time for the drone to reach the critical position.

[0037] In a specific embodiment, determining the optimal path of the drone from the critical position to the target position specifically includes the following steps: Create a three-dimensional environmental map of the second area, set multiple navigation positions in the three-dimensional environmental map, establish a weighted topological map of the second area with each navigation position as a node, calculate the similarity between the reference image corresponding to each navigation position and the reference images of adjacent navigation positions, set the weights between adjacent navigation positions in the weighted topological map based on the similarity, and define the node sequence with the minimum sum of edge weights from the starting node to the ending node in the weighted topological map as the optimal path of the drone. The starting node and the ending node are the critical position and the target position respectively.

[0038] Specifically, sensors such as sonar sensors, lidar (LiDAR), or vision systems are used to scan, for example, an underwater environment to create an environmental map. Multiple key points are selected on the environmental map as navigation positions. The navigation positions are landmarks that are easy for the drone to identify and locate, such as rocks, fixed weeds, sunken ships, etc. Reference images are captured in advance at each navigation position. A weighted topological map is established with the navigation positions as nodes. The similarity between the current image of the drone's current position and the reference image of the navigation position is calculated according to methods such as cosine distance. The higher the similarity, the more common features or attributes there are between the two nodes (such as images, locations, concepts, etc.). Here, the similarity between the two images can be set as the weight value of the corresponding edge. The sum of the edge weights of each path formed from the critical position to the target position is calculated according to the edge weight value. The smallest node sequence is defined as the optimal path of the drone.

[0039] As Figure 3 shown, it is the weighted topological map of the second region. There are 8 paths from the critical position to the target position, which are 0->1->3->5->7, 0->1->3->6->7, 0->1->2->3->5->7, 0->1->2->4->6->7, 0->1->2->3->6->7, 0->2->3->5->7, 0->2->3->6->7, 0->2->4->6->7. The sums of the edge weights are: 2.3, 1.9, 2.9, 3.1, 2.5, 2.1, 1.7, 2.3 respectively. Then 0->2->3->6->7 is selected as the optimal path.

[0040] In a specific embodiment, generating an estimated value of the state adjustment amount of the drone specifically includes the following steps: The state control model is a neural network model. The neural network model extracts the feature quantities in the current image and the reference image, obtains the auxiliary data of the current image and the reference image at the first position. The auxiliary data includes the image shooting position, the acceleration and angular velocity of the drone. The state difference between the current position of the drone and the adjacent first position is calculated based on the auxiliary data. The state difference includes the position difference, the direction difference, and the distance difference. The state difference is labeled to obtain the labeled data.

[0041] The feature quantity and the labeled data are used as the training data of the state control model. The state control model predicts and generates the state difference between the current position and the adjacent first position. The state difference is defined as the estimated value of each state adjustment amount of the drone.

[0042] Specifically, a convolutional neural network is used to extract feature quantities from the current image and the reference image. In the auxiliary data, the data of multiple sonar sensors are combined, and triangulation or other positioning algorithms are used to estimate the image capture position of the drone. The acceleration and angular velocity of the drone are measured by the accelerometer and gyroscope in the inertial navigation system. By integrating the acceleration and angular velocity, the displacement and attitude (direction) changes of the drone from the current position to the adjacent first position are calculated.

[0043] Methods such as Kalman Filter are used to filter the data to remove noise and inconsistencies. The state difference between the current position and the first position of the drone is labeled to obtain labeled data. The extracted feature quantities and the labeled data are used as training data to train the neural network model. During the training process, the model learns how to predict the adjustment amount of the drone from the current position to the adjacent first position in actual situations based on the feature quantities of the current image and the reference image and the differences of various state feature quantities, so as to achieve more automated and accurate flight control.

[0044] In a specific embodiment, obtaining the return critical time when the drone reaches the critical position specifically includes the following steps: Obtain the target image at the target position and extract the feature information, compare it with the standard features of the target object in the database to lock the target object. Define the time point when the drone reaches the second position as the starting time point. The second position is a position at a first value away from the critical position. Starting from the starting time point, take pictures of the second area every target time period to obtain area images. Obtain the detection range of the visual camera within each target time period, and extract the feature information from the area images. Match the feature information within each target time period with the first feature in the database. The first feature is the feature information obtained from the area image of the critical position. Match the detection range with the standard detection range. If both the feature information and the detection range match successfully, then define the current target time period as the return critical time.

[0045] Specifically, assume that the target object is a mine ambushed by the enemy in the sea. The UAV uses the onboard visual camera 7 to capture the target image. The feature information extracted from the target image is compared and analyzed with the standard features of the mine stored in the database to identify and lock the mine. Define the time point when the UAV reaches the second position as the starting time point. The second position is a position with a depth value of the first numerical value from the critical position. The depth value can be, for example, 1 m. The depth sensor on the UAV can be used to measure the depth of the current position. When the difference between the current position depth and the critical position depth is small, it indicates that the UAV may be approaching the critical position. Obtaining the environmental image of the underwater area starting from the second position can avoid unnecessary image acquisition when far from the target area and reduce the pressure of data processing and storage. As the UAV gets closer and closer to the critical position, the area covered by the UAV's perspective becomes smaller, and the focal length effect will make the field of view narrower, so the detection range also becomes smaller and smaller.

[0046] Match the feature information in each target time period with the first feature in the database to determine whether the UAV can capture an image of the critical position. At the same time, match the current detection range with the standard detection range to further verify the position between the UAV and the critical position. Through double matching, the position of the UAV can be verified from multiple dimensions, improving the accuracy of positioning. Define the target time period with successful matching as the return critical time. At this time, the UAV has not reached the critical position yet, but it is already relatively close. Therefore, adjust the configuration data of the UAV at this time to adapt to the flight requirements in the air in advance.

[0047] In a specific embodiment, adjusting the configuration parameters of the UAV based on the adaptive adjustment mechanism specifically includes the following steps: Based on different dynamic models of the UAV in the first area and the second area, calculate the difference value of the physical parameters of the UAV in the two areas. The physical parameters include the central position, moment of inertia, and resistance information of the UAV. Calculate the compensation parameters of the UAV based on the difference value of the physical parameters. Adjust the control signal of the UAV based on the compensation parameters. Adjust the configuration parameters of the UAV based on the control signal. The configuration parameters include flight parameters and sensor parameters.

[0048] Specifically, in order to enable the UAV to achieve stable and efficient flight in different regions, it is necessary to adaptively adjust the configuration parameters of the UAV according to the dynamic models of each region. Based on the physical characteristics of the UAV and environmental conditions, a suitable dynamic model is established. For example, an aerodynamic model is established in the air region, and a hydrodynamic model is established in the water region. The dynamic model is used to guide how to measure and estimate physical parameters. For example, the drag coefficient of the UAV in the air is obtained through wind tunnel tests or CFD simulations, and the water drag coefficient is obtained through pool tests or hydrodynamic calculations. Compare the physical parameters of the UAV in the first region and the second region, including the center position, moment of inertia, and drag information. By calculating the difference value of the physical parameters, the change of the physical characteristics of the UAV in different regions is quantified. Based on the difference value of the physical parameters, a compensation algorithm (such as a PID controller, a fuzzy logic controller, or an adaptive control algorithm) is used to calculate the compensation parameters.

[0049] Based on the compensation parameters, a new control signal (such as a thrust, rudder control signal, etc.) is generated, and the compensation control signal is fused with the original control signal to obtain the final control signal. Through the control system of the UAV, the configuration parameters are updated in real time, enabling the UAV to move according to the new parameters.

[0050] The cross-media control method based on the UAV vision system in the embodiments of the present application has been described above. Next, the cross-media control system based on the UAV vision system in the embodiments of the present application will be described. Please refer to Figure 4 , an embodiment of a cross-media control system based on the UAV vision system in the embodiments of the present application includes: A departure control module, configured to set the departure position of the UAV in the first region, start timing from the departure of the UAV, obtain the first predicted time when the UAV reaches the critical position between the first region and the second region, and the current time of the current position of the UAV during flight. If the difference between the first predicted time and the current time is less than the first threshold, the configuration parameters of the UAV are adjusted based on the adaptive adjustment mechanism.

[0051] An image correction module, configured to, after reaching the second region, take the critical position as the starting point of the current position, obtain the current image of the UAV at the current position, and perform calibration processing on the current image.

[0052] A path formation module, configured to determine the optimal path of the UAV from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, and input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position into the state control model to generate an estimated value of the state adjustment amount of the UAV.

[0053] A state adjustment module is used for the drone to move from the current position to an adjacent first position based on a state adjustment amount, take the adjacent first position as the new current position, and repeat this step until the optimal path is traversed.

[0054] A return journey control module, when the drone returns from the second area to the first area, obtains the return critical time when the drone reaches the critical position, and adjusts the configuration parameters of the drone based on the return critical time.

[0055] This application also provides a computer-readable storage medium. The computer-readable storage medium can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer, the computer is made to execute the steps of the cross-media control method based on the drone vision system.

[0056] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described system, system, and unit can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0057] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of this application, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0058] The above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application.

Claims

1. A cross-media control method based on a UAV vision system, characterized in that: The method comprises: Step S1: setting a departure position of the UAV in the first area, starting timing from the departure of the UAV, obtaining a first predicted time for the UAV to reach a critical position between the first area and the second area, and a current time of the current position of the UAV during flight, and if the difference between the first predicted time and the current time is less than a first threshold, adjusting the configuration parameters of the UAV based on an adaptive adjustment mechanism; Step S2: after arriving at the second area, taking the critical position as the starting point of the current position, obtaining a current image of the drone at the current position, and performing calibration processing on the current image; Step S3: determining an optimal path of the UAV from the critical position to the target position, setting a plurality of first positions on the optimal path, establishing a state control model, inputting a calibrated current image and a reference image corresponding to the first position adjacent to the critical position into the state control model, and generating an estimated value of the state adjustment amount of the UAV; Step S4: the drone moves from the current position to an adjacent first position based on the state adjustment amount, takes the adjacent first position as a new current position, and repeats this step until the optimal path is traversed; Step S5: When the drone returns from the second area to the first area, a return critical time for the drone to reach the critical position is obtained, and configuration parameters of the drone are adjusted based on the return critical time.

2. The method according to claim 1, characterized in that Obtaining a first predicted time for the drone to arrive at a critical position between the first area and the second area includes: Real-time flight data of the UAV in the first area is obtained, a time prediction model is established, and the real-time flight data is input into the time prediction model, where the time prediction model is a neural network model. The time prediction model is trained based on historical flight data of the UAV in the first area, where the historical flight data includes a flight path of the UAV from the first area to a critical position of the second area, historical wind data, and flight time, where each flight path consists of multiple position points, and each position point has a corresponding flight time.

3. The method according to claim 1, characterized in that Determining an optimal path for the UAV from the critical position to the target position includes: A three-dimensional environment map of the second area is created, a plurality of navigation positions are set in the three-dimensional environment map, a weighted topological map of the second area is established with each navigation position as a node, similarity between a reference image corresponding to each navigation position and a reference image of an adjacent navigation position is calculated, weights between adjacent navigation positions in the weighted topological map are set based on the similarity, a node sequence with the smallest sum of edge weights from a start node to an end node in the weighted topological map is defined as an optimal path for the UAV, and the start node and the end node are respectively the critical position and the target position.

4. The method according to claim 3, characterized in that Generating an estimate of the state adjustment of the drone, including: The state control model is a neural network model, the neural network model extracts feature quantities from the current image and the reference image, obtains auxiliary data of the current image and the reference image at a first position, the auxiliary data includes an image shooting position, an acceleration and an angular velocity of the drone, calculates a state difference of the drone from the current position to an adjacent first position based on the auxiliary data, the state difference includes a position difference, a direction difference and a distance difference, and annotates the state difference to obtain annotated data; The feature quantity and the labeled data are used as training data for the state control model. The state control model predicts and generates a state difference from the current position to an adjacent first position. The state difference is defined as an estimated value of each state adjustment amount of the UAV.

5. The method according to claim 1, characterized in that Obtaining the critical return time of the UAV to the critical position, including: A target image at the target position is acquired and feature information is extracted, and compared with the standard features of the target object in a database, the target object is locked, and the time point when the drone arrives at the second position is defined as the starting time point, the second position is a position that is a first value away from the critical position, and the second area is photographed every target time period from the starting time point to acquire a regional image, the detection range of the visual camera in each target time period is acquired, and feature information is extracted from the regional image, and the feature information in each target time period is matched with the first feature in the database, the first feature is the feature information obtained from the regional image of the critical position, and the detection range is matched with the standard detection range. If the feature information and the detection range are matched successfully, the current target time period is defined as the return critical time.

6. The method according to claim 1, characterized in that Adjusting the configuration parameters of the drone based on the adaptive adjustment mechanism includes: Based on the different dynamic models of the UAV in the first area and the second area, the difference values ​​of the physical parameters of the UAV in the two areas are calculated, and the physical parameters include the center position, moment of inertia and resistance information of the UAV. The compensation parameters of the UAV are obtained based on the difference values ​​of the physical parameters. The control signal of the UAV is adjusted based on the compensation parameters. The configuration parameters of the UAV are adjusted based on the control signal, and the configuration parameters include flight parameters and sensor parameters.

7. A cross-media control system based on a drone vision system, used to implement the cross-media control method based on a drone vision system as described in any one of claims 1 to 6, characterized in that: The system comprises: a departure control module, used to set the departure position of the drone in the first area, start timing from the departure of the drone, obtain a first predicted time for the drone to reach the critical position of the first area and the second area, and a current time of the current position of the drone during flight, and if the difference between the first predicted time and the current time is less than a first threshold, adjust the configuration parameters of the drone based on an adaptive adjustment mechanism; An image correction module, configured to, after arriving at the second area, take the critical position as the starting point of the current position, obtain a current image of the drone at the current position, and perform calibration processing on the current image; a path forming module, used to determine an optimal path of the UAV from the critical position to the target position, set a plurality of first positions on the optimal path, establish a state control model, input a calibrated current image and a reference image corresponding to the first position adjacent to the critical position into the state control model, and generate an estimated value of the state adjustment amount of the UAV; A state adjustment module, configured to cause the drone to move from the current position to an adjacent first position based on the state adjustment amount, taking the adjacent first position as a new current position, and repeating this step until the optimal path is traversed; The return control module obtains a return critical time for the drone to reach the critical position when the drone returns from the second area to the first area, and adjusts the configuration parameters of the drone based on the return critical time.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the cross-media control method based on the drone vision system as described in any one of claims 1 to 6 is implemented.

Images