Cross-medium control method and system based on unmanned aerial vehicle vision system and storage medium

Abstract

The application relates to the field of unmanned aerial vehicle control, and discloses a cross-medium control method and system based on an unmanned aerial vehicle vision system and a storage medium. The method comprises the following steps: acquiring a first predicted time at which an unmanned aerial vehicle reaches a critical position between a first region and a second region and a current time at a current position in the first region; if a time difference is less than a first threshold, adjusting configuration parameters of the unmanned aerial vehicle; determining an optimal path of the unmanned aerial vehicle from the critical position to a target position in the second region, and setting a plurality of first positions on the optimal path; establishing a state control model, inputting a current image of the current position and a reference image of an adjacent first position, and outputting an estimated value of an unmanned aerial vehicle state adjustment amount; taking the adjacent first position as a new current position, repeating the step until the optimal path is traversed; when the unmanned aerial vehicle returns to the first region, obtaining a return critical time of reaching the critical position, and adjusting the configuration parameters, so that the stability of the unmanned aerial vehicle in different media is improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control, and in particular to a cross-media control method, system, and storage medium based on a UAV vision system. Background Technology

[0002] With the rapid development of drone technology, drones are being used more and more widely in various complex environments, including air, underwater, and land. However, drones face many challenges when crossing media (such as moving from air to underwater). The stability of drones is affected when crossing media. In underwater environments, GPS signals are often very weak or completely unavailable, which renders traditional GPS-based positioning methods ineffective.

[0003] Existing underwater path planning methods, such as Chinese patent application CN113342008A, disclose a path planning system and method for sea-air collaborative underwater target tracking. This method includes: an airborne monitoring equipment cluster acquiring target location information; performing a first path planning along the navigation channel of surface monitoring equipment to construct a surface navigation area map; and transmitting the surface navigation area map to the seaborne monitoring equipment cluster; the seaborne monitoring equipment cluster, based on the surface navigation area map and its own location information, performing a second path planning along the navigation channel of surface monitoring equipment to reach the vicinity of the target, and detecting the underwater environment of the vicinity to construct an underwater obstacle environment map, which is then transmitted to the underwater monitoring equipment cluster; and the underwater monitoring equipment cluster, based on the underwater obstacle environment map, performing a third path planning to track the target's location. This invention employs cluster collaborative optimization, reducing the number of iterations, improving optimization efficiency, rapidly tracking target locations, and enhancing autonomous collaborative tracking capabilities. However, the aforementioned information requires multi-level transmission from the air to the surface and then to the underwater environment, which may lead to delays, poor real-time performance, and potentially affect tracking efficiency and accuracy.

[0004] For example, existing cross-medium control methods, such as Chinese patent application CN115509246A, disclose a longitudinal attitude control method for cross-medium takeoff of an amphibious unmanned aerial vehicle (UAV) with floats. This method includes the following steps: S1, dividing the UAV's control system into three parts: a power system, a controller, and a longitudinal motion dynamics model of the UAV; S2, the power system uses a virtual force N to change the UAV's forward speed and changes its flight attitude via elevators; S3, the controller controls the longitudinal attitude during takeoff; S4, the dynamics model adjusts the UAV's longitudinal attitude based on the forces and torques acting on the UAV according to the output of the power system. The aforementioned prior art mainly utilizes a physics model to achieve attitude control of the UAV across media. However, this method requires precise parameter settings; otherwise, severe attitude fluctuations will occur, affecting flight stability.

[0005] Therefore, a cross-medium control method based on UAV vision system is needed to ensure the stability of UAV when operating across media, while improving the efficiency of UAV underwater. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a cross-media control method, system, and storage medium based on a UAV vision system, which can improve the stability of UAVs when operating across media.

[0007] In a first aspect, this application provides a cross-media control method based on an unmanned aerial vehicle (UAV) vision system, the method comprising:

[0008] Step S1: Set the starting position of the UAV in the first area, start timing from the start of the UAV, obtain the first predicted time when the UAV reaches the boundary position between the first area and the second area, and the current time of the UAV's current position during flight. If the difference between the first predicted time and the current time is less than a first threshold, adjust the configuration parameters of the UAV based on the adaptive adjustment mechanism.

[0009] Step S2: After reaching the second region, take the critical position as the starting point of the current position, acquire the current image of the UAV at the current position, and perform calibration processing on the current image;

[0010] Step S3: Determine the optimal path for the UAV from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position into the state control model, and generate an estimated value of the state adjustment amount of the UAV.

[0011] Step S4: The UAV moves from the current position to the adjacent first position based on the state adjustment amount, and takes the adjacent first position as the new current position. This step is repeated until the optimal path is traversed.

[0012] Step S5: When the UAV returns from the second area to the first area, obtain the return critical time when the UAV reaches the critical position, and adjust the configuration parameters of the UAV based on the return critical time.

[0013] In conjunction with the first aspect, in the first implementation of the first aspect of this application, obtaining the first predicted time for the drone to reach the critical position between the first and second regions includes:

[0014] The real-time flight data of the UAV in the first area is acquired, and a time prediction model is established. The real-time flight data is input into the time prediction model, which is a neural network model. The time prediction model is trained based on the historical flight data of the UAV in the first area. The historical flight data includes the flight path of the UAV from the first area to the critical position of the second area, historical wind data, and flight time. Each flight path consists of multiple location points, and each location point has a corresponding flight time.

[0015] In conjunction with the first aspect, in the second implementation of the first aspect of this application, determining the optimal path for the UAV from the critical position to the target position includes:

[0016] A three-dimensional environment map of the second region is created, and multiple navigation positions are set in the three-dimensional environment map. A weighted topology map of the second region is built with each navigation position as a node. The similarity between the reference image corresponding to each navigation position and the reference images of adjacent navigation positions is calculated. The weight between adjacent navigation positions is set based on the similarity. The node sequence with the smallest sum of edge weights from the starting node to the ending node in the weighted topology map is defined as the optimal path of the UAV. The starting node and the ending node are the critical position and the target position, respectively.

[0017] In conjunction with the first aspect, in the third implementation of the first aspect of this application, generating an estimate of the state adjustment amount of the UAV includes:

[0018] The state control model is a neural network model. The neural network model extracts feature quantities from the current image and the reference image, obtains auxiliary data of the current image and the reference image at the first position, the auxiliary data includes the image shooting position, the acceleration and angular velocity of the UAV, calculates the state difference of the UAV from the current position to the adjacent first position based on the auxiliary data, the state difference includes position difference, direction difference and distance difference, and labels the state difference to obtain labeled data;

[0019] The feature quantity and the labeled data are used as training data for the state control model. The state control model predicts and generates the state difference between the current position and the adjacent first position. The state difference is defined as the estimated value of each state adjustment of the UAV.

[0020] In conjunction with the first aspect, in the fourth implementation of the first aspect of this application, obtaining the return critical time for the drone to reach the critical position includes:

[0021] The system acquires a target image at the target location and extracts feature information. This image is then compared with standard features of target objects in the database to locate the target object. The time point at which the drone arrives at the second location is defined as the starting time point. The second location is a position a first value away from the critical location. Starting from the starting time point, images of the second region are captured at target time intervals to acquire region images. The detection range of the visual camera within each target time interval is obtained, and feature information is extracted from the region images. The feature information within each target time interval is matched with a first feature in the database. The first feature is the feature information obtained from the region image at the critical location. The detection range is matched with a standard detection range. If both the feature information and the detection range match successfully, the current target time interval is defined as the return critical time.

[0022] In conjunction with the first aspect, in the fifth implementation of the first aspect of this application, the configuration parameters of the UAV are adjusted based on an adaptive adjustment mechanism, including:

[0023] Based on the different dynamic models of the UAV in the first region and the second region, the difference values ​​of the physical parameters of the UAV in the two regions are calculated. The physical parameters include the center position, moment of inertia and drag information of the UAV. The compensation parameters of the UAV are calculated based on the difference values ​​of the physical parameters. The control signal of the UAV is adjusted based on the compensation parameters. The configuration parameters of the UAV are adjusted based on the control signal. The configuration parameters include flight parameters and sensor parameters.

[0024] Secondly, this application provides a cross-media control system based on an unmanned aerial vehicle (UAV) vision system, the system comprising:

[0025] The departure control module is used to set the departure position of the UAV in the first area, start timing from the departure of the UAV, obtain the first predicted time when the UAV reaches the boundary position between the first area and the second area, and the current time of the UAV's current position during flight. If the difference between the first predicted time and the current time is less than a first threshold, the configuration parameters of the UAV are adjusted based on an adaptive adjustment mechanism.

[0026] The image correction module is used to, after reaching the second region, take the critical position as the starting point of the current position, acquire the current image of the UAV at the current position, and perform calibration processing on the current image;

[0027] The path formation module is used to determine the optimal path for the UAV from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position into the state control model, and generate an estimated value of the state adjustment amount of the UAV.

[0028] The state adjustment module is used to move the UAV from the current position to an adjacent first position based on the state adjustment amount, take the adjacent first position as the new current position, and repeat this step until the optimal path is traversed.

[0029] The return control module obtains the return critical time when the drone returns from the second area to the first area, and adjusts the configuration parameters of the drone based on the return critical time.

[0030] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned cross-media control method based on a UAV vision system.

[0031] The technical solution provided in this application predicts the time for the UAV to reach the critical position by establishing a neural network model. When the difference between the predicted time and the actual time is less than a set threshold, the configuration parameters of the UAV are adjusted to ensure the safety and stability of the UAV during the cross-media process. The adaptive adjustment mechanism can dynamically adjust the configuration parameters of the UAV according to real-time data, so that the UAV can better adapt to different flight environments and mission requirements.

[0032] By establishing a weighted topology map of the second region and determining the optimal path from the critical position to the target position, flight time and energy consumption can be reduced. By setting multiple first positions on the optimal path, the complex path can be decomposed into multiple segments, each of which can be precisely controlled and adjusted, thereby improving the safety and reliability of the path. Through the state control model, precise state adjustment quantities can be generated, enabling the UAV to accurately move from the current position to the adjacent first position, ensuring the accuracy of path tracking, and also solving the problem of weak GPS signals and navigation failure in underwater environments. When the UAV returns to the first region, by predicting the time to reach the critical position, configuration parameters can be adjusted in advance, ensuring the stability of the UAV's flight state during the return process, reducing return failures caused by navigation errors or environmental changes, and improving the safety of the return. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of one embodiment of the cross-media control method based on an unmanned aerial vehicle (UAV) vision system in this application.

[0035] Figure 2 This is a structural diagram of the UAV in the embodiments of this application;

[0036] Figure 3 This is a weighted topology map of the second region in the embodiments of this application;

[0037] Figure 4 This is a schematic diagram of an embodiment of a cross-media control system based on an unmanned aerial vehicle (UAV) vision system in this application.

[0038] In the diagram: 1. Propeller; 2. Motor; 3. Visual sensor; 4. Ground station signal receiver; 5. Flight control board; 6. ESC; 7. Visual camera; 8. UAV support. Detailed Implementation

[0039] This application provides a cross-media control method, system, and storage medium based on a UAV vision system. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0040] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the cross-media control method based on an unmanned aerial vehicle (UAV) vision system in this application includes:

[0041] Step S1: Set the starting position of the UAV in the first area, start timing from the start of the UAV's departure, obtain the first predicted time when the UAV reaches the boundary position between the first and second areas, and the current time of the UAV's current position during flight. If the difference between the first predicted time and the current time is less than the first threshold, adjust the configuration parameters of the UAV based on the adaptive adjustment mechanism.

[0042] Specifically, such as Figure 2 The diagram shows the structure of a drone, including: propeller 1, motor 2, vision sensor 3, ground station signal receiver 4, flight control board 5, electronic speed controller (ESC) 6, vision camera 7, and drone support 8. Propeller 1 provides the lift and thrust required for drone flight; motor 2 is the main power source for the drone, and its motion control can be achieved by adjusting its speed; vision sensor 3 is used to capture environmental images and videos, supporting the drone's visual navigation, target detection, and obstacle avoidance functions; ground station signal receiver 4 receives command signals from the ground control station, enabling remote control and data transmission; flight control board 5 is the core control unit of the drone, responsible for attitude stabilization, navigation control, and data processing; ESC 6 is an electronic speed controller that controls the speed and direction of motor 2, adjusting the power output of propeller 1 according to the commands of flight control board 5 to achieve stable flight; drone support 7 provides structural support and stability for the drone; drone support 8 is used for real-time image or video capture, supporting the drone in monitoring, mapping, or target identification, and can be applied in various scenarios such as mapping and search and rescue.

[0043] The UAV's vision system consists of a vision camera 7 and a flight control board 5. First, the vision camera 7 acquires images of the UAV's surrounding environment, identifies environmental features within the images, and sends the acquired image data to the flight control board 5. Upon receiving the image data, the flight control board 5 automatically analyzes the image, quickly identifying the type and characteristics of target objects by comparing the image data with information in its database. Furthermore, the flight control board 5 uses image processing algorithms to calculate the target's precise location and distance, thus providing the UAV with accurate position information.

[0044] Suppose that the UAV receives a control command from a remote location via ground station signal receiver 4, requiring the UAV to depart from the first area (air area), pass through a critical position to enter the second area (water area), search for target objects in the water area, confirm the target object's location, and return to the air area. The target object may be an enemy mine, underwater spy device, etc. When the UAV needs to transition from air to water, its dynamic characteristics, drag, buoyancy, etc. will change. In order to ensure that the UAV can safely and stably transition from the air area to the water area, this invention uses a neural network model to predict the first predicted time for the UAV to reach the critical position of the first and second areas. The timing starts from the start of the UAV, and the difference between the first predicted time and the current time is calculated periodically (every 3 seconds). If the difference is less than a first threshold, the first threshold (e.g., 5 seconds) is set. That is, when the UAV is close to the critical position, the configuration parameters can be adjusted in advance by the ESC 6 to ensure a smooth transition from air to water, improving the stability and reliability of cross-medium missions.

[0045] Step S2: After reaching the second region, take the critical position as the starting point of the current position, acquire the current image of the UAV at the current position, and perform calibration processing on the current image.

[0046] Specifically, when a drone enters an underwater area from an air area, the underwater environment may introduce additional noise into the image, such as particles or bubbles. Moreover, water absorbs and scatters light, causing color distortion. Therefore, it is necessary to calibrate the current image captured by the drone using a vision camera, for example, by using an automatic white balance algorithm to correct the color distortion caused by water absorbing different wavelengths of light.

[0047] Step S3: Determine the optimal path for the UAV from the critical position to the target position, set multiple first positions on the optimal path, establish a state control model, input the calibrated current image and the reference image corresponding to the first position adjacent to the critical position into the state control model, and generate an estimate of the UAV's state adjustment amount.

[0048] Specifically, multiple representative navigation locations are pre-set based on the underwater environment map. These can be specific underwater landmarks, markers, or known navigation points. At each navigation location, a reference image is captured using a visual camera 7, and the reference image and its corresponding navigation location information are stored in the UAV's storage device. A weighted topology map from the critical location to the target location is built based on the reference image. The optimal path is selected from the network topology map, and the navigation locations on the optimal path, excluding the target location, are defined as first locations. These first locations will serve as relay points for the UAV's underwater navigation, guiding the UAV's movement in the underwater environment.

[0049] The state control model is a neural network model. This model takes the calibrated current image and the reference image corresponding to the first position adjacent to the critical position in the optimal path as input. The reference image is a standard image stored in the database in advance. The convolutional layer is used to extract the feature vectors of the current image and the reference image. By comparing the feature vectors of the current image and the reference image, the adjustment amount of the current state of the UAV (such as position, direction, speed, etc.) relative to the adjacent state (first position) is estimated.

[0050] Step S4: The UAV moves from its current position to the adjacent first position based on the state adjustment amount, and takes the adjacent first position as the new current position. This step is repeated until the optimal path is traversed.

[0051] Specifically, taking the adjacent first position as the new current position, the current image at the new current position and the reference image at the adjacent first position in the optimal path are used as inputs to the state control model to generate a state adjustment value for the new current position. This method enables the UAV to accurately identify its current position after entering the underwater environment, achieving high-precision positioning. Furthermore, it utilizes a weighted topology map to enable navigation in underwater areas, enhancing the UAV's adaptability to complex environmental changes.

[0052] Step S5: When the UAV returns from the second area to the first area, obtain the return critical time when the UAV reaches the critical position, and adjust the configuration parameters of the UAV based on the return critical time.

[0053] Specifically, when the UAV returns from the second area (water) to the first area (air), due to factors such as buoyancy in the water, the calculation method for the return critical time from the target position to the critical position is different from the calculation method for the first predicted time. The specific calculation method will be explained later. When the return critical time is about to be reached, the configuration parameters of the UAV are adjusted, including calibrating the GPS and IMU to ensure the accuracy of sensor data, and adjusting the angle of the elevator and ailerons to ensure the flight stability of the UAV in the air.

[0054] In this embodiment, a neural network model is established to predict the time it takes for the drone to reach the critical position. When the difference between the predicted time and the actual time is less than a set threshold, the configuration parameters of the drone are adjusted to ensure the safety and stability of the drone during cross-media processes. The adaptive adjustment mechanism can dynamically adjust the configuration parameters of the drone based on real-time data, enabling the drone to better adapt to different flight environments and mission requirements.

[0055] By establishing a weighted topology map of the second region and determining the optimal path from the critical position to the target position, flight time and energy consumption can be reduced. By setting multiple first positions on the optimal path, the complex path can be decomposed into multiple segments, each of which can be precisely controlled and adjusted, thereby improving the safety and reliability of the path. Through the state control model, precise state adjustment quantities can be generated, enabling the UAV to accurately move from the current position to the adjacent first position, ensuring the accuracy of path tracking, and also solving the problem of weak GPS signals and navigation failure in underwater environments. When the UAV returns to the first region, by predicting the time to reach the critical position, configuration parameters can be adjusted in advance, ensuring the stability of the UAV's flight state during the return process, reducing return failures caused by navigation errors or environmental changes, and improving the safety of the return.

[0056] In one specific embodiment, obtaining the first predicted time for the drone to reach the critical position of the first and second regions includes the following steps:

[0057] Real-time flight data of the UAV in the first area is acquired, and a time prediction model is established. The real-time flight data is input into the time prediction model, which is a neural network model. The time prediction model is trained based on the historical flight data of the UAV in the first area. The historical flight data includes the flight path of the UAV from the first area to the critical position of the second area, historical wind data, and flight time. Each flight path consists of multiple location points, and each location point has a corresponding flight time.

[0058] Specifically, historical flight data is collected from multiple drone flights from a first region (e.g., air) to a second region (e.g., underwater). This data includes flight paths, arrival times at each location, arrival times at critical locations, and wind data. This historical flight data is used as the training set for a neural network model. The model extracts features from the historical flight data, such as location coordinates, time intervals, wind strength and direction, and learns the relationship between wind data and arrival times at critical locations. During actual drone flight, current flight data is acquired, including location, wind data, and location information at critical locations. This current flight data is input into the trained neural network model, which outputs the first predicted time for the drone to reach the critical location.

[0059] In one specific embodiment, determining the optimal path for the UAV from the critical position to the target position includes the following steps:

[0060] Create a 3D environment map of the second region, set multiple navigation positions in the 3D environment map, and build a weighted topology map of the second region with each navigation position as a node. Calculate the similarity between the reference image corresponding to each navigation position and the reference images of adjacent navigation positions. Based on the similarity, set the weight between adjacent navigation positions in the weighted topology map. Define the node sequence with the minimum sum of edge weights from the start node to the end node in the weighted topology map as the optimal path of the UAV. The start node and the end node are the critical position and the target position, respectively.

[0061] Specifically, sensors such as sonar sensors, LiDAR, or vision systems are used to scan the environment, for example, underwater, to create an environmental map. Multiple key points are selected on the environmental map as navigation locations. The navigation locations are landmarks that are easy for the UAV to identify and locate, such as rocks, fixed weeds, shipwrecks, etc. Reference images are captured in advance at each navigation location. A weighted topology graph is built with the navigation locations as nodes. The similarity between the current image of the UAV's current location and the reference image of the navigation location is calculated using methods such as cosine distance. The higher the similarity, the more common features or attributes there are between the two nodes (such as images, locations, concepts, etc.). Here, the similarity between the two images can be set as the weight value of the corresponding edge. The sum of the edge weights of each path formed from the critical position to the target position is calculated based on the edge weight value. The minimum node sequence is defined as the optimal path of the UAV.

[0062] like Figure 3 The diagram shows the weighted topology of the second region. There are 8 paths from the critical position to the target position: 0->1->3->5->7, 0->1->3->6->7, 0->1->2->3->5->7, 0->1->2->4->6->7, 0->1->2->3->6->7, 0->2->3->5->7, 0->2->3->6->7, and 0->2->4->6->7. The sum of the edge weights are 2.3, 1.9, 2.9, 3.1, 2.5, 2.1, 1.7, and 2.3, respectively. Therefore, 0->2->3->6->7 is selected as the optimal path.

[0063] In one specific embodiment, generating an estimate of the UAV's state adjustment amount specifically includes the following steps:

[0064] The state control model is a neural network model. The neural network model extracts the feature quantities in the current image and the reference image, and obtains auxiliary data of the current image and the reference image at the first position. The auxiliary data includes the image shooting position, the acceleration and angular velocity of the drone. Based on the auxiliary data, the state difference of the drone from the current position to the adjacent first position is calculated. The state difference includes position difference, direction difference and distance difference. The state difference is labeled to obtain labeled data.

[0065] The feature values ​​and labeled data are used as training data for the state control model. The state control model predicts and generates the state difference between the current position and the first adjacent position. The state difference is defined as the estimated value of each state adjustment of the UAV.

[0066] Specifically, a convolutional neural network is used to extract features from the current image and reference image. In the auxiliary data, data from multiple sonar sensors are combined, and triangulation or other positioning algorithms are used to estimate the image capture position of the UAV. The acceleration and angular velocity of the UAV are measured by the accelerometer and gyroscope in the inertial navigation system. By integrating the acceleration and angular velocity, the displacement and attitude (direction) change of the UAV from the current position to the adjacent first position are calculated.

[0067] Using methods such as Kalman filtering, the data is filtered to remove noise and inconsistencies. Labeled data is obtained by annotating the state difference between the UAV's current position and the first adjacent position. The extracted features and labeled data are used as training data to train the neural network model. During training, the model learns how to predict the adjustment amount of the UAV from the current position to the adjacent first position in actual situation based on the feature quantities of the current image and the reference image, as well as the differences of various state feature quantities, so as to achieve more automated and precise flight control.

[0068] In one specific embodiment, obtaining the return critical time for the drone to reach the critical position specifically includes the following steps:

[0069] The system acquires target images at the target location and extracts feature information. This information is then compared with standard features of target objects in the database to lock onto the target object. The time point when the drone arrives at the second location is defined as the starting time point. The second location is a position a first value away from the critical location. Starting from the starting time point, images of the second region are captured every target time period to acquire region images. The detection range of the visual camera within each target time period is obtained, and feature information is extracted from the region images. The feature information within each target time period is matched with the first feature in the database. The first feature is the feature information obtained from the region image at the critical location. The detection range is matched with the standard detection range. If both the feature information and the detection range match successfully, the current target time period is defined as the return critical time.

[0070] Specifically, assuming the target object is an enemy mine buried in the sea, the UAV uses its onboard visual camera 7 to capture the target image. The feature information extracted from the target image is compared and analyzed with the standard features of mines stored in the database to identify and lock the mine. The time point when the UAV reaches the second position is defined as the starting time point. The second position is the position at which the depth value of the critical position is the first value. For example, the depth value is 1m. The depth of the current position can be measured using the depth sensor on the UAV. When the difference between the depth of the current position and the depth of the critical position is small, it indicates that the UAV may be approaching the critical position. Starting from the second position, environmental images of the underwater area can be acquired, which can avoid unnecessary image acquisition when far away from the target area and reduce the pressure of data processing and storage. As the UAV gets closer and closer to the critical position, the area covered by the UAV's field of view becomes smaller. The focal length effect will narrow the field of view, so the detection range also becomes smaller.

[0071] The feature information within each target time period is matched with the first feature in the database to determine whether the drone can capture images of the critical position. At the same time, the current detection range is matched with the standard detection range to further verify the position between the drone and the critical position. Through dual matching, the position of the drone can be verified from multiple dimensions, improving the accuracy of positioning. The target time period with successful matching is defined as the return critical time. At this time, the drone has not yet reached the critical position, but it is already quite close. Therefore, the configuration data of the drone is adjusted at this time to adapt to the flight requirements in the air in advance.

[0072] In one specific embodiment, adjusting the configuration parameters of the UAV based on the adaptive adjustment mechanism includes the following steps:

[0073] Based on the different dynamic models of the UAV in the first and second regions, the differences in the physical parameters of the UAV in the two regions are calculated. The physical parameters include the center position, moment of inertia and drag information of the UAV. The compensation parameters of the UAV are calculated based on the differences in the physical parameters. The control signals of the UAV are adjusted based on the compensation parameters. The configuration parameters of the UAV are adjusted based on the control signals. The configuration parameters include flight parameters and sensor parameters.

[0074] Specifically, to enable stable and efficient flight of UAVs in different areas, the configuration parameters of the UAV need to be adaptively adjusted based on the dynamic models of each area. A suitable dynamic model is established based on the physical characteristics of the UAV and environmental conditions; for example, an aerodynamic model is established for the air region, and a hydrodynamic model is established for the water region. This dynamic model guides the measurement and estimation of physical parameters, such as obtaining the drag coefficient of the UAV in the air through wind tunnel testing or CFD simulation, and obtaining the water drag coefficient through pool testing or hydrodynamic calculations. The physical parameters of the UAV in the first and second regions are compared, including center position, moment of inertia, and drag information. By calculating the differences in these physical parameters, the changes in the physical characteristics of the UAV in different regions are quantified. Based on these differences, compensation algorithms (such as PID controllers, fuzzy logic controllers, or adaptive control algorithms) are used to calculate compensation parameters.

[0075] Based on the compensation parameters, new control signals (such as thrust and rudder control signals) are generated. The compensation control signals are then fused with the original control signals to obtain the final control signals. Through the UAV's control system, the configuration parameters are updated in real time, enabling the UAV to move according to the new parameters.

[0076] The cross-media control method based on the UAV vision system in the embodiments of this application has been described above. The cross-media control system based on the UAV vision system in the embodiments of this application is described below. Please refer to [link / reference]. Figure 4 One embodiment of a cross-media control system based on an unmanned aerial vehicle (UAV) vision system in this application includes:

[0077] The departure control module is used to set the starting position of the UAV in the first area, start timing from the departure of the UAV, obtain the first predicted time when the UAV reaches the boundary position between the first area and the second area, and the current time of the UAV's current position during flight. If the difference between the first predicted time and the current time is less than a first threshold, the configuration parameters of the UAV are adjusted based on an adaptive adjustment mechanism.

[0078] The image correction module is used to acquire the current image of the UAV at the current location after reaching the second region, with the critical position as the starting point of the current position, and to perform calibration processing on the current image.

[0079] The path formation module is used to determine the optimal path for the UAV from the critical position to the target position. Multiple first positions are set on the optimal path, and a state control model is established. The calibrated current image and the reference image corresponding to the first position adjacent to the critical position are input into the state control model to generate an estimate of the UAV's state adjustment amount.

[0080] The state adjustment module is used to move the UAV from its current position to the adjacent first position based on the state adjustment amount, and take the adjacent first position as the new current position. This step is repeated until the optimal path is traversed.

[0081] The return control module obtains the critical time for the drone to reach the critical position when the drone returns from the second area to the first area, and adjusts the configuration parameters of the drone based on the critical time.

[0082] This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a cross-media control method based on a UAV vision system.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for cross-medium control based on a UAV vision system, characterized in that, The method comprises: Step S1: setting a starting position of a UAV in a first area, starting timing from the UAV, obtaining a first predicted time of the UAV reaching a critical position of the first area and a second area, and a current time of a current position of the UAV in a flight process, and if a difference between the first predicted time and the current time is less than a first threshold, adjusting configuration parameters of the UAV based on an adaptive adjustment mechanism; Step S2: after reaching the second area, taking the critical position as a starting point of the current position, obtaining a current image of the UAV at the current position, and performing calibration processing on the current image; Step S3: determining an optimal path of the UAV from the critical position to a target position, setting a plurality of first positions on the optimal path, establishing a state control model, inputting the calibrated current image and a reference image corresponding to the first position adjacent to the critical position into the state control model, and generating an estimated value of a state adjustment amount of the UAV; Wherein, the state control model is a neural network model, the neural network model extracts feature quantities in the current image and the reference image, obtains auxiliary data of the current image and the reference image at the first position, the auxiliary data includes an image shooting position, an acceleration and an angular velocity of the UAV, calculates a state difference value of the UAV from the current position to the adjacent first position based on the auxiliary data, the state difference value includes a position difference, a direction difference and a distance difference, labels the state difference value to obtain labeled data, and takes the feature quantities and the labeled data as training data of the state control model, the state control model predicts to generate the state difference value from the current position to the adjacent first position, and defines the state difference value as an estimated value of each state adjustment amount of the UAV; Step S4: the UAV moves from the current position to the adjacent first position based on the state adjustment amount, takes the adjacent first position as a new current position, and repeats this step until the optimal path is traversed; Step S5: when the UAV returns from the second area to the first area, a return critical time of the UAV reaching the critical position is obtained, and configuration parameters of the UAV are adjusted based on the return critical time.

2. The method of claim 1, wherein, Obtaining a first predicted time of the UAV reaching a critical position of the first area and a second area comprises: Obtaining real-time flight data of the UAV in the first area, establishing a time prediction model, inputting the real-time flight data into the time prediction model, the time prediction model is a neural network model, training the time prediction model based on historical flight data of the UAV in the first area, the historical flight data includes a flight path of the UAV from the first area to the critical position of the second area, historical wind data and flight time, each flight path is composed of a plurality of position points, and each position point has corresponding flight time.

3. The method of claim 1, wherein, Determining an optimal path of the UAV from the critical position to the target position comprises: create a three-dimensional environment map of the second region, set a plurality of navigation positions in the three-dimensional environment map, establish a weighted topological graph of the second region with each navigation position as a node, calculate the similarity of the reference image corresponding to each navigation position and the reference image of the adjacent navigation position, set the weight between the adjacent navigation positions of the weighted topological graph based on the similarity, define the node sequence with the smallest sum of edge weights from the start node to the end node in the weighted topological graph as the optimal path of the unmanned aerial vehicle, and the start node and the end node are the critical position and the target position respectively.

4. The method of claim 1, wherein, obtain the return critical time of the unmanned aerial vehicle reaching the critical position, including: acquire the target image at the target position and extract feature information, compare with the standard features of the target object in the database, lock the target object, define the time point when the unmanned aerial vehicle reaches the second position as the starting time point, the second position is a position with a first value distance from the critical position, start from the starting time point and take pictures of the second region every target time period to obtain regional images, obtain the detection range of the visual camera in each target time period, and extract feature information from the regional images, match the feature information in each target time period with the first feature in the database, the first feature is the feature information obtained from the regional image of the critical position, the detection range is matched with the standard detection range, if the feature information and the detection range are successfully matched, the current target time period is defined as the return critical time.

5. The method of claim 1, wherein, adjust the configuration parameters of the unmanned aerial vehicle based on the adaptive adjustment mechanism, including: based on the different dynamic models of the unmanned aerial vehicle in the first region and the second region, calculate the difference value of the physical parameters of the unmanned aerial vehicle in the two regions, the physical parameters include the center position, moment of inertia and resistance information of the unmanned aerial vehicle, calculate the compensation parameters of the unmanned aerial vehicle based on the difference value of the physical parameters, adjust the control signal of the unmanned aerial vehicle based on the compensation parameters, adjust the configuration parameters of the unmanned aerial vehicle based on the control signal, the configuration parameters include flight parameters and sensor parameters.

6. An unmanned aerial vehicle vision system based cross-medium control system for implementing the unmanned aerial vehicle vision system based cross-medium control method according to any one of claims 1-5, characterized in that, The system comprises: a departure control module for setting the departure position of the unmanned aerial vehicle in the first region, starting timing from the departure of the unmanned aerial vehicle, acquiring the first predicted time of the unmanned aerial vehicle reaching the critical position of the first region and the second region, and the current time of the current position of the unmanned aerial vehicle in the flight process, if the difference between the first predicted time and the current time is less than a first threshold, adjusting the configuration parameters of the unmanned aerial vehicle based on the adaptive adjustment mechanism; an image correction module for acquiring the current image of the unmanned aerial vehicle at the current position after reaching the second region, and performing calibration processing on the current image. The path forming module is configured to determine an optimal path of the UAV from a critical position to a target position, set a plurality of first positions on the optimal path, establish a state control model, input a reference image corresponding to the first position adjacent to the critical position into the state control model after calibration of a current image, and generate an estimated value of a state adjustment amount of the UAV. The state adjustment module is configured to move the UAV from the current position to an adjacent first position based on the state adjustment amount, take the adjacent first position as a new current position, and repeat the step until the optimal path is traversed. The return control module is configured to obtain a return critical time of the UAV reaching the critical position when the UAV returns from the second region to the first region, and adjust configuration parameters of the UAV based on the return critical time.

7. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: The instructions are executed by the processor to implement the cross-medium control method based on the UAV vision system according to any one of claims 1-5.

Images