Water-air unmanned aerial vehicle monitoring method and system based on acousto-optic fusion and multi-source positioning

Through sound-optical fusion, multi-source positioning technology and environmental adaptive navigation control, the problem of low positioning and operating efficiency of underwater drones in turbid water bodies is solved, and higher perception accuracy and stability are achieved, and water is improved. The execution efficiency of the tasks.

CN119022911APending Publication Date: 2024-11-26GUANGDONG OPEN UNIV (GUANGDONG POLYTECHNIC VOCATIONAL COLLEGE)
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411062976.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing underwater drone technology is difficult to meet the needs of precise positioning and efficient operation in turbid water environments, especially in terms of perception and positioning.

Method used

Using methods and systems based on sound-optical fusion and multi-source positioning, data fusion processing is carried out to improve perception and positioning capabilities by acquiring underwater monitoring data, including sonar data, optical data and multiple positioning data, and using environmental adaptation The navigation control algorithm adjusts the navigation parameters of the drone.

Benefits of technology

It improves the perception ability of the underwater environment and the accuracy of target positioning, improves the stability and task execution efficiency of the drone under different water conditions, provides timely and accurate information support, and enhances the execution efficiency of underwater tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119022911A_ABST
    Figure CN119022911A_ABST
Patent Text Reader

Abstract

The invention discloses a water-air unmanned aerial vehicle monitoring method and system based on acousto-optic fusion and multi-source positioning. The method comprises the following steps: acquiring underwater monitoring data; fusing the sonar data and the optical data to generate acousto-optic fusion data, and calculating a first target prediction position based on the acousto-optic fusion data; fusing the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate positioning fusion data, and calculating a second target prediction position and an unmanned aerial vehicle position based on the positioning fusion data; performing environment adaptive navigation control on the underwater environment data and the water body turbidity degree data to obtain water body change data; and determining a target underwater position of the underwater monitoring target according to the underwater monitoring data, the first target prediction position, the second target prediction position, the unmanned aerial vehicle position and the water body change data. The sensing and positioning precision of the unmanned aerial vehicle is improved, the task execution efficiency of the unmanned aerial vehicle is improved, and the method can be widely applied to the technical field of unmanned aerial vehicle underwater positioning monitoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of underwater positioning and monitoring of unmanned aerial vehicles, and in particular to a method and system for monitoring underwater and airborne unmanned aerial vehicles based on acoustic-optical fusion and multi-source positioning. Background Art

[0002] With the rapid development of drone technology, it has been widely used in military reconnaissance, geographic mapping, disaster monitoring, investigation, underwater monitoring and search and rescue. However, in turbid water environments or monitoring and rescue missions that require precise positioning, current underwater drone technology often fails to meet mission requirements, especially in terms of perception and positioning, as well as operational efficiency.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present application aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the main purpose of the embodiments of the present application is to propose a water-air drone monitoring method and system based on acoustic-optical fusion and multi-source positioning, which can improve the drone system's perception and positioning capabilities of the underwater environment and improve the drone's underwater operation efficiency.

[0005] To achieve the above objectives, one aspect of an embodiment of the present application proposes a water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning, the method comprising the following steps:

[0006] Acquiring underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, raw sonar data, raw optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data;

[0007] Performing acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculating a first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data;

[0008] Performing multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculating the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data;

[0009] Performing environment adaptive navigation control processing on the underwater environment data and the water turbidity data based on an environment adaptive navigation control algorithm to dynamically acquire target water body change data;

[0010] The target underwater position of the underwater monitoring target is determined according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

[0011] To achieve the above purpose, another aspect of the embodiment of the present application proposes a water-air drone monitoring system based on acoustic-optical fusion and multi-source positioning, and the system includes the following modules:

[0012] An underwater monitoring data acquisition module is used to acquire underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data;

[0013] an acoustic-optical data fusion module, configured to perform acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculate a first target predicted position of an underwater monitoring target based on the target acoustic-optical fusion data;

[0014] A positioning data fusion module is used to perform multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculate the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data;

[0015] A navigation control processing module, used for performing environment adaptive navigation control processing on the underwater environment data and the water turbidity degree data based on an environment adaptive navigation control algorithm, so as to dynamically obtain target water body change data;

[0016] The target underwater position determination module is used to determine the target underwater position of the underwater monitoring target based on the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

[0017] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0018] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0019] The embodiments of the present application include at least the following beneficial effects: the present application provides a method and system for monitoring water-air drones based on acoustic-optical fusion and multi-source positioning, the scheme acquires underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; the original sonar data and the original optical data are subjected to acoustic-optical data fusion processing to generate target acoustic-optical fusion data, and the first target predicted position of the underwater monitoring target is calculated based on the target acoustic-optical fusion data; the inertial navigation system data is processed , ultra-short baseline system data and long baseline system data are subjected to multi-source positioning fusion processing to generate target positioning fusion data, and the second target predicted position of the underwater monitoring target and the target UAV position of the water-to-air UAV are calculated based on the target positioning fusion data; based on the environment adaptive navigation control algorithm, the underwater environment data and the water turbidity data are subjected to environment adaptive navigation control processing to dynamically obtain the target water body change data; according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data, the target underwater position of the underwater monitoring target is determined. The embodiment of the present application provides a rich source of information for subsequent data analysis and processing by collecting a variety of underwater monitoring data, which can improve the comprehensiveness and accuracy of data analysis and processing; the system's perception of the underwater environment and the accuracy of target positioning can be improved through the fusion processing of sound and light data; and the positioning data of multiple positioning systems are subjected to multi-source positioning fusion processing, which can improve the positioning accuracy of underwater monitoring targets and the position of the drone itself; by analyzing underwater environmental data and water turbidity data, the navigation parameters of the drone are adjusted in real time using an environmental adaptive navigation control algorithm to cope with complex underwater environments, that is, the control algorithm can adjust the movement path and speed of the drone according to the real-time changes of the water body, thereby dynamically acquiring and adapting to the changes of the target water body, and enhancing the stability and task execution efficiency of the drone under different water conditions; finally, the target underwater position of the underwater monitoring target is determined based on the underwater monitoring data, the predicted position of the first target, the predicted position of the second target, the target drone position, and the target water body change data, so as to provide timely and accurate information support for subsequent monitoring or rescue tasks and improve the efficiency of underwater task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of the steps of the water-air UAV monitoring method based on sound-light fusion and multi-source positioning provided in an embodiment of the present application;

[0021] Figure 2 It is a structural schematic diagram of a water-air drone provided in an embodiment of the present application;

[0022] Figure 3It is a flow chart of an acoustic-optical fusion positioning method provided in an embodiment of the present application;

[0023] Figure 4 It is a flow chart of a target positioning method based on acoustic-optical fusion data provided in an embodiment of the present application;

[0024] Figure 5 It is a structural schematic diagram of a water-air UAV monitoring system based on acoustic-optical fusion and multi-source positioning provided in an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0027] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0028] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] With the rapid development of drone technology, it has been widely used in military reconnaissance, geographic mapping, disaster monitoring, investigation, underwater monitoring and search and rescue. However, in turbid water environments or monitoring and rescue missions that require precise positioning, current underwater drone technology often cannot meet mission requirements, especially in terms of perception and positioning, operating efficiency and communication stability.

[0031] For example, traditional underwater drone monitoring and rescue technology relies on GPS (Global Positioning System) positioning or magnetometer and geomagnetic sensor positioning to carry out monitoring and rescue tasks in turbid water bodies, resulting in the limitation of the single positioning performance of drones in mission environments with magnetic fields or extreme water bodies, resulting in insufficient precise positioning capabilities, functional limitations, limited adaptability, limited autonomy, insufficient flexibility, low redundancy and other problems. In particular, in monitoring and rescue missions that require high depth, high turbidity and high response, the relevant technology is not enough to meet the needs of speed and efficiency. In addition, traditional communication technologies are greatly interfered in underwater environments and have no efficient communication mechanism, which affects the execution safety of tasks and the accurate transmission of data.

[0032] At present, there are still some technical difficulties in underwater navigation, control algorithms, underwater decision-making capabilities, effective communication, sensor performance and data processing of underwater drones. The two main problems of underwater drones with related technologies are high interference in underwater communications and great influence of water turbidity. They rely on wired communications such as optical fiber and low-depth underwater wireless communications, which limits the maneuverability of underwater drones in water and their ability to perform complex tasks. The development of underwater drones is a key area of ​​development in the future. Limited by the level of development of related technologies, the ability to autonomously complete complex monitoring and rescue tasks still needs to be improved.

[0033] In view of this, an embodiment of the present application provides a water-air drone monitoring method and system based on acoustic-optical fusion and multi-source positioning. The scheme obtains underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; the original sonar data and the original optical data are subjected to acoustic-optical data fusion processing to generate target acoustic-optical fusion data, and the first target predicted position of the underwater monitoring target is calculated based on the target acoustic-optical fusion data; the inertial navigation system data, ultra-short baseline system data and long baseline system ...; the first target predicted position of the underwater monitoring target is obtained based on the target acoustic-optical fusion data; the first target predicted position of the underwater monitoring target is obtained based on the target acoustic-optical fusion data; the first target predicted position of the underwater monitoring target is obtained based on the target acoustic-optical fusion data; the first target predicted position of the underwater monitoring target is obtained based on the target acoustic-optical fusion data. The baseline system data and the long baseline system data are subjected to multi-source positioning fusion processing to generate target positioning fusion data, and the second target predicted position of the underwater monitoring target and the target UAV position of the water-to-air UAV are calculated based on the target positioning fusion data; the underwater environment data and the water turbidity data are subjected to environment adaptive navigation control processing based on the target adaptive navigation algorithm to dynamically obtain the target water body change data; the target underwater position of the underwater monitoring target is determined based on the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data. The embodiment of the present application provides a rich source of information for subsequent data analysis and processing by collecting a variety of underwater monitoring data, which can improve the comprehensiveness and accuracy of data analysis and processing; the system's perception of the underwater environment and the accuracy of target positioning can be improved through the fusion processing of sound and light data; and the positioning data of multiple positioning systems are subjected to multi-source positioning fusion processing, which can improve the positioning accuracy of underwater monitoring targets and the position of the drone itself; by analyzing underwater environmental data and water turbidity data, the navigation parameters of the drone are adjusted in real time using an environmental adaptive navigation control algorithm to cope with complex underwater environments, that is, the control algorithm can adjust the movement path and speed of the drone according to the real-time changes of the water body, thereby dynamically acquiring and adapting to the changes of the target water body, and enhancing the stability and task execution efficiency of the drone under different water conditions; finally, the target underwater position of the underwater monitoring target is determined based on the underwater monitoring data, the predicted position of the first target, the predicted position of the second target, the target drone position, and the target water body change data, so as to provide timely and accurate information support for subsequent monitoring or rescue tasks and improve the efficiency of underwater task execution.

[0034] The water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning provided in the embodiment of the present application relates to the technical field of drone underwater positioning monitoring. The water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements the water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning, etc., but is not limited to the above forms.

[0035] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (Personal Computers, personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0036] See also Figure 1 , Figure 1 This is an optional step flow chart of the water-air drone monitoring method based on sound-light fusion and multi-source positioning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0037] Step S101, acquiring underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data;

[0038] See also Figure 2 , Figure 2 is a schematic diagram of the structure of a water-air drone provided in an embodiment of the present application, such as Figure 2 As shown, Figure 2 The structure of a water-air drone provided by an embodiment of the present application is shown, wherein serial number 001 represents a water-air drone, serial number 101 represents a mooring disc, and serial number 102 represents a communication buoy. The communication buoy is a moored communication buoy, and the communication buoy line can be retracted and released through the mooring disc. The mooring disc is fixed to the drone body, and the mooring disc can be controlled by a motor to rotate to achieve the retracting and releasing of the line. The underwater drone communication mechanism of the communication buoy can improve the communication stability of the system. Among them, the water-air drone in the embodiment of the present application can also be called an underwater buoy-type drone or simply an underwater drone.

[0039] Specifically, the water-air drone system architecture in the embodiment of the present application may include six modules: drone body, acoustic-optical fusion sensing module, multi-source positioning system, communication buoy system, edge computing unit, and control center. The components and functions of each module are as follows:

[0040] (1) an unmanned aerial vehicle body equipped with a propulsion device for underwater operation and aerial operation, so that the unmanned aerial vehicle body can be operated in water and in the air;

[0041] (2) Acoustic-optical fusion sensing module, including monitoring cameras, multi-beam sonar, imaging sonar, laser imaging equipment and photoacoustic imaging equipment, used to obtain underwater environmental data and water turbidity data, thereby achieving high-precision perception and positioning of complex underwater environments;

[0042] (3) Multi-source positioning system, including inertial navigation system (INS), ultra-short base line (USBL) and long base line (LBL). Specifically, the method of combining the ultra-short base line positioning system and the long base line positioning system is mainly used, supplemented by the inertial navigation system and visual inertial navigation, to accurately locate the water and air UAV in an environment without GPS signals and obtain positioning data;

[0043] (4) Communication buoy system, including buoy communication device, consisting of hydroacoustic communication buoys on and in the water, transmitting, relaying and receiving antennas, for effective data transmission through hydroacoustic communication and radio communication;

[0044] (5) An edge computing unit (airborne edge computing terminal) is configured on the main body of the water-air UAV and is used to process sensor data (acoustic-optical fusion data, positioning data and environmental data) and execute an environmental adaptive navigation control algorithm to dynamically adjust the navigation parameters and motion path of the water-air UAV; wherein the edge computing unit can calculate the first target predicted position of the monitoring target based on the acoustic-optical fusion data, calculate the second target predicted position of the monitoring target and the current position of the UAV based on the positioning fusion data, adjust the navigation parameters of the water-air UAV through the environmental adaptive navigation control algorithm to cope with complex underwater environments and obtain water body change data, and can also calculate the target underwater position of the underwater monitoring target based on the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

[0045] (6) The control center (remote computing and processing center) is connected to the UAV system through the network to receive sensor data and control instructions, and manage and adjust the water and air UAV monitoring based on acoustic and optical fusion and multi-source positioning according to real-time conditions.

[0046] Through the complementary nature of the above modules, the water-air UAV can obtain monitoring information and working condition data in real time, improve the perception and positioning accuracy, mission execution efficiency and communication stability of the water-air UAV, thereby achieving dynamic adaptation and precise navigation in complex underwater environments, and providing an efficient and reliable solution for target monitoring and rescue in complex underwater environments, thereby optimizing the execution of monitoring and rescue missions.

[0047] Among them, the acoustic and optical fusion sensing module is used to obtain underwater environmental information to obtain acoustic and optical data, and then obtain the monitored target position based on the acquired acoustic and optical data; the ultra-short baseline system and long baseline positioning system of the multi-source positioning framework system use sound waves to obtain the target position, the inertial navigation system mainly obtains the position and posture of the UAV in a stable environment, and the visual inertial navigation is to identify and track feature points or landmarks in the surrounding environment, and use this information to obtain the position and posture of the UAV in dynamic and rapid motion.

[0048] The positioning data also includes sonar positioning data and optical positioning data, and the raw optical data includes raw image data of the underwater monitoring target.

[0049] In some embodiments, step S101 may include: calculating the target underwater distance between the underwater monitoring target and the water-to-air drone based on the original sonar data; calculating the target underwater direction of the underwater monitoring target in the sound wave beam according to the horizontal direction angle between the underwater monitoring target and the sound wave beam of the original sonar data, the vertical direction angle between the underwater monitoring target and the sound wave beam, and the beam weight vector of the sound wave beam; calculating the first three-dimensional coordinate position of the underwater monitoring target according to the target underwater direction and the target underwater distance, and using the first three-dimensional coordinate position as sonar positioning data.

[0050] In some embodiments, step S101 may include: preprocessing the original image data to obtain target image data; performing feature extraction on the target image data to obtain target feature points and target sub-features corresponding to the target feature points; performing feature matching on the target sub-features with several sub-features in the target feature library to obtain candidate sub-features; calculating the similarity between the target sub-features and the candidate sub-features to obtain a similarity result; calculating the second three-dimensional coordinate position of the underwater monitoring target based on the similarity result, the two-dimensional coordinate position of the target feature point, and the parallax of the target feature point, and using the second three-dimensional coordinate position as optical positioning data.

[0051] See also Figure 3 , Figure 3 is a flow chart of an acoustic-optical fusion positioning method provided in an embodiment of the present application, such as Figure 3 As shown, Figure 3 The process of estimating the position of the monitoring target through sonar positioning is demonstrated, and the process of estimating the position of the monitoring target through optical positioning is also demonstrated. The process of estimating the position of the monitoring target through sonar positioning can include four parts: sonar positioning, formation of sonar wave velocity, estimation of the direction of the monitoring target, and estimation of the position of the monitoring target; the process of estimating the position of the monitoring target through optical positioning can include five parts: optical positioning, sub-feature generation, feature matching, stereo depth estimation assisted by the monitoring camera, and estimation of the position of the monitoring target. Specifically, after receiving the monitoring and rescue mission, the underwater drone uses acoustic sensors, optical sensors, monitoring cameras, visual cameras, perception modules and positioning modules (baseline positioning modules, inertial positioning modules, visual positioning modules) to obtain underwater environmental data, water turbidity, target perception and positioning data and other information, and then analyzes and preliminarily processes the acoustic and optical fusion positioning data.

[0052] In the specific implementation, the specific process of estimating the monitoring target position through sonar positioning is as follows:

[0053] (1) Use multi-beam sonar for underwater positioning. Specifically, the propagation speed of sound waves in water is a constant (usually about 1500 m / s underwater), and the distance from the underwater monitoring target to the underwater drone sonar is set to d i, which can be expressed by the round trip time t of the sound wave i The distance d from the underwater monitoring target to the underwater drone sonar is calculated by the sound speed c i , d i The calculation formula is as follows:

[0054]

[0055] Among them, d i represents the distance from the underwater monitoring target to the underwater UAV sonar obtained through the i-th acoustic wave beam; t i It represents the round trip time that the underwater drone receives the sound wave after the i-th sound wave beam sends it.

[0056] (2) Sonar beam formation. Specifically, assume that there are N sonar beams, and the beam weight vector w of each beam i is i It can be expressed as a complex vector. The beam weight vector describes the response of the sound wave in a specific direction. Array signal processing techniques (such as beamforming algorithm, MVDR (Minimum Variance Distortionless Response) algorithm, etc.) are used to generate these beam weight vectors.

[0057] (3) Estimation of the direction of the monitoring target. Specifically, for each beam i, the direction of the target θ = (θ i ,φ i ) can be estimated by the phase and amplitude information of the beam output, and the direction θ of the target can be estimated by the following formula using the beam output signal after beamforming:

[0058]

[0059] Where R represents the autocorrelation matrix of the received beam signal; θ i and φ i w respectively represent the horizontal and vertical angles of the target relative to the underwater drone sound wave beam i, and these angles are used to determine the direction of the target in three-dimensional space; i represents the beam weight vector; H represents the beam weight vector w after beamforming i The transposed form of Represented as w i Conjugate transpose.

[0060] (4) Estimation of the monitoring target position. Specifically, assuming that the underwater drone sonar array is located at the coordinate origin (0,0,0), the three-dimensional position of the target P (x i ,y i ,z i ) can estimate the direction θ and distance d of the target according to the direction from the target to each beam i. iThe calculation results are:

[0061] x i =d i ·sin(θ i )·cos(φ i )

[0062] y i =d i ·sin(θ i )·sin(φ i )

[0063] z i =d i ·cos(θ i )

[0064] Among them, the three-dimensional position of the target P(x i ,y i ,z i ) represents the three-dimensional position information of the target obtained after the i-th acoustic wave beam.

[0065] In the specific implementation, the specific process of estimating the position of the monitoring target through optical positioning is as follows:

[0066] The optical positioning of underwater drones uses monitoring cameras or other optical sensors to obtain images or videos of underwater targets, and uses image processing and computer vision technology to achieve target positioning. The specific formulas and steps of the algorithm involved here are as follows:

[0067] (1) Feature extraction and sub-feature generation of optical positioning. Specifically, feature points or sub-features are extracted from the image preprocessed by the monitoring camera, and feature extraction algorithms including SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and Gaussian difference pyramid are used to extract feature points p n =(x n ,y n ) and the corresponding sub-feature d n Among them, the sub-feature d n Used to describe the local image structure around the nth feature point for subsequent feature matching.

[0068] (2) Image acquisition and preprocessing. Specifically, the monitoring camera carried by the underwater drone acquires images or videos of underwater targets. Due to the particularity and turbidity of the underwater environment, image enhancement algorithms and deblurring algorithms are used to preprocess the images to reduce noise, enhance contrast and clarity, etc.

[0069] (3) Sub-feature generation. Specifically, the image area around each feature point is converted into a feature vector d n , describing its local structure and appearance.

[0070] (4) Feature matching. For each sub-feature d n Find the sub-feature d that is closest to the target feature in the target feature library nj , calculate the Euclidean distance or cosine similarity between the sub-feature and the closest sub-feature. Among them, the Euclidean distance is used to measure the distance between two feature vectors d n and d nj The specific calculation formula is as follows:

[0071]

[0072] Among them, N represents the dimension of the sub-feature; k represents the kth feature dimension, d n [k] and d nj [k] represents the feature vector d n and d nj The value of the kth feature dimension.

[0073] Among them, the similarity measurement calculation formula of cosine similarity is:

[0074]

[0075] After calculating the eigenvector d n and d nj After the similarity between the two points, the monitoring target position can be estimated based on the feature point P n The position and matching results are used to calculate the actual position (x, y, z) of the target using triangulation. Assuming that the visual camera of the underwater drone is located at the origin (0, 0, 0), the feature point p n =(x n ,y n ) is the position on the image plane. The three-dimensional coordinates (x, y, z) of the target underwater can be obtained by the disparity d of multiple feature points. n And their relative position relationship with the camera can be estimated, that is, the visual camera can be used to obtain the matching results, and the disparity d of multiple feature points can be obtained based on the matching results. n And their relative position relationship with the camera. The calculation method for target position estimation by optical positioning method is as follows:

[0076]

[0077]

[0078] z=d n

[0079] Among them, f x and f y Respectively represent the horizontal focal length and vertical focal length of the camera, d n Represents the feature point p n Disparity (depth estimation).

[0080] The stereo vision method for depth estimation is: using the images obtained by the monitoring camera and the visual camera system, by calculating the disparity d n Estimate the depth z of the underwater target. Parallax d n The relationship between the z-depth and the baseline length B of the monitoring camera and the parallax d n The pixel value is calculated, and the depth estimation formula is as follows:

[0081]

[0082] Where B represents the baseline length of the monitoring camera, f x Expressed as the horizontal focal length of the camera.

[0083] It should be noted that the depth z of the target in the underwater three-dimensional coordinates (x, y, z) is based on the z obtained by the visual camera, which is used in application scenarios with low precision requirements; while the depth z obtained in the stereo vision method based on depth estimation is based on the depth z obtained by the visual camera and the monitoring camera, which is used in application scenarios with high precision requirements, thereby improving the robustness and adaptability of the system.

[0084] The multi-beam positioning of sonar is affected by factors such as the multipath effect of sound wave propagation, array structure, noise, and target motion, while optical positioning is affected by factors such as the illumination of the underwater environment and changes in water quality. In the embodiment of the present application, sound and light fusion is first performed to more comprehensively understand the state of the underwater environment and target characteristics, providing a solid foundation for subsequent data processing and decision support, and for the subsequent use of Kalman filters to predict the state at the next moment based on the current measurement value and the dynamic model of the system, and to correct errors, thereby improving the accuracy and stability of positioning, and then using deep learning methods to process measurement errors and uncertainties to improve positioning accuracy and robustness in turbid waters.

[0085] Step S101 shown in the embodiment of the present application can more comprehensively understand the state of the underwater environment and target characteristics by collecting various types of data, thereby providing a solid foundation for subsequent data processing and decision support.

[0086] Step S102, performing an acoustic-optical data fusion process on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculating a first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data;

[0087] In some embodiments, step S102 may include: preprocessing the original sonar data to obtain candidate sonar data; performing information extraction processing on the candidate sonar data to obtain target environment perception information; performing feature extraction processing on the target environment perception information to obtain a target sound feature vector; preprocessing the original optical data to obtain candidate optical data; performing a first feature extraction processing on the candidate optical data to obtain a target visual feature vector; performing a second feature extraction processing on the candidate optical data to obtain a target image key feature vector, and using the target visual feature vector and the target image key feature vector as the target optical feature vector; performing a first fusion processing on the target sound feature vector and the target optical feature vector to obtain a first fused feature vector; performing timestamp synchronization processing on the target sound feature vector and the target optical feature vector in the first fused feature vector, and performing a second fusion processing on the target sound feature vector and the target optical feature vector after the timestamp synchronization processing to obtain a second fused feature vector, and using the second fused feature vector as the target sound and light fusion data; inputting the target sound and light fusion data into the target machine learning model to obtain the first target predicted position of the underwater monitoring target.

[0088] Among them, the edge computing unit (airborne edge computing terminal) configured on the drone body can calculate the first target predicted position of the underwater monitoring target based on the sound and light fusion data.

[0089] See also Figure 4 , Figure 4 is a flow chart of a target positioning method based on acoustic-optical fusion data provided in an embodiment of the present application, such as Figure 4 As shown, the target positioning method based on acoustic and optical fusion data may include data preprocessing, data processing, feature extraction, first fusion processing, second fusion processing, and monitoring target positioning estimation. In the specific implementation, the acquired acoustic and optical data is preprocessed using a Kalman filter, and then the fusion processing of the acoustic and optical data is optimized using a deep learning method to improve the perception and positioning capabilities of the monitoring and rescue system. Specifically, the specific implementation process of the four parts of data preprocessing, feature extraction, data synchronization, and fusion output is as follows:

[0090] (1) First, the data is preprocessed, specifically, noise filtering and signal enhancement are performed on the sonar data and optical data. The specific implementation formula of the preprocessing is:

[0091] y filtered =β(x raw ×H(f)

[0092] Where H(f) represents the transfer function of the selected Kalman filter; β represents the gain control; y filtered represents the enhanced output after filtering; xraw Represents the original data.

[0093] Then, the Kalman filter is used to perform state estimation, specifically estimating the position X and velocity V of the underwater drone, as well as possible measurement deviations.

[0094] Next, error correction is performed, specifically adjusting the estimate of the system state through the residual between the measured value and the predicted value to reduce the positioning error.

[0095] Specifically, it is assumed that the state of the underwater drone can be described by the following linear dynamic model:

[0096] x k =F k x k-1 +B k u k +w k

[0097] Among them, x k represents the state vector (including position and velocity) of the system at time k; F k represents the state transition matrix, describing how the system state changes over time; B k represents the control input matrix, which shows how external control affects the system state; u k represents the control input vector, which can be an external control input (such as thrust); w k represents the process noise, that is, the uncertainty of the system model.

[0098] The measurement model is expressed as:

[0099] z k =H k x k +v k

[0100] Among them, z k represents the measurement vector at time k; H k represents the measurement matrix, which maps the state vector to the measurement space; v k Represents the measurement noise, which is the error in the measurement process.

[0101] The prediction step can be expressed as:

[0102]

[0103]

[0104] in, represents the prior estimate of the state at time k; represents the prior estimation error covariance matrix; Q krepresents the process noise covariance matrix; represents the estimated value of the state vector; P k-1 represents the state estimation error covariance matrix at time k-1.

[0105] The update step can be expressed as:

[0106]

[0107]

[0108]

[0109] in, represents the Kalman gain; K k represents the Kalman gain matrix, which is used to incorporate measurement updates into the state estimate; R k represents the measurement noise covariance matrix; I represents the unit operation matrix, which is used to adjust the uncertainty of the state estimation.

[0110] In the specific implementation, the Kalman filter is used to continuously update the state estimate through the above steps. and the error covariance matrix P k , thereby achieving accurate estimation of the position and speed of underwater drones and precise positioning of monitoring targets. Then, deep learning methods are used to optimize the fusion processing of acoustic and optical data to improve the system's perception ability, especially in turbid water bodies with insufficient light.

[0111] (2) Use Convolutional Neural Network (CNN) to extract features from sonar and optical images. The feature extraction formula is expressed as:

[0112] f(x)=max(0,x×W+b)

[0113] Among them, f(x) represents the output obtained after processing the input x through the neural network layer; x represents the input data; b represents the bias parameter; W represents the weight parameter.

[0114] (3) Next, data synchronization and fusion are performed. Acoustic data and optical data are synchronized using timestamps, and then features are merged through a fusion layer (such as weighted average or neural network). The calculation formula for synchronization fusion is as follows:

[0115] z=σ(α·f sonar +(1-α)·f optical )

[0116] Among them, f sonarIt is represented as the feature vector extracted from acoustic data, and the sonar data features after preprocessing and feature extraction; f optical It represents the feature vector extracted from optical data, which is the optical image feature after preprocessing and feature extraction; α represents the fusion weight; z represents the feature vector after preliminary fusion; σ represents the fusion weight factor.

[0117] Among them, the four parts of data preprocessing, feature extraction, data synchronization and fusion output can be specifically divided into six parts: data preprocessing, data processing, feature extraction, first fusion processing, second fusion processing and monitoring target positioning estimation. The specific implementation process of the target positioning method based on acoustic and optical fusion data is as follows:

[0118] (1) Data preprocessing: Sonar data and optical data acquired in underwater environments are preprocessed to reduce noise, enhance signals, and extract features.

[0119] 1) Sonar data preprocessing: including filtering, denoising and feature extraction of sound signals, using spectrum analysis or time-frequency domain analysis.

[0120] 2) Optical data preprocessing: The optical image in turbid water is blurry and requires image enhancement, dehazing or feature extraction. Convolutional neural networks can be used to improve image quality or extract target features.

[0121] (2) Data processing

[0122] 1) Sonar data processing: Using deep learning models (convolutional neural networks) to process sonar data can learn the spatial and temporal features in the sound signal, thereby extracting useful information for monitoring and positioning. Useful information may include but is not limited to: the location, shape, movement, speed information of the target and characteristic information of the environment.

[0123] 2) Optical data processing: Use deep learning models to process optical images in turbid water and extract the visual features of the target.

[0124] (3) Feature extraction

[0125] 1) Sound feature extraction: The sound signal is converted into a meaningful feature vector through the convolution layer, pooling layer and fully connected layer. The calculation formula for converting the sound signal into a meaningful feature vector is:

[0126] f a =f sound (y a ; W a )

[0127] Among them, y a Indicates sound input data; f aRepresents the feature vector extracted from the sound data; W a represents the weight parameter of the sound feature extraction model; f sound Represents the sound feature vector.

[0128] 2) Optical feature extraction: The optical image is processed by convolutional neural network to extract the key features in the image. The calculation formula for extracting the key features in the image is:

[0129] f o =f vision (y o ; W o )

[0130] Among them, f o Represents optical input data; y o represents the feature vector extracted from optical data; W o represents the weight parameter of the optical feature extraction model; f vision represents the optical eigenvector.

[0131] (4) The first data fusion.

[0132] The feature vectors of acoustic data and optical data are fused to improve positioning accuracy and stability. The first fusion formula is:

[0133] f1=[f a ,f o ]

[0134] Among them, f1 represents the feature vector after the first fusion, which contains the information of sound and optical data.

[0135] (5) Then, data synchronization and second fusion are performed. The acoustic data and optical data are synchronized using timestamps, and then features are merged through the fusion layer. The second fusion formula is:

[0136] f2=αf a +(1-α)f o

[0137] Among them, f2 represents the feature vector after the second fusion, f a represents the feature vector extracted from acoustic data; f o represents the feature vector extracted from optical data; α represents the fusion weight, which is used to control the relative contribution of acoustic features and optical features in the final fusion features. Usually, α is a weight factor in the range of [0,1], indicating the relative importance of acoustic and optical features.

[0138] (6) Monitoring target positioning estimation. Using the feature vector after the second fusion for positioning estimation, the machine learning model can be used to predict the target's position or generate positioning confidence. The calculation formula for monitoring target positioning estimation is:

[0139]

[0140] in, represents the predicted target position vector; f fusion W represents the machine learning model used to fuse features; fusion represents the weight parameter of the fusion model; f j Denoted as the jth eigenvector.

[0141] It should be noted that in the specific implementation process of the target positioning method based on acoustic and optical fusion data, both the sonar data and the optical data are first preprocessed before further data processing, and in the data processing and feature extraction parts, the sonar data part first preliminarily extracts useful information in the data processing stage, and then further extracts features from the useful information to obtain a sound feature vector; the visual features of the target extracted in the data processing stage of the optical data part and the key features in the image extracted in the feature extraction stage can be understood as parallel content, and there is no need to perform further feature extraction based on the visual features of the target obtained by optical data processing, that is, the deep learning model and the convolutional neural network are used to process the optical image to obtain the visual features of the target and the key features in the image respectively, and then the visual features of the target obtained by the optical data processing and the key features in the image obtained by the optical feature extraction are used as optical feature vectors, and finally the optical feature vectors are fused with the sound feature vectors to obtain the target acoustic and optical fusion data.

[0142] In the specific implementation, the deep learning method is used to train and learn specific underwater scene features through a large amount of data, which can enhance the identification of targets or obstacles in underwater scenes, improve image quality in turbid water bodies, enhance positioning and navigation, and thus improve the positioning accuracy and stability in turbid water bodies to a certain extent.

[0143] Step S102 shown in the embodiment of the present application generates target sound and light fusion data by performing sound and light data fusion processing on the original sonar data and the original optical data, and calculates the first target predicted position of the underwater monitoring target based on the target sound and light fusion data. The sound and light fusion technology can combine the penetration of sonar and the high resolution of optics to more effectively identify targets in turbid water bodies, while eliminating noise and interference in a single data source, thereby improving the prediction accuracy of the target position.

[0144] Step S103, performing multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculating the second target predicted position of the underwater monitoring target and the target UAV position of the water-to-air UAV based on the target positioning fusion data;

[0145] In some embodiments, step S103 may include: performing a first filtering process on the inertial navigation system data using an extended Kalman filter, and performing a first positioning fusion process on the inertial navigation system data after the first filtering process and the ultra-short baseline system data to obtain state fusion data; performing a second filtering process on the ultra-short baseline system data and the long baseline system data using a particle filter, and performing a second positioning fusion process on the ultra-short baseline system data and the long baseline system data after the second filtering process and the visual system data to obtain measurement fusion data; performing a third fusion process on the state fusion data and the measurement fusion data based on a target fusion strategy to obtain target positioning fusion data; and calculating the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data.

[0146] Among them, the edge computing unit (airborne edge computing terminal) configured on the drone body can calculate the second target predicted position of the underwater monitoring target and the current position of the drone based on the positioning fusion data.

[0147] In the specific implementation, the extended Kalman filter (EKF) and particle filter are combined to fuse the inertial navigation system (INS) data, ultra-short baseline system (USBL) data and long baseline system (LBL) data to optimize the target position estimation and continuously tune the algorithm to ensure the stability and accuracy of the positioning system in a dynamic water environment.

[0148] Specifically, the specific process of fusing the inertial navigation system data, the ultra-short baseline system data and the long baseline system data by combining the extended Kalman filter and the particle filter is as follows:

[0149] First, an extended Kalman filter is used to fuse INS data and USBL data to provide a preliminary position estimate. The fusion formula is:

[0150] x t+1 =f(x t ,u t )+w t

[0151] Among them, x t represents the state vector at time t, that is, the state of the system, including position, speed, etc.; f() represents the state transfer function; ut represents the control input; w t represents process noise.

[0152] Particle filtering is used to integrate LBL data and visual system data to optimize target position estimation. Each particle represents a possible state estimate, which gradually approaches the actual state through resampling. The algorithm is continuously tuned through laboratory testing and field verification, that is, the algorithm of the positioning system composed of INS (inertial navigation system), USBL (ultra-short baseline system) and LBL (long baseline system) data is tuned to improve the stability and accuracy of the positioning system in dynamic water environments.

[0153] Among them, the specific formula of the extended Kalman filter (EKF) includes:

[0154] (1) The state prediction formula is:

[0155] in, represents the estimated state of the underwater drone target at the previous moment; x k Represents the state vector, which indicates the state of the system, including position, speed, etc.; u k represents the control input (such as inertial measurement); represents the state estimate of the state vector x at time k.

[0156] (2) The covariance prediction formula is:

[0157] Among them, F k-1 The Jacobian matrix of the state transfer matrix; P k represents the state covariance matrix, reflecting the uncertainty of state estimation; F k The Jacobian matrix of the state transfer matrix describes the derivative of the state change with respect to the state; P k-1|k-1 represents the state estimation covariance of the previous moment; Q k represents the process noise covariance matrix, which represents the uncertainty and noise in the system model; Q k-1 represents the covariance of process noise; P k|k-1 represents the error covariance matrix of the prior estimate of the state vector x at time k.

[0158] (3) The state update (correction) formula is:

[0159]

[0160]

[0161] Among them, z k represents the measurement vector; represents the mapping function from state to measurement; Hk Represents the Jacobian matrix of the observation matrix; R k represents the covariance of the measurement noise; K k represents the Kalman gain matrix, which is used to weightedly fuse the predicted state with the measurement; represents the corrected state estimate of the state vector x at time k.

[0162] Among them, the specific formula of the particle filter includes:

[0163] (1) Monitoring data initialization: by extracting a particle set from a prior distribution Where N is the number of particles.

[0164] (2) State prediction: For each particle, the state transfer function and process noise are applied to generate a set of predicted particles:

[0165] in, It is expressed as a state transition model, which describes how the particle evolves from the state at the previous moment to the current moment.

[0166] (3) Weight update: For each predicted particle Calculate its weight, the calculation formula is:

[0167] in, Expressed as the weight of the particle, reflecting the importance of each particle in the posterior distribution; It is expressed as the probability density of a given state of the measurement model; represents the state of the i-th particle at time a; represents the state of the i-th particle at time a-1; u a represents the external input at time a in state prediction; Specifically, it is the process noise of the i-th particle at time a; It is represented as the process noise of the ith particle at time a-1; z a Expressed as the measured value at time a.

[0168] (4) Resampling: Resample particles according to weights to ensure diversity and representativeness at the next moment.

[0169] In a specific implementation, the process of data fusion performed by the underwater drone in the embodiment of the present application is processed by a combination of an extended Kalman filter and a particle filter. The specific process of data fusion can be implemented by the following steps:

[0170] (1) State fusion: Use the extended Kalman filter to process the inertial measurement data of the inertial navigation system, and then fuse the inertial navigation system data with the ultra-short baseline positioning system data to achieve high accuracy and low latency.

[0171] (2) Measurement fusion: Use particle filters to process the position measurement data of the ultra-short baseline positioning system and the long baseline positioning system, and then fuse the long baseline positioning system data and the visual system data. Among them, the position measurement data of the ultra-short baseline positioning system and the long baseline positioning system and the visual system data are usually nonlinear and uncertain.

[0172] (3) Fusion strategy: The state fusion obtained by the extended Kalman filter and the particle filter is weighted or simply connected with the data output by the measurement fusion to obtain the final fusion result.

[0173] By using the extended Kalman filter to process inertial measurement data, the particle filter to process position measurement data, and combining the state fusion data and measurement fusion data through appropriate fusion strategies, the positioning accuracy and stability of underwater drones can be improved. It can fully utilize the advantages of different acoustic and optical sensors while overcoming their respective limitations, and is suitable for precise positioning, navigation, and target monitoring tasks in complex underwater environments and extreme water bodies.

[0174] Step S103 shown in the embodiment of the present application generates target positioning fusion data by performing multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data, and calculates the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data. Through multi-source positioning fusion, the advantages of each system can be fully utilized, the target position and the drone position can be quickly updated, and timely information support can be provided for subsequent monitoring or rescue missions, while improving the reliability and stability of positioning.

[0175] Step S104, performing environment adaptive navigation control processing on the underwater environment data and the water turbidity data based on an environment adaptive navigation control algorithm to dynamically acquire target water body change data;

[0176] In some embodiments, step S104 may include: performing environment adaptive navigation control processing on underwater environment data and water turbidity data based on an environment adaptive navigation control algorithm to dynamically acquire target water body change data.

[0177] Among them, the edge computing unit (airborne edge computing terminal) configured on the main body of the drone can adjust the navigation parameters of the drone through an environmental adaptive navigation control algorithm to cope with complex underwater environments and obtain water body change data.

[0178] In the embodiment of the present application, a highly adaptive real-time adaptive navigation algorithm is used to process underwater environmental data and water turbidity data to improve the effectiveness of the drone in various underwater environments, so that the drone can safely and effectively perform monitoring and auxiliary execution tasks, and can adjust the behavior of the drone in real time according to environmental changes. In the specific implementation, a highly adaptive real-time adaptive navigation algorithm is used in combination with MPC (Model Predictive Control) and RL (Reinforcement Learning) technologies to achieve control that can adjust the behavior of the drone in real time according to environmental changes.

[0179] In order to describe the implementation process of the adaptive navigation algorithm combining MPC and RL in detail, it is necessary to cover the following aspects: the state and observation model of the system, the setting and solution of MPC, the integration of RL, and the implementation of feedback linearization control. The specific implementation process is as follows:

[0180] 1. State and observation model of the system.

[0181] (1) State model:

[0182] The state vector x contains information such as position coordinates, velocity components, and attitude angles. Dynamic behavior of the system It is described by the following differential equation:

[0183]

[0184] Among them, f represents the dynamic model of the system, u represents the control input, w represents the process noise, and t represents time.

[0185] (2) Observation model:

[0186] The observation vector y comes from sensors (such as sonar and optical sensors). The relationship between the observation vector and the state vector x is given by the observation model. The specific expression is as follows:

[0187] y=h(x)+v

[0188] Where h represents the mapping function from state to observation and v represents the measurement noise.

[0189] 2. Setting and solving MPC.

[0190] (1) Objectives:

[0191] The goal of MPC is to optimize the current control input by predicting the future state. The optimization goal is usually to minimize the cumulative error or cost of future predictions, as shown in the following expression:

[0192]

[0193] Among them, Q and R represent the weight matrices of state error and control input respectively, N represents the length of the prediction time domain, J represents the performance indicator or cost to be minimized, k represents the current time step, T represents the transpose, represents the state vector x at time step k+1 k+1 The transpose of x k+1 represents the value of the system state vector at time step k+1, represents the control input vector u at time step k k The transpose of u k represents the control input vector at time step k.

[0194] (2) Constraints:

[0195] Constraints include state constraints, input constraints, and dynamic constraints:

[0196] x k+1 =Ax k +Bu k

[0197] x min ≤x k ≤x max

[0198] u min ≤u k ≤u max

[0199] Where A and B represent the influence matrices of the state and control input, respectively, which are used to describe the linear approximation of the system, x k+1 represents the state vector at time step k+1, x k represents the state vector at time step k, x min Represents the system state vector x k The minimum allowed value, x max Represents the system state vector x k The maximum permissible value, u min represents the control input vector u k The minimum allowed value, u max represents the control input vector u k The maximum allowed value.

[0200] (3) Solution:

[0201] The MPC optimization problem is solved through quadratic programming (QP) or other optimization algorithms to calculate the optimal control input u0 at the current moment.

[0202] 3. Integration of RL

[0203] (1) Strategy learning:

[0204] RL is used to learn and optimize the strategy π to optimize the long-term cumulative return in a complex and uncertain environment. The update of the strategy is based on the action value function Q, which is expressed as follows:

[0205] Q(s,a)=E[r+γmax a′ Q(s′,a′)|s,a]

[0206] Among them, γ represents the discount factor, r represents the immediate reward, s and a represent the state and action respectively, and E represents the expectation operator.

[0207] (2) Collaboration between MPC and RL:

[0208] In the synergistic combination of MPC and RL, MPC acts as a short-term optimizer to calculate the current optimal control input; RL acts as a long-term policy learner to optimize the objective function or constraints of MPC, or directly adjust policy parameters. The objective function of MPC can be dynamically updated to reflect the learning results of the RL policy.

[0209] 4. Feedback linearization control

[0210] (1) Objectives:

[0211] Feedback linearization control is used to simplify the control design of the system. The goal is to transform the nonlinear system into a linear system to facilitate controller design.

[0212] (2) Methods:

[0213] The control input u is designed to eliminate nonlinear effects and achieve linear control. The specific expression is:

[0214] u=u nom +K(zz nom )

[0215] Among them, u nom represents the reference control input, K represents the gain matrix, z and z nom Represent the current observation value and the benchmark observation value respectively.

[0216] Specifically, when underwater drones use highly adaptive real-time navigation algorithms combined with specific environmental perception sound and light fusion systems and controller designs, they can achieve more accurate and efficient real-time target navigation and control of underwater drones, thereby verifying the effectiveness of the algorithm in various underwater environments and enabling drones to perform tasks safely and effectively.

[0217] Step S104 shown in the embodiment of the present application performs environmental adaptive navigation control processing on underwater environmental data and water turbidity data to dynamically obtain target water body change data, wherein the environmental adaptive control can dynamically adjust the navigation parameters of the drone according to the real-time changes of the water body, thereby improving its adaptability and stability in complex environments. It can be understood that by analyzing the underwater environmental data and water turbidity data, the navigation parameters of the drone are adjusted in real time using the environmental adaptive navigation control algorithm to cope with the complex underwater environment. That is, the control algorithm can adjust the motion path and speed of the drone according to the real-time changes of the water body, thereby dynamically obtaining and adapting to the changes of the target water body, thereby enhancing the stability and task execution efficiency of the drone under different water conditions.

[0218] Step S105, determining the target underwater position of the underwater monitoring target according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

[0219] In some embodiments, step S105 may include: calculating the underwater monitoring data, the first target predicted position, the second target predicted position, the target drone position and the target water body change data based on the airborne edge computing terminal of the water-air drone to obtain the target underwater position of the underwater monitoring target.

[0220] Among them, the water-air drone is equipped with a buoy releasing device, which is used to release the hydroacoustic communication buoy into the water and on the water.

[0221] In some embodiments, it may also include: sending underwater monitoring data, a first target predicted position, a second target predicted position, a target drone position, and target water body change data to a hydroacoustic communication buoy through a water-air drone; channel coding the underwater monitoring data, a first target predicted position, a second target predicted position, a target drone position, and target water body change data through a communication buoy system in the hydroacoustic communication buoy to obtain channel coded data; sending the channel coded data to a remote operation and processing center through the communication buoy system; so that the remote operation and processing center decodes the channel coded data to obtain target decoded data, and the remote operation and processing center determines the target underwater position of the underwater monitoring target based on the target decoded data.

[0222] In an embodiment of the present application, a communication buoy system is utilized to transmit data collected by an underwater drone (target monitoring location data, underwater environment data, water turbidity data, and drone location data) to the buoy in real time via hydroacoustic communication, and then the data is sent to a remote computing and processing center via radio signals, thereby improving the efficiency of real-time data transmission and processing.

[0223] In the specific implementation, the mechanism implementation process and the formulas involved in the communication buoy system include:

[0224] The embodiment of the present application adopts the Multiple Input Multiple Output (MIMO) system and Orthogonal Frequency Division Multiplexing (OFDM) technology for communication. Specifically, the signal coverage and quality are optimized by adjusting the antenna configuration and power allocation of the MIMO system; efficient signal coding technology is introduced, and LDPC (Low-Density Parity-Check) and Turbo coding are used to reduce errors and data loss in underwater communications; through iterative testing and optimization, the technical solution can be effectively applied to actual underwater operations.

[0225] Specifically, OFDM technology is used to modulate and demodulate the signal, thereby significantly improving the robustness of underwater UAV communication buoy data transmission. The specific process and formula include:

[0226] M1, OFDM modulation formula:

[0227] K1. Generate time domain symbols. Assume that the data symbol to be sent is s[n], and the time domain symbol x[n] is obtained after modulation. The formula for generating the time domain symbol is:

[0228]

[0229] Where N represents the number of subcarriers in each OFDM symbol, n = 0, 1, ..., N-1; s[k] represents the data symbol sent on the kth subcarrier; x[n] represents the output signal in the time domain, which is a complex number here, representing the signal received at time n; n represents the discrete time index in the time domain, indicating the moment when the signal is received; k represents the discrete frequency index in the frequency domain, indicating the number or frequency component of the subcarrier; represents a complex exponential term, where j is the imaginary unit and π is the circumference of a circle.

[0230] K2, adding a cyclic prefix (CP) can offset the inter-symbol interference caused by multipath. The formula for adding a cyclic prefix is:

[0231]

[0232] Among them, N CP Indicates the cyclic prefix length.

[0233] K3, generate frequency domain symbols. For the time domain symbol x CP[n] is subjected to a discrete Fourier transform (DFT) to obtain the frequency domain symbol X[k]. The formula for generating the frequency domain symbol is:

[0234]

[0235] M2, OFDM demodulation formula:

[0236] L1, received signal: The received signal is r[n], remove the cyclic prefix to get r data [n].

[0237] L2, frequency domain signal: for r data [n] is subjected to discrete Fourier transform to obtain the received frequency domain symbol R[k]:

[0238]

[0239] L3, channel estimation H[k], using the pilot symbol or specific training sequence sent by the transmitter, the receiver estimates the channel frequency response H[k] on each subcarrier, and estimates it through the following formula:

[0240]

[0241] Among them, X pilot [k] represents the frequency domain representation of the transmitted pilot symbol.

[0242] L4, frequency domain equalization, the receiver uses the estimated channel frequency response H[k] to perform frequency domain equalization to compensate for the fading and phase distortion caused by the channel:

[0243]

[0244] in, Represents the data symbol obtained by demodulation.

[0245] Optimize signal coverage and quality by adjusting the antenna configuration and power allocation of the MIMO system.

[0246] P opto =argmin P ∣∣HX-Y∣∣ 2

[0247] Among them, P opt The calculation formula is used to represent the process of optimizing the power allocation P at the receiving end to minimize the error between the received signal Y and the expected signal HX; optIt means to determine the best power allocation strategy by minimizing the error so that the receiving end can recover the transmitted data more accurately; H represents the channel matrix; X represents the transmitted signal; Y represents the received signal; P represents the power allocation matrix.

[0248] The embodiments of the present application can further improve the reliability and error rate performance of communications by adopting advanced coding technologies such as LDPC (low-density parity check) and Turbo coding, and by modifying and optimizing channel estimation and frequency domain equalization.

[0249] Among them, the communication system between the underwater drone and the released multiple buoys uses MIMO technology to improve communication performance. The multiple input multiple output (MIMO) system of the buoy-type underwater drone specifically includes:

[0250] The signal model at the transmitter side assumes that the underwater drone at the transmitter side has N tx antennas, the transmitted signal vector is Each element represents a modulation symbol on a transmitting antenna, and the expression of the transmitted signal vector is:

[0251]

[0252] Channel model, assuming that the underwater communication channel can be represented as an N tx ×N rx The channel matrix, Among them, N tx , represents the number of antennas at the transmitter (underwater drone), N rx Indicates the number of receiving antennas at the buoy end.

[0253] The signal received by the receiving end is expressed as: y=Hx+n.

[0254] Where n represents additive white Gaussian noise (AWGN for short); σ 2 represents the variance of additive Gaussian noise, represents the intensity of noise, and satisfies in, Represents the complex field; N rx ×N rx Indicates the independence of the noise on each antenna; Indicated as N rx ×N rx The identity matrix of .

[0255] Received signal quality measurement: The signal-to-noise ratio (SNR) of the received signal is usually used to measure the communication quality. For the i-th receiving antenna, its SNR is:

[0256]

[0257] Among them, H i represents the i-th row of the channel matrix H.

[0258] Optimizing monitoring targets and optimizing the buoy communication signal coverage and quality of underwater drones can be done in the following ways:

[0259] (1) Optimize the antenna configuration of underwater drones and adjust the number of antennas N at the transmitter and receiver tx and N rx , to maximize the capacity and coverage of the communication system.

[0260] (2) Power allocation optimization: Based on the channel state information (CSI), the power allocation on each antenna of the transmitter is optimized to maximize the received signal quality of each receiver.

[0261] The specific optimization problem can be formalized as a convex optimization problem:

[0262]

[0263] Where x represents the transmitted symbol vector, and the optimization goal is to maximize the overall system data rate or minimize the bit error rate.

[0264] By adjusting the antenna configuration and power allocation of the MIMO system, the communication signal coverage and quality between the underwater drone and the buoy can be optimized.

[0265] Step S105 shown in the embodiment of the present application provides real-time data to the remote operation and processing center by sending underwater monitoring data, the predicted position of the first target, the predicted position of the second target, the target UAV position, and the target water body change data to the remote operation and processing center, so that the remote operation and processing center can carry out effective task planning and decision-making, thereby improving the coordination and efficiency of underwater operations, and enabling the decision-making center to make decisions quickly based on the latest information and effectively allocate resources.

[0266] In the specific implementation, after the water-air drone receives a rescue or monitoring task, the water-air drone performs the corresponding task. During the task execution, the airborne edge computing terminal of the water-air drone calculates the acquired underwater monitoring data, target predicted position, target drone position, and target water body change data, and can obtain the target underwater position of the underwater monitoring target. The airborne edge computing terminal of the water-air drone can achieve the purpose of detection and navigation at the same time until the task is completed. On the other hand, the acquired underwater monitoring data, target predicted position, target drone position, and target water body change data can be transmitted back to the remote operation processing center of the ground station through the buoy communication system, and then processed by the remote operation processing center of the ground station and returned to the drone for operation. This method can be used as a control backup method, and can also be used as a real-time remote monitoring method for drone operations.

[0267] Steps S101 to S105 shown in the embodiment of the present application are obtained by acquiring underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; performing sound and light data fusion processing on the original sonar data and the original optical data to generate target sound and light fusion data, and calculating the first target predicted position of the underwater monitoring target based on the target sound and light fusion data; performing multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculating the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data; performing environmental adaptive navigation control processing on the underwater environment data and the water turbidity data based on an environmental adaptive navigation control algorithm to dynamically obtain target water body change data; determining the target underwater position of the underwater monitoring target according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target drone position and the target water body change data. The embodiment of the present application provides a rich source of information for subsequent data analysis and processing by collecting a variety of underwater monitoring data, which can improve the comprehensiveness and accuracy of data analysis and processing; the system's perception of the underwater environment and the accuracy of target positioning can be improved through the fusion processing of sound and light data; and the positioning data of multiple positioning systems are subjected to multi-source positioning fusion processing, which can improve the positioning accuracy of underwater monitoring targets and the position of the drone itself; by analyzing underwater environmental data and water turbidity data, the navigation parameters of the drone are adjusted in real time using an environmental adaptive navigation control algorithm to cope with complex underwater environments, that is, the control algorithm can adjust the movement path and speed of the drone according to the real-time changes of the water body, thereby dynamically acquiring and adapting to the changes of the target water body, and enhancing the stability and task execution efficiency of the drone under different water conditions; finally, the target underwater position of the underwater monitoring target is determined based on the underwater monitoring data, the predicted position of the first target, the predicted position of the second target, the target drone position, and the target water body change data, so as to provide timely and accurate information support for subsequent monitoring or rescue tasks and improve the efficiency of underwater task execution.

[0268] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0269] The embodiments of the present application provide a water-to-air drone monitoring method based on sound-light fusion and multi-source positioning, which aims to improve the efficiency of underwater monitoring and rescue and the accuracy of data transmission, and greatly reduce communication interference; optimize the operating efficiency, flight time and communication stability of underwater drones, and provide an efficient and reliable technical solution for target monitoring and rescue in complex underwater environment areas.

[0270] The overall solution of the water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning provided in the embodiment of the present application includes:

[0271] N1. After receiving the monitoring and rescue mission, the underwater drone uses acoustic and optical sensors, monitoring cameras, visual cameras, perception modules and positioning modules (baseline positioning module, inertial positioning module, visual positioning module) to obtain underwater environmental data, water turbidity, target perception and positioning data, and conducts acoustic and optical fusion positioning data analysis and preliminary processing.

[0272] N2. Use the Kalman filter to pre-process the acquired sound and light data, and then use the deep learning method to optimize the fusion processing of the sound and light data to improve the perception and positioning capabilities of the monitoring and rescue system.

[0273] N3. Combine the extended Kalman filter and particle filter to perform on the inertial navigation system data, ultra-short baseline system data and long baseline system data, and then optimize the target position estimation, continuously tune the algorithm, and improve the stability and accuracy of the positioning system in a dynamic water environment.

[0274] N4. Use highly adaptive real-time adaptive navigation algorithms to process underwater environmental data and water turbidity data to improve the effectiveness of drones in various underwater environments, so that drones can perform monitoring and rescue tasks safely and effectively, and can adjust the behavior of drones in real time according to environmental changes.

[0275] N5. The airborne edge computing terminal based on the water-air drone calculates the underwater monitoring data, the target predicted position, the target drone position and the target water body change data to obtain the target underwater position of the underwater monitoring target. In addition, the communication buoy system can be used to transmit the data collected by the underwater drone to the buoy in real time through hydroacoustic communication, and then send the data to the remote computing processing center through radio signals, thereby improving the efficiency of real-time data transmission and processing.

[0276] The embodiments of the present application optimize the operating efficiency, perception and positioning accuracy, effective endurance time and communication height stability of underwater drones, which is beneficial to improving the overall performance of underwater drone systems and provides an efficient, reliable and stable technical solution for east-west monitoring and rescue of turbid water bodies in extreme water environment areas.

[0277] It should be pointed out that this embodiment only briefly illustrates the general process of the water-air UAV monitoring method based on sound-light fusion and multi-source positioning. The detailed description of each step can refer to the relevant content in the aforementioned embodiment and will not be repeated here. It can be understood that the present invention is not limited to this.

[0278] The embodiment of the present application obtains underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; the original sonar data and the original optical data are subjected to acoustic and optical data fusion processing to generate target acoustic and optical fusion data, and the first target predicted position of the underwater monitoring target is calculated based on the target acoustic and optical fusion data; the inertial navigation system data, the ultra-short baseline system data and the long baseline system data are subjected to multi-source positioning fusion processing to generate target positioning fusion data, and the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone are calculated based on the target positioning fusion data; the underwater environment data and the water turbidity data are subjected to environmental adaptive navigation control processing based on an environmental adaptive navigation control algorithm to dynamically obtain target water body change data; the target underwater position of the underwater monitoring target is determined according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target drone position and the target water body change data. The embodiment of the present application provides a rich source of information for subsequent data analysis and processing by collecting a variety of underwater monitoring data, which can improve the comprehensiveness and accuracy of data analysis and processing; the system's perception of the underwater environment and the accuracy of target positioning can be improved through the fusion processing of sound and light data; and the positioning data of multiple positioning systems are subjected to multi-source positioning fusion processing, which can improve the positioning accuracy of underwater monitoring targets and the position of the drone itself; by analyzing underwater environmental data and water turbidity data, the navigation parameters of the drone are adjusted in real time using an environmental adaptive navigation control algorithm to cope with complex underwater environments, that is, the control algorithm can adjust the movement path and speed of the drone according to the real-time changes of the water body, thereby dynamically acquiring and adapting to the changes of the target water body, and enhancing the stability and task execution efficiency of the drone under different water conditions; finally, the target underwater position of the underwater monitoring target is determined based on the underwater monitoring data, the predicted position of the first target, the predicted position of the second target, the target drone position, and the target water body change data, so as to provide timely and accurate information support for subsequent monitoring or rescue tasks and improve the efficiency of underwater task execution.

[0279] The application scenarios of the water-to-air drone monitoring method based on sound-light fusion and multi-source positioning provided in the embodiment of the present application are not limited to rescue scenarios, but can also be underwater detection, underwater monitoring and other scenarios, such as inspection, operation, flaw detection, surveying and mapping, etc.

[0280] See also Figure 5 The embodiment of the present application also provides a water-air drone monitoring system 500 based on acoustic-optical fusion and multi-source positioning, which can implement the above-mentioned water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning. The system 500 includes the following modules:

[0281] The underwater monitoring data acquisition module 501 is used to acquire underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data;

[0282] The acoustic-optical data fusion module 502 is used to perform acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculate the first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data;

[0283] The positioning data fusion module 503 is used to perform multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculate the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data;

[0284] A navigation control processing module 504 is used to perform environment adaptive navigation control processing on the underwater environment data and the water turbidity data based on an environment adaptive navigation control algorithm to dynamically obtain target water body change data;

[0285] The target underwater position determination module 505 is used to determine the target underwater position of the underwater monitoring target based on the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

[0286] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0287] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned water-air drone monitoring method based on sound-light fusion and multi-source positioning when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0288] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0289] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0290] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0291] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the water-air UAV monitoring method based on acoustic-optical fusion and multi-source positioning of the embodiment of this application;

[0292] Input / output interface 603, used to implement information input and output;

[0293] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0294] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0295] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0296] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned water-air UAV monitoring method based on sound-light fusion and multi-source positioning.

[0297] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0298] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0299] The embodiments of the present application provide a water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning, a water-air drone monitoring system based on acoustic-optical fusion and multi-source positioning, an electronic device and a storage medium, which obtain underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; perform acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculate the first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data; perform acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculate the first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data; perform acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data; and ...; and perform acoustic-optical data The multi-source positioning fusion processing is performed on the underwater navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and the second target predicted position of the underwater monitoring target and the target UAV position of the water-to-air UAV are calculated based on the target positioning fusion data; the underwater environment data and the water turbidity data are subjected to environment adaptive navigation control processing based on the environment adaptive navigation control algorithm to dynamically obtain the target water body change data; the target underwater position of the underwater monitoring target is determined according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data. The embodiment of the present application provides a rich source of information for subsequent data analysis and processing by collecting a variety of underwater monitoring data, which can improve the comprehensiveness and accuracy of data analysis and processing; the system's perception of the underwater environment and the accuracy of target positioning can be improved through the fusion processing of sound and light data; and the positioning data of multiple positioning systems are subjected to multi-source positioning fusion processing, which can improve the positioning accuracy of underwater monitoring targets and the position of the drone itself; by analyzing underwater environmental data and water turbidity data, the navigation parameters of the drone are adjusted in real time using an environmental adaptive navigation control algorithm to cope with complex underwater environments, that is, the control algorithm can adjust the movement path and speed of the drone according to the real-time changes of the water body, thereby dynamically acquiring and adapting to the changes of the target water body, and enhancing the stability and task execution efficiency of the drone under different water conditions; finally, the target underwater position of the underwater monitoring target is determined based on the underwater monitoring data, the predicted position of the first target, the predicted position of the second target, the target drone position, and the target water body change data, so as to provide timely and accurate information support for subsequent monitoring or rescue tasks and improve the efficiency of underwater task execution.

[0300] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0301] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0302] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0303] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0304] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0305] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0306] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0307] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0308] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0309] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0310] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A water-air drone monitoring method based on acoustic-optical fusion and multi-source positioning, characterized in that: The method comprises the following steps: Acquiring underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, raw sonar data, raw optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; Performing acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculating a first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data; Performing multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculating the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data; Performing environment adaptive navigation control processing on the underwater environment data and the water turbidity data based on an environment adaptive navigation control algorithm to dynamically acquire target water body change data; The target underwater position of the underwater monitoring target is determined according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

2. The method according to claim 1, characterized in that: The positioning data also includes sonar positioning data, and the acquisition of underwater monitoring data includes: Calculating the target underwater distance between the underwater monitoring target and the water-to-air drone based on the original sonar data; Calculate the target underwater direction of the underwater monitoring target in the acoustic beam according to the horizontal direction angle between the underwater monitoring target and the acoustic beam of the original sonar data, the vertical direction angle between the underwater monitoring target and the acoustic beam, and the beam weight vector of the acoustic beam; The first three-dimensional coordinate position of the underwater monitoring target is calculated according to the target underwater direction and the target underwater distance, and the first three-dimensional coordinate position is used as the sonar positioning data.

3. The method according to claim 1, characterized in that The original optical data includes original image data of the underwater monitoring target, the positioning data also includes optical positioning data, and the acquiring of underwater monitoring data includes: Preprocessing the original image data to obtain target image data; Performing feature extraction on the target image data to obtain target feature points and target sub-features corresponding to the target feature points; Perform feature matching on the target sub-feature with a plurality of sub-features in a target feature library to obtain candidate sub-features; Calculating the similarity between the target sub-feature and the candidate sub-feature to obtain a similarity result; The second three-dimensional coordinate position of the underwater monitoring target is calculated according to the similarity result, the two-dimensional coordinate position of the target feature point and the parallax of the target feature point, and the second three-dimensional coordinate position is used as the optical positioning data.

4. The method according to claim 1, characterized in that: The performing acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculating a first target predicted position of the underwater monitoring target based on the target acoustic-optical fusion data, comprises: Preprocessing the original sonar data to obtain candidate sonar data; Performing information extraction processing on the candidate sonar data to obtain target environment perception information; Performing feature extraction processing on the target environment perception information to obtain a target sound feature vector; Preprocessing the original optical data to obtain candidate optical data; Performing a first feature extraction process on the candidate optical data to obtain a target visual feature vector; Performing a second feature extraction process on the candidate optical data to obtain a target image key feature vector, and using the target visual feature vector and the target image key feature vector as a target optical feature vector; Performing a first fusion process on the target sound feature vector and the target optical feature vector to obtain a first fused feature vector; Performing timestamp synchronization processing on the target sound feature vector and the target optical feature vector in the first fusion feature vector, and performing a second fusion processing on the target sound feature vector and the target optical feature vector after the timestamp synchronization processing to obtain a second fusion feature vector, and using the second fusion feature vector as the target sound and light fusion data; The target sound and light fusion data is input into the target machine learning model to obtain the first target predicted position of the underwater monitoring target.

5. The method according to claim 1, characterized in that The multi-source positioning fusion processing is performed on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and the second target predicted position of the underwater monitoring target and the target UAV position of the water-to-air UAV are calculated based on the target positioning fusion data, including: Performing a first filtering process on the inertial navigation system data using an extended Kalman filter, and performing a first positioning fusion process on the inertial navigation system data after the first filtering process and the ultra-short baseline system data to obtain state fusion data; Performing a second filtering process on the ultra-short baseline system data and the long baseline system data by using a particle filter, and performing a second positioning fusion process on the ultra-short baseline system data and the long baseline system data after the second filtering process and the visual system data to obtain measurement fusion data; Performing a third fusion process on the state fusion data and the measurement fusion data based on a target fusion strategy to obtain the target positioning fusion data; The second target predicted position of the underwater monitoring target and the target UAV position of the water-to-air UAV are calculated based on the target positioning fusion data.

6. The method according to claim 1, characterized in that The method according to claim 1 is characterized in that determining the target underwater position of the underwater monitoring target according to the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data comprises: Based on the airborne edge computing terminal of the water-to-air drone, the underwater monitoring data, the first target predicted position, the second target predicted position, the target drone position and the target water body change data are calculated to obtain the target underwater position of the underwater monitoring target.

7. The method according to claim 1, characterized in that The water-air drone is equipped with a buoy releasing device, and the buoy releasing device is used to release the hydroacoustic communication buoy in the water position and the water position. The method also includes: The underwater monitoring data, the first target predicted position, the second target predicted position, the target drone position and the target water body change data are sent to the hydroacoustic communication buoy by the water-air drone, and the underwater monitoring data, the first target predicted position, the second target predicted position, the target drone position and the target water body change data are channel-coded by the communication buoy system in the hydroacoustic communication buoy to obtain channel-coded data; The channel coded data is sent to a remote operation and processing center through the communication buoy system, so that the remote operation and processing center decodes the channel coded data to obtain target decoded data, and the remote operation and processing center determines the target underwater position of the underwater monitoring target according to the target decoded data.

8. The water-air drone monitoring system based on acoustic-optical fusion and multi-source positioning is characterized by: The system includes the following modules: An underwater monitoring data acquisition module is used to acquire underwater monitoring data; the underwater monitoring data includes underwater environment data, water turbidity data, original sonar data, original optical data and positioning data, and the positioning data includes inertial navigation system data, ultra-short baseline system data and long baseline system data; an acoustic-optical data fusion module, configured to perform acoustic-optical data fusion processing on the original sonar data and the original optical data to generate target acoustic-optical fusion data, and calculate a first target predicted position of an underwater monitoring target based on the target acoustic-optical fusion data; A positioning data fusion module is used to perform multi-source positioning fusion processing on the inertial navigation system data, the ultra-short baseline system data and the long baseline system data to generate target positioning fusion data, and calculate the second target predicted position of the underwater monitoring target and the target drone position of the water-to-air drone based on the target positioning fusion data; A navigation control processing module, used for performing environment adaptive navigation control processing on the underwater environment data and the water turbidity degree data based on an environment adaptive navigation control algorithm, so as to dynamically obtain target water body change data; The target underwater position determination module is used to determine the target underwater position of the underwater monitoring target based on the underwater monitoring data, the first target predicted position, the second target predicted position, the target UAV position and the target water body change data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Underwater sound single carrier communication frequency domain array processing method based on ultra-short baseline array

    CN119854072A

  • A frequency domain array processing method for underwater acoustic single-carrier communication based on ultra-short baseline array

    CN119854072B

  • Method and system for positioning personnel under mine

    CN120121033A

  • A method and system for locating personnel in a mine

    CN120121033B

  • Equipment fault detection method based on multi-source data

    CN121935752A