MEMS-based multi-modal water-air-land cross-medium cooperative sensing system and method

Abstract

The invention discloses a multi-mode water-air-land cross-medium cooperative sensing system and method based on an MEMS (Micro Electro Mechanical System). The system comprises an underwater robot, a buoy station, an unmanned aerial vehicle and a ground control station, each node is provided with an MEMS sensor integrating acoustics, optics, magnetism, inertia and other modes, and data transmission and relay are achieved through acoustics, optical communication and radio links. And the ground control station carries out fusion processing, target identification and decision control on multi-source data through time synchronization and space registration, and issues a cooperative instruction to each node to realize a perception-decision-re-perception closed loop. The method comprises five steps of node deployment and calibration, multi-modal data acquisition, hierarchical transmission and sharing, adaptive weighted fusion and closed-loop cooperative control. According to the invention, global coverage, high-precision and multi-mode environment and target perception can be provided in a complex water-air-land environment, and the detection reliability and the communication efficiency are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent sensing and unmanned system, and particularly relates to a multi-modal water-air-land cross-medium cooperative sensing system and method based on MEMS. BACKGROUND

[0002] The cross-medium environment (including underwater, air and ground) has complex and changeable physical conditions, and a single type of sensor faces the problems of sensing blind area and limited precision in such an environment. For example, in the underwater environment, the optical sensor has limited detection distance due to the influence of turbid water, and the acoustic sensor is limited by noise interference and resolution; while in the air / land environment, radar, camera and other sensors are powerless to underwater targets. This leads to the fact that the traditional single sensor can only obtain limited environmental information from a certain angle, and it is difficult to fully describe the target or scene, and the anti-interference ability is also weak. In order to improve the sensing coverage and precision, the industry has begun to explore the cooperative sensing mode of multiple platforms and multiple sensors. For example, autonomous underwater vehicles can use acoustic, magnetic and other sensors to detect underwater targets, and unmanned aerial vehicles use optical, radar and other sensors to monitor the water surface and airspace; the two can jointly detect and identify targets existing in the underwater and water surface at the same time, which significantly improves the detection success rate and precision. The American SeaWeb plan also developed a cross-medium communication buoy, which integrates underwater acoustic communication devices and various radio equipment to realize the interconnection of underwater sensing network and air / ground communication network. Such wireless / acoustic communication buoy as the gateway for bidirectional communication between underwater information nodes and water surface network is regarded as the key link of cross-domain information transmission.

[0003] Meanwhile, the progress of Microelectro Mechanical Systems (MEMS) sensing technology provides a new way for the miniaturization and integration of multi-modal sensors. MEMS sensors have the advantages of significant miniaturization, low power consumption and low cost, and can integrate multiple sensing units on a single chip. For example, a bionic artificial fish lateral line sensing system with a diameter of less than 6 mm has been developed, which can detect multiple modal signals such as water flow, sound pressure and electric field in such a small structure, and embedding it into underwater robots can greatly enhance the obstacle avoidance and environmental perception capabilities. However, the existing technology has not fully applied these advanced MEMS multi-modal sensors to the water-air-land cross-medium collaborative sensing network. On the one hand, there is a lack of unified sensing and data fusion mechanism between air drones, water buoys, underwater robots and ground stations, making it difficult to form a truly three-dimensional situational awareness; on the other hand, there are still technical bottlenecks in time synchronization, coordinate registration and fusion algorithm of sensor data in different media, resulting in the overall performance of the cross-medium sensing system not being fully utilized. Therefore, there is an urgent need for an innovative cross-medium collaborative sensing system architecture that organically combines multi-modal sensing of underwater, air and land platforms to achieve seamless sensing and information interconnection in complex environments, breaking through the limitations of traditional technology. SUMMARY

[0004] Technical purpose: In view of the deficiencies of the prior art that the different medium environment sensing is isolated, there are more blind areas and information fusion is difficult, the present application discloses a multi-modal water-air-land cross-medium collaborative sensing system and method based on MEMS.

[0005] Technical scheme: In order to achieve the above technical purpose, the present application adopts the following technical scheme:

[0006] A multi-modal water-air-land cross-medium collaborative sensing system based on MEMS, comprising:

[0007] An underwater sensing module comprising an underwater robot equipped with a MEMS multi-modal sensor, for collecting acoustic, optical, magnetic, inertial and other types of first environmental information in the water medium, and transmitting the environmental information to the water surface relay module through the underwater communication unit;

[0008] A water surface relay module comprising a buoy station deployed on the water surface, the buoy station having a water acoustic communication device and a radio communication device, for receiving the first environmental information sent by the underwater robot, communicating with the underwater robot through the water acoustic link, and transmitting the underwater environmental information to the air sensing module and the ground control module through the radio link;

[0009] An aerial perception module comprising a UAV carrying MEMS multi-modal sensors for collecting second environmental information of images, infrared, and laser radar point clouds in the air and sending the second environmental information to the ground control module through an airborne wireless communication unit or relaying through the buoy station;

[0010] A ground control module comprising a ground control station for receiving multi-source environmental information sent by the buoy station and the UAV, performing time synchronization, coordinate registration, and data fusion processing on the multi-source environmental information to obtain comprehensive perception results of the target and environmental state information, and for sending control instructions to the underwater robot, the buoy station, and the UAV to cooperatively complete the perception task.

[0011] Preferably, the MEMS multi-modal sensors carried by the underwater robot comprise: a micro hydrophone array for detecting underwater acoustic signals; a micro optical imaging sensor for collecting underwater images or optical ranging data; a MEMS magnetometer for detecting underwater magnetic field anomalies; a MEMS inertial measurement unit for measuring the attitude and motion parameters of the underwater robot; and a bionic flow field and electric field sensor array for sensing local water flow disturbance and underwater electric field signals.

[0012] Preferably, the buoy station has a wireless / underwater acoustic dual communication module, wherein the wireless communication module comprises at least one remote radio device to realize bidirectional data communication with the shore ground control station or the UAV, and the underwater acoustic communication module comprises a transceiving transducer to realize bidirectional underwater acoustic communication with the underwater robot; the buoy station further comprises an environmental sensor for measuring third environmental information of the water surface or water body, and can send the third environmental information together with the first and second environmental information.

[0013] Preferably, the MEMS multi-modal sensors carried by the UAV comprise a visible light camera, an infrared imager, and a micro laser radar for obtaining multi-spectral images and distance data of a wide area in the air and on the ground; and further comprise a micro magnetometer and a barometer for assisting navigation and target detection; the onboard wireless communication unit of the UAV supports ad hoc relay function and can act as a communication relay node between the buoy station and the ground control station to expand the communication coverage and improve the bandwidth.

[0014] Preferably, the ground control station comprises: a data processing and fusion unit for performing fusion calculation on multi-source environmental information data from the underwater robot, the buoy station, and the UAV; a human-computer interaction terminal for displaying the fused environmental model, target information, and receiving user instructions; a communication interface module for interacting data with external systems through wireless or wired networks; the data processing and fusion unit is configured to execute a multi-sensor data fusion algorithm including time synchronization, spatial coordinate conversion, feature extraction, and weighted fusion decision steps to generate a three-dimensional situation awareness result.

[0015] A multi-modal water-air-land cross-medium collaborative perception method based on MEMS, applied to a multi-modal water-air-land cross-medium collaborative perception system based on MEMS as described above, comprising the following steps:

[0016] S1, deploying underwater robots, buoy stations, unmanned aerial vehicles and ground control station nodes in the target monitoring area, synchronously calibrating the time and coordinates of each node, so that they have a unified time reference and spatial reference coordinate system;

[0017] S2, the underwater robot collects acoustic, optical, magnetic and inertial first environmental information data in the underwater medium through its MEMS multi-modal sensor, the unmanned aerial vehicle collects image, infrared and laser point cloud second environmental information data in the air medium through its MEMS multi-modal sensor, the buoy station collects water surface environmental parameters as third environmental information data, and the ground control station receives or collects ground environmental information as fourth environmental information data;

[0018] S3, the underwater robot sends the first environmental information data to the buoy station through underwater acoustic communication, the buoy station packages the received first environmental information data and sends it to the ground control station and the unmanned aerial vehicle through wireless communication, and the unmanned aerial vehicle sends the second environmental information data to the ground control station through a wireless link, thereby realizing cross-medium transmission and sharing of multi-source environmental information data;

[0019] S4, the ground control station fuses the received first, second, third and fourth environmental information data, including time alignment, coordinate registration, feature extraction, and calculates the comprehensive perception result based on a weighted fusion algorithm, wherein each sensor feature is assigned a weight coefficient, and the weight coefficient is adaptively adjusted according to the signal-to-noise ratio of the sensor or the environmental conditions to improve the accuracy of the fusion result. The identification result, spatial position and motion trajectory of the target are obtained through fusion processing, and the environmental model of the monitoring area is reconstructed;

[0020] S5, the ground control station evaluates the target threat level or task demand according to the fusion result, generates corresponding control instructions and sends them to the underwater robot, the unmanned aerial vehicle and the buoy station to adjust their respective working modes or actions. Under the control of the control instructions, each module cooperates to execute further perception or operation tasks, and feeds back new environmental information data to the ground control station, enters the next round of data fusion and decision-making, and forms a closed-loop collaborative perception process.

[0021] Preferably, the weighted fusion algorithm in step S4 includes: normalizing the features from different sensors of acoustic, optical, magnetic and inertial, setting the weight coefficients corresponding to each sensor feature, and calculating the comprehensive perception score according to the following formula:

[0022] ,

[0023] wherein is the feature value extracted by the kth sensor, is the weight coefficient thereof and if S exceeds a preset threshold value, it is determined that the target exists and the identification is completed.

[0024] Preferably, the step S1 each node deployment and synchronization includes:

[0025] The buoy station is used for sending a synchronization signal to the underwater robot and the unmanned aerial vehicle, the underwater robot is calibrated by using the known GPS position of the buoy station and the communication time delay, or the underwater robot is navigated by using a known positioning acoustic base station previously arranged in the water area;

[0026] The unmanned aerial vehicle is used for shooting an image of the buoy station to calibrate the spatial position relationship of the unmanned aerial vehicle relative to the buoy station, so as to establish a unified spatial coordinate reference for the heterogeneous platform.

[0027] Preferably, the step S5 includes: when the fusion result detects a target, the ground control station sends an instruction to control the unmanned aerial vehicle to change the flight trajectory to approach the target in the air and reduce the height to obtain clearer images, control the underwater robot to adjust the route to track the underwater target or float to the water surface to continue monitoring, control the buoy station to switch the communication mode or release additional sensor buoys, so as to ensure the continuous tracking and information acquisition of the target in different media;

[0028] When the task demand changes, the ground control station dynamically allocates new sub-tasks to each module to realize intelligent cooperation among the multiple platforms.

[0029] Beneficial effects: the multi-modal water-air-land cross-medium cooperative perception system and method based on MEMS provided by the application has the following beneficial effects:

[0030] 1. The underwater robot, the buoy, the unmanned aerial vehicle and the ground station are cooperated to realize the three-dimensional coverage perception of the underwater, water surface, air and land interface area, and the limitation that a single platform cannot consider all media is overcome. For example, the underwater hidden target and the weak trace thereof on the water surface can be detected simultaneously, seamless tracking is realized, and the sensing blind area of the traditional monitoring at the medium interface is eliminated.

[0031] 2. The application fuses acoustic, optical, electrical, magnetic, inertial and other multi-modal information, plays the complementary advantages of various sensors, improves the accuracy of target identification and the reliability of environmental perception. When a certain sensor is invalid due to environmental interference, other modalities can still provide effective information, so that the system can still work stably under bad conditions such as noise and weak signal. In weak light or turbid environment, acoustic and magnetic sensors are mainly relied on, and in strong noise environment, vision and electric field sensors are mainly relied on, and switching is flexible.

[0032] 3、The application designs a communication network combining buoy relay and unmanned aerial vehicle relay, breaking the barrier of underwater and aerial communication isolation. The combination of underwater acoustic communication, radio and satellite link ensures real-time data upload and command issuance. Especially in emergency situations, underwater robots can quickly rendezvous with unmanned aerial vehicles through the throwing of buoys or direct floating, realizing high-speed transmission and sharing of information, and this multi-link communication mechanism effectively improves the communication bandwidth and anti-interference capability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, a brief introduction will be given to the drawings needed to be used in the embodiments or prior art description.

[0034] Figure 1 The figure is a schematic diagram of the system structure of the application.

[0035] Figure 2 The figure is a flow chart of the method of the application. DETAILED DESCRIPTION

[0036] The application will be described more clearly and completely by means of a preferred embodiment and in combination with the drawings, but the application is not limited in the scope of the described embodiments.

[0037] The application provides a multi-modal water-air-land cross-medium collaborative perception system based on MEMS, comprising:

[0038] The underwater perception module comprises an underwater robot equipped with MEMS multi-modal sensors, which is used to collect acoustic, optical, magnetic, inertial and other types of first environmental information in the water medium, and send the environmental information to the water surface relay module through the underwater communication unit.

[0039] The equipped MEMS multi-modal sensors include: an acoustic sensing unit (such as a MEMS vector hydrophone) for collecting underwater acoustic signals, an optical sensing unit (such as a low-light camera or a laser range finder) for collecting underwater images or distance data, a magnetic sensing unit (such as a three-axis MEMS magnetometer) for detecting underwater magnetic field anomalies, and an inertial sensing unit (such as a MEMS inertial measurement unit IMU) for attitude measurement. In addition, the underwater perception module also integrates bionic flow field / electric field sensors, such as multi-modal micro-sensor arrays similar to fish lateral lines, for sensing local water flow disturbance and electric field changes, and realizing the detection of weak hydrodynamic signals. The application preferably miniaturizes the size of the underwater perception module to centimeter or millimeter level, facilitating embedding in various autonomous underwater vehicles. The underwater perception module also includes underwater communication devices, such as underwater acoustic transceivers, for sending collected data to the water surface buoy through acoustic links.

[0040] The water surface relay module includes a buoy station deployed on the water surface, the buoy station having a water acoustic communication device and a radio communication device, for receiving the first environmental information sent by the underwater robot, communicating with the underwater robot through a water acoustic link, and sending the underwater environmental information to the air perception module and the ground control module through a radio link.

[0041] The buoy station is deployed at the water-air interface and has underwater and air communication capabilities. It includes a water acoustic communication module for data interaction with the underwater perception module, and a wireless communication module (such as a radio frequency transmitter-receiver, a satellite communication terminal, etc.) for communication with air or land-based platforms. The buoy station is equipped with MEMS environmental sensors (such as temperature, humidity, air pressure, and water quality sensors) to monitor local hydrological and meteorological data in real time. Through the buoy station, the relay realizes bidirectional data transmission and time synchronization between the underwater node and the air / ground network. The buoy station can also carry a solar power supply unit and a positioning module (GPS) for long-term autonomous operation and to provide a unified space-time reference for the system.

[0042] The air perception module includes a UAV, which is equipped with a MEMS multi-modal sensor for collecting second environmental information of images, infrared, and laser radar point clouds in the air, and sending the second environmental information to the ground control module through an onboard wireless communication unit or via the buoy station relay.

[0043] The UAV-mounted MEMS multi-modal sensor suite includes a high-resolution visible light camera, an infrared thermal imager, a miniature laser radar, and MEMS magnetometers, barometers, etc., for obtaining wide-area air / ground images, electromagnetic signals, and environmental parameters. The UAV establishes a high-speed data link with the buoy station or the ground control station through the onboard wireless communication module, realizing remote data transmission and command reception. The UAV can be quickly deployed in the air according to the task requirements, and can patrol a large area. When receiving target clues from the underwater module or the buoy, it can be quickly maneuvered to the target above for fine perception.

[0044] The ground control module includes a ground control station, which is used to receive multi-source environmental information sent by the buoy station and the UAV, to perform time synchronization, coordinate registration, and data fusion processing on the multi-source environmental information, to obtain comprehensive perception results and environmental state information of the target, and to send control instructions to the underwater robot, the buoy station, and the UAV to cooperatively complete the perception task.

[0045] The module includes a data processing center, a human-computer interaction terminal and necessary sensing units. The ground control station is connected with the buoy or the unmanned aerial vehicle through long-range wireless communication (cellular network, satellite link, etc.), and receives the sensing data uploaded by each module. A high-performance processor and a data fusion unit are arranged in the control station, which are used for executing a multi-source information fusion algorithm and situation analysis, and can display the results to the user or a command system. The ground station can also deploy additional sensors (such as seismic sensors, acoustic monitors, etc.) for detecting targets or environmental information on land, as an extension of the system for land sensing. The ground control station simultaneously coordinates and controls the entire system, such as sending task instructions and adjusting the working mode of each module.

[0046] The above modules jointly constitute a distributed collaborative sensing network: the underwater robot uploads the underwater multi-modal sensing data to the buoy station through the underwater acoustic link of the buoy; the buoy station performs preliminary caching and format conversion on the data, and then sends the data to the unmanned aerial vehicle or directly to the ground control station through the wireless link. The aerial image and monitoring data collected by the unmanned aerial vehicle are transmitted back to the ground control station through the wireless link. In the ground station, the multi-source data from underwater, air and land are aligned and fused according to the time stamp and coordinate to generate a unified environmental sensing model and target situation map. The units of the system of the present application can be flexibly combined and deployed, supporting multi-to-one or one-to-multi networking forms: for example, multiple underwater robots correspond to one buoy station relay, multiple buoy stations and unmanned aerial vehicles cooperatively cover a larger area, etc., having good scalability and robustness.

[0047] As shown in Figure 2 The present application provides a multi-modal water-air-land cross-medium collaborative sensing method based on MEMS, applied to a multi-modal water-air-land cross-medium collaborative sensing system based on MEMS as described above, comprising the following steps:

[0048] S1, deploying underwater robots, buoy stations, unmanned aerial vehicles and ground control station nodes in a target monitoring area, and synchronously calibrating the time and coordinates of each node to have a unified time reference and spatial reference coordinate system.

[0049] Deploying underwater sensing modules, water surface buoy stations, aerial unmanned aerial vehicles and ground control stations in a target environment. Turn on the sensors of each node, and calibrate their time synchronously through the ground control station. For example, use GPS time service or buoy station broadcast time signal to realize unified time reference of different medium nodes; use the known coordinates of the buoy station and communication ranging to calibrate the initial position of the underwater robot and the unmanned aerial vehicle, and establish a unified spatial coordinate system. If necessary, place a calibration base station (which can emit acoustic and optical beacons) at a known position in the underwater environment, to correct the cumulative error of the underwater IMU and provide a reference for the relative positioning of the aerial platform.

[0050] S2, the underwater robot collects acoustic, optical, magnetic, inertial first environmental information data in underwater medium through its MEMS multi-modal sensors, the unmanned aerial vehicle collects image, infrared, laser point cloud second environmental information data in air medium through its MEMS multi-modal sensors, the buoy station collects water surface environmental parameters as third environmental information data, and the ground control station receives or collects ground environmental information as fourth environmental information data.

[0051] Each perception module collects target and environmental information in the respective medium. During autonomous navigation, the underwater robot acquires underwater environmental data and target feature information through MEMS multi-modal sensors. For example, the acoustic unit captures underwater acoustic signals or echoes of the target, the optical / imaging unit acquires underwater video images, the magnetic sensor monitors abnormal magnetic field signals (such as magnetic anomalies generated by metal targets), the bionic flow field sensor array senses small water flow disturbances or wake, and the inertial unit records its own motion parameters. The water surface buoy station continuously monitors water quality, flow rate and weather data, and receives acoustic signals from the underwater robot. The aerial unmanned aerial vehicle cruises according to the set route, uses the camera and other sensors to scout the wide sea and air area or land surface, and once an abnormal target (such as a suspicious wave, flash or moving object on the water surface) is found, records its image, position information and spectral characteristics; at the same time, environmental parameters such as air temperature and humidity, wind speed, etc. are monitored. The sensors on the ground control station optionally collect ground vibration sound, video monitoring and other information as supplements.

[0052] S3, the underwater robot sends the first environmental information data to the buoy station through underwater acoustic communication, the buoy station packages the received first environmental information data and sends it to the ground control station and the unmanned aerial vehicle through wireless communication, and the unmanned aerial vehicle sends the second environmental information data to the ground control station through wireless link, thereby realizing cross-medium transmission and sharing of multi-source environmental information data.

[0053] After the data collected by each module is preliminarily processed, it is transmitted to other nodes for sharing in real time or periodically. The underwater robot sends acoustic, magnetic and other raw data to the buoy through underwater acoustic communication; for larger volume data (such as underwater images), it can be transmitted directly to the unmanned aerial vehicle or ground control station through radio when it rises to the water surface. The buoy station acts as a relay, packages the collected underwater data and sends it to the ground control station and the aerial unmanned aerial vehicle through wireless link. The unmanned aerial vehicle uses air-to-ground wireless link to transmit the images and videos taken and the measured data to the ground station in real time; if necessary, the unmanned aerial vehicle can also act as an air relay to transfer the buoy station data to a control center at a farther distance. During data transmission, a caching and grouping mechanism is adopted to adapt to different communication channel bandwidths and ensure timely delivery of critical event data.

[0054] S4, the ground control station fuses the received first, second, third and fourth environmental information data, including time alignment, coordinate registration, feature extraction, and calculates a comprehensive perception result based on a weighted fusion algorithm, wherein each type of sensor feature is assigned a weight coefficient, and the weight coefficient is adaptively adjusted according to the signal-to-noise ratio of the sensor or the environmental condition to improve the accuracy of the fusion result, and the identification result, spatial position and motion trajectory of the target are obtained through the fusion processing, and the environmental model of the monitoring area is reconstructed.

[0055] The fusion method of the application comprises the following steps:

[0056] S41, filtering and denoising the data of different sensors, standardizing the format, and aligning according to the time stamp. The coordinate conversion relationship established in step S1 is used to map the target position detected by the underwater sensor to the geographic coordinate system or register with the aerial image coordinate system. For example, the underwater sonar ranging data is converted into an equivalent plane position, and the corresponding area is marked on the unmanned aerial vehicle image, thereby improving the spatial correspondence of information from different sources.

[0057] S42, for a target or an event, feature parameters are extracted from each modality data. For example, the spectral features, intensity and Doppler information of underwater acoustic signals, the shape and color features of optical images, the intensity gradient of magnetic signals, and the motion pattern of inertial data. These features are normalized and used as input for subsequent fusion.

[0058] S43, the application dynamically allocates fusion weights based on the reliability of each sensing source, and estimates the state of the target through weighted combination. Specifically, let s a , s o , s m , s i respectively represent the feature quantities or discrimination indexes (normalized to dimensionless quantities) extracted by acoustic, optical, magnetic and inertial sensors for a certain target, and the corresponding weight coefficients w a , w o , w m , w i are determined according to the current environment and the signal-to-noise ratio of the sensor, and the fusion model proposed by the application calculates the comprehensive perception score S as:

[0059]

[0060] wherein each weight satisfies . In the above formula, s a represents the normalized signal feature value of the acoustic sensor output, for example, the matched filter peak value of the target acoustic signal; s o represents the target recognition confidence obtained by the optical / imaging sensor, such as the probability output by the image recognition algorithm; s mrepresents the magnetic anomaly strength index measured by the magnetic sensor; s i represents the characteristic value detected by the inertial / vibration sensor (such as the amplitude of micro-vibration caused by the target), etc. w a , w o , w m , w i are the weight coefficients of the above respective features, reflecting the reliability weight of the modal information under the current condition. For example, in the case of low light and low visibility in the underwater environment, the weight of the optical information w o is reduced, and the weight of the acoustic information w a is increased; when the target generates a strong magnetic signal, w m , etc. The weight can be adaptively adjusted by fuzzy logic or Bayesian estimation according to the signal-to-noise ratio of each sensor, the level of environmental interference, etc. The comprehensive score S calculated is used to judge the existence and attributes of the target: if S exceeds a preset threshold, it is determined that the target is detected and an alarm is triggered; at the same time, the type of the target (such as whether it is a mechanical device, a human body, a living being, etc.) and the state are further determined by combining the contribution of each modal feature.

[0061] S44, estimate the spatial position and motion trajectory of the target by fusing each source information. Collaborative positioning is performed using sonar ranging and unmanned aerial vehicle vision measurement: for example, according to the direction of the underwater acoustic target and the pixel position of the target in the aerial image, the geographical coordinates of the target are estimated by triangulation. When the target moves across the medium (such as floating out of the water or landing on the shore), the system realizes seamless connection of the trajectory according to the relay detection results of the sensors in different stages. The extended Kalman filter (EKF) or particle filter algorithm is used to fuse multi-source observations at different times, continuously correct the target state estimation, and improve the stability and accuracy of tracking.

[0062] S45, in addition to the target, the perception data of the environment (such as underwater terrain, obstacle distribution, water quality parameters, weather data, etc.) are fused and modeled. The underwater sonar survey data and unmanned aerial vehicle aerial image are superimposed to construct a three-dimensional digital model of the water-land interface area; combined with the multi-point water quality sensing data of the buoy and the unmanned aerial vehicle remote sensing information, a distribution map of pollutant diffusion or a marine environmental parameter field is generated. The fused environmental model can be used for decision support and situation visualization.

[0063] S5, the ground control station evaluates the threat level or task demand of the target according to the fusion results, generates corresponding control instructions, and sends them to the underwater robot, unmanned aerial vehicle, and buoy station to adjust their respective working modes or actions. Under the control of the control instructions, each module cooperates to execute further perception or operation tasks, and feeds back new environmental information data to the ground control station, enters the next round of data fusion and decision-making, forming a closed-loop collaborative perception process.

[0064] The fused situation awareness results are presented in real time on the human-machine interface of the ground control station, including the detected target position, type, threat assessment, and environmental map information. The operator or higher command system makes decision instructions accordingly. The system supports automated feedback control: for example, when a suspicious target is detected, the ground control station can autonomously plan a new unmanned aerial vehicle route to the target area for low-altitude hovering for evidence, or instruct a nearby underwater robot to deploy sensors around the target to form a siege; in environmental monitoring applications, the system adjusts the sampling frequency or movement position of each node according to the monitoring results to obtain more detailed data. Each subsystem acts according to the cooperative strategy to form a closed loop of perception-decision-re-perception, continuously improving the adaptability and perception accuracy of the dynamic environment.

[0065] Embodiment 1

[0066] Referring to Figure 1 The MEMS-based multi-modal cross-medium cooperative perception system provided in the embodiment includes an underwater robot, a buoy station, an unmanned aerial vehicle, and a ground control station. The underwater robot (for example, an autonomous underwater vehicle AUV) carries multiple MEMS sensor modules and a communication module, which are used to perform environmental perception and target detection tasks underwater. Specifically, the underwater robot is provided with: an acoustic sensor (such as a micro hydrophone array) for receiving underwater acoustic signals, an optical sensor (such as a low-illumination camera or a blue-green laser imager) for acquiring underwater images and optical ranging information, a magnetic sensor (such as a three-axis MEMS magnetometer) for detecting underwater magnetic field changes, and an inertial sensor (a MEMS inertial measurement unit including an accelerometer and a gyroscope) for measuring the motion attitude and navigation parameters of the robot itself. The above sensors all realize data acquisition and preprocessing through the embedded controller on the underwater robot. The head or side of the underwater robot is also integrated with a bionic multi-modal sensing unit (not separately marked, which can be regarded as an extension of the acoustic sensor), which is similar in structure to the lateral line sensory organ of fish and can detect local flow velocity gradients and weak electric field signals, thereby perceiving the motion of adjacent objects or bioelectric signals. The bionic multi-modal sensing unit is made of MEMS sensing technology, has small volume and high sensitivity, and plays an important role in improving underwater near-field perception effect. In addition to the perception devices, the underwater robot is also installed with a underwater acoustic communication module for bidirectional data communication with the buoy station. The underwater acoustic communication module includes an underwater loudspeaker and a hydrophone, as well as modulation and demodulation and signal processing circuit, which can convert digital data into acoustic signals for underwater transmission and receive acoustic instructions from the buoy station.

[0067] The buoy station is deployed on the water surface of the task sea area and kept relatively fixed by mooring anchors or self-holding positioning. The buoy station is internally configured with a wireless communication module and a corresponding underwater acoustic communication interface for connecting the underwater and air networks. In the embodiment, the wireless communication module of the buoy station includes a vertically erected VHF antenna and a satellite communication terminal, which can realize remote radio communication with the ground control station or the unmanned aerial vehicle. The lower part of the buoy station is connected with an underwater acoustic transceiver (corresponding to the underwater acoustic communication module) for exchanging data and commands with underwater robots through sound waves. The buoy station is internally provided with a single-chip microcomputer or an industrial computer, which is responsible for protocol conversion and data forwarding, and simultaneously collects data of environmental sensors (such as a water temperature meter, a salinity meter, and an accelerometer for monitoring waves) carried by the buoy station. The buoy station has wind and wave resistance design and low-power power management, and can be stably operated for a long time, and can keep communication with underwater and air nodes through an intermittent wake-up mechanism.

[0068] The unmanned aerial vehicle is a fixed-wing or multi-rotor unmanned aerial vehicle, which undertakes the roles of air patrol and communication relay in the embodiment. The unmanned aerial vehicle is installed with various sensors such as visible light cameras, infrared sensors, and laser radars, which are used to obtain target information on the ground and water surface from the air. The visible light camera can shoot real-time high-definition videos for target identification and scene monitoring; the infrared sensor can detect heat source targets at night or in bad weather; and the laser radar can obtain three-dimensional point cloud data of the terrain and objects to assist in ranging and mapping. The unmanned aerial vehicle also carries auxiliary sensors (not marked in the figure) such as micro-magnetometers and barometric altimeters to enhance its navigation and perception capabilities. The on-board communication module of the unmanned aerial vehicle supports various wireless links such as digital data links, radios, or 4G / 5G cellular communication modules to communicate with the buoy station and the ground control station. When performing tasks, the unmanned aerial vehicle can fly autonomously according to the predetermined route, and can accept remote control instructions from the ground control station for path adjustment or hovering point monitoring. The unmanned aerial vehicle 3 can also serve as a communication node, which relays and forwards the buoy data when the direct communication between the buoy station and the ground control station is not smooth due to long distance, so as to realize dynamic self-organization of the network.

[0069] The ground control station can be a fixed control center on the shore, a mobile control station on a vehicle, or a backpacked field terminal. In the embodiment, the ground control station is equipped with a high-performance computer as a data fusion processing unit, and a terminal (such as a monitor screen, a console, etc.) for display and interaction. The ground control station maintains connection with the buoy station / unmanned aerial vehicle through a remote communication link, receives front-end sensing data, and sends control instructions. The data fusion processing unit runs the data processing and fusion software described in the method of the present application, and can analyze and calculate multi-source data to output unified sensing results. The control station operator can view the real-time constructed environment map, target information list, etc. through the terminal, and can issue high-level tasks to the system (such as re-planning the unmanned aerial vehicle route, adding underwater robots, etc.). The ground control station itself can also connect to other external databases or sensor networks (such as a sonar array laid on the shore, a video monitoring system), and include their information in the fusion range to further expand the system's sensing capability.

[0070] The collaborative sensing method of the system in the embodiment is described below. Initially, the ground control station issues task parameters to each node and performs system calibration and synchronization. Then the system enters a cyclic operation, including four main stages of sensing, communication, fusion, and feedback.

[0071] (1) Information sensing stage: the underwater robot patrols underwater or monitors at a fixed point according to the established route, and turns on acoustic, optical, magnetic, inertial, etc. sensors to detect the underwater environment. Once the underwater acoustic sensor captures suspicious target noise or echo signal, the underwater robot immediately records the signal characteristics and estimates the target direction; at the same time, the magnetic sensor detects whether there is an abnormal magnetic field change to determine whether the target contains metal components, etc. If the bionic side line sensor array of the underwater robot senses local flow disturbance (such as wake flow caused by moving targets), it will jointly determine the existence of the target and its movement direction in combination with acoustic signals. When the unmanned aerial vehicle patrols in the air, it monitors the water surface in real time through the visible light camera, such as finding unknown ripples, floating objects or signs of targets exposed to the water surface (such as periscopes, boats, etc.), and immediately captures images and locates the target relative to itself. At the same time, the infrared sensor scans the water surface temperature distribution to find possible heat trace signals. The laser radar periodically measures the distance data of the ground and sea surface to construct the environment model and assist target detection. The buoy station continuously monitors the hydrological and meteorological data, and when the underwater robot detects a target, the buoy station also receives the initial report information of the target sent by the robot through underwater sound.

[0072] (2) Data transmission stage: each platform shares the important information perceived through the network. The underwater robot uses the underwater acoustic communication module to send the detected target direction, acoustic fingerprint features and other information to the buoy station in the form of acoustic signals; if large data volume information such as images cannot be sent in real time through underwater acoustic, it is temporarily stored in the underwater robot, and then transmitted through high-speed wireless when the conditions allow (such as the robot rising or approaching the buoy). After receiving the underwater data packet, the buoy immediately relays the data to the ground control station through the wireless communication module; for the case of needing cooperation with the unmanned aerial vehicle, the buoy can also broadcast the underwater target information to the unmanned aerial vehicle cruising in the sky. After obtaining the target clues from the underwater or buoy station, the unmanned aerial vehicle packages the corresponding region image captured by the camera, infrared hot spot and other data, and sends them back to the ground control station through air-to-ground link. At the same time, the unmanned aerial vehicle also acts as a relay between the buoy and the ground station: when the buoy is far away from the shore base station or direct communication is not smooth, the unmanned aerial vehicle flies over the buoy to receive its wireless signal and forwards it to the ground control station. The whole transmission stage adopts acknowledgement mechanism and cache retransmission strategy to ensure that the unreliability of cross-medium link does not lose key data. For the underwater acoustic channel with limited bandwidth, the system sets information priority, such as binary decision of target existence and high-priority alarm information transmission priority; detailed original sensing data is transmitted when the bandwidth is sufficient, so as to realize urgent transmission and hierarchical transmission.

[0073] (3) Data fusion stage: after the ground control station collects heterogeneous data from multiple channels such as underwater, buoy, unmanned aerial vehicle and other channels, it immediately starts the multi-sensor information fusion algorithm (see the description of data fusion steps in the foregoing invention content). First, time synchronization and space registration of multi-source data are performed: the time stamp of each sensing data is corrected using the buoy timing, and the geometric relationship of each observation in the unified reference system is calculated using the buoy position information and the unmanned aerial vehicle pose. For example, for the target direction reported by the underwater sonar, the rough geographic position of the target can be calculated in the ground station combined with the buoy station and underwater robot position, and then the corresponding region is searched in the unmanned aerial vehicle image to verify the target track. Then the features from acoustic, optical, magnetic and other sensors are extracted, and the adaptive weighted fusion method is used to calculate the comprehensive perception score S. If S exceeds the threshold, the system determines that the target exists, and determines the target category (such as whether it is a submarine, a ship, a living being, etc.) combined with each modality information, and finally marks the target positioning result and attribute information on the map. At the same time, the ground control station fuses the environmental detection data returned by the underwater robot (such as the underwater topographic profile drawn by the sonar depth finder) and the aerial image of the coastline shot by the unmanned aerial vehicle, to generate a three-dimensional environmental model; fuse the water quality parameters of the buoy and the water surface image of the unmanned aerial vehicle to evaluate the range of water pollution or algal bloom. All these fusion results are presented in the form of map overlay or list on the display interface of the terminal.

[0074] (4) Coordinated decision and action stage: When the fusion result shows that the target is successfully detected and confirmed to have the necessity of further tracking or disposal, the system will enter the coordinated response stage. The ground control station automatically generates a new task plan according to the position and moving direction of the target, and issues it to the relevant platform for execution. For example, if the target is moving in a certain direction, the ground control station instructs another underwater robot near the corresponding area to intercept detection, or dispatches a standby unmanned aerial vehicle to the pre-judgment position in front of the target to form a multi-angle monitoring encirclement. If the target has floated to the water surface, the system may make the unmanned aerial vehicle lower the flight height and use the loudspeaker carried to issue a warning or deploy a marker device; if the target lands on the shore and enters the land area, the camera network or unmanned vehicle on land is instructed to continue tracking. The units of the present application form a clear division of labor through the coordination of the ground station: some are responsible for continuous perception (such as the original underwater robot continues to track underwater), some are responsible for supplementary perception (such as the unmanned aerial vehicle changes the optical sensor mode for night vision), some are responsible for communication relay (such as the buoy and an unmanned aerial vehicle maintain the link), and others are responsible for executing specific actions (such as deploying on-site markers). Throughout the process, the ground station continuously receives new feedback data from each node, updates the fusion result in real time and adjusts the coordination strategy, realizes closed-loop control, and ends the task.

[0075] In this embodiment, through the above system structure and method process, cross-media cooperative detection is successfully realized: a certain simulated target (test target with sound source) detected by the underwater robot is accurately positioned and identified with the cooperation of the buoy station and the unmanned aerial vehicle. Test results show that multi-modal fusion perception greatly improves the detection reliability and accuracy compared to single sensing mode; the system still has a high detection probability for weak target signals in complex environments, while effectively reducing the false alarm rate. This verifies the feasibility and superiority of the present application scheme.

[0076] Embodiment 2: Improvement of cross-domain technology fusion

[0077] The embodiment is based on the embodiment 1, and further shows the application scene of the fusion of cross-domain technologies to improve the performance. For the underwater-air cooperative positioning accuracy problem, the application introduces a cross-medium visual positioning technology: an upward LED flash beacon is added to the underwater robot, and a high-sensitivity downward camera is added to the unmanned aerial vehicle. When the underwater robot needs to accurately position itself or provide a reference for the unmanned aerial vehicle, the LED is triggered to emit a blue light signal of a specific frequency; the unmanned aerial vehicle captures the flashing light spot on the water surface in the air, determines the pixel coordinates thereof through image processing, and inversely calculates the absolute position of the underwater robot in combination with the self-GPS positioning. This method uses the concept of underwater optical communication, transplants the optical guidance technology (commonly used for underwater docking) to cross-medium positioning, and makes up for the defects of limited underwater acoustic positioning accuracy and the inability of satellite signals to penetrate water bodies. Experiments show that under clear water surface conditions, the positioning error between the UUV and the UAS can be controlled within sub-meter level, greatly improving the accuracy of cooperative navigation and positioning.

[0078] For example, for the cross-medium communication bottleneck, the application combines a sound-light communication conversion cross-border technology: a water surface laser vibration sensor is provided on the buoy station, and is aligned with a specific area of the water surface. When the underwater robot cannot transmit data at a high speed through the conventional underwater acoustic channel, it can send a high-frequency acoustic signal to modulate information towards the water surface, and the bit stream is transmitted through the water surface micro-vibration; the laser vibration sensor on the buoy receives and demodulates the water surface vibration signal, restores it to digital data, and sends it to the ground control station through radio. This acoustic-optical combined communication means uses the air-water interface as a carrier, and can realize cross-medium high-speed communication without physical contact, which belongs to the innovative application of the existing optical communication technology in the marine field. Tests show that under the condition of dozens of meters of water depth, this method can realize the data transmission rate of kbps level, and can be a powerful supplement to the traditional underwater acoustic communication.

[0079] It should be noted that the specific module configurations and process steps described in the above embodiments can be changed and optimized according to actual application requirements. For example, different sensor types, number configurations, and data fusion algorithm implementation details can be changed by those skilled in the art without departing from the spirit of the application. The protection scope of the application is not limited to the embodiments, and any obvious replacement or combination made without departing from the inventive concept should be regarded as falling within the protection scope of the application.

Claims

1. A multi-modal water-air-land cross-medium cooperative perception system based on MEMS, characterized in that, The application relates to a multi-source multi-modal underwater, air and ground integrated perception system. The underwater perception module comprises an underwater robot equipped with MEMS multi-modal sensors, which is used for collecting acoustic, optical, magnetic and inertial multi-type first environment information in a water medium and transmitting the environment information to a water surface relay module through an underwater communication unit. The water surface relay module comprises a buoy station arranged on the water surface, which has underwater acoustic communication devices and radio communication devices, is used for receiving the first environment information transmitted by the underwater robot, communicates with the underwater robot through an underwater acoustic link, and transmits the underwater environment information to the air perception module and the ground control module through a radio link. The air perception module comprises a UAV, which is equipped with MEMS multi-modal sensors for collecting image, infrared and laser radar point cloud second environment information in the air, and transmitting the second environment information to the ground control module or relaying transmission through the buoy station. The ground control module comprises a ground control station, which is used for receiving multi-source environment information transmitted by the buoy station and the UAV, performing time synchronization, coordinate registration and data fusion processing on the multi-source environment information, obtaining comprehensive perception results and environment state information of a target, and sending control instructions to the underwater robot, the buoy station and the UAV to cooperatively complete a perception task.

2. The multi-modal water-air-land cross medium cooperative perception system based on MEMS according to claim 1, characterized in that, The MEMS multi-modal sensors carried by the underwater robot comprise a micro hydrophone array for detecting underwater acoustic signals, a micro optical imaging sensor for collecting underwater images or optical ranging data, a MEMS magnetometer for detecting underwater magnetic field anomalies, a MEMS inertial measurement unit for measuring the attitude and motion parameters of the underwater robot, and a bionic flow field and electric field sensor array for sensing local water flow disturbance and underwater electric field signals.

3. The multi-modal water-air-land cross medium cooperative perception system based on MEMS according to claim 1, characterized in that, The buoy station has wireless / underwater acoustic dual communication modules, wherein the wireless communication module comprises at least one remote radio device to realize bidirectional data communication with the shore ground control station or the UAV, and the underwater acoustic communication module comprises a transceiving transducer to realize bidirectional underwater acoustic communication with the underwater robot; the buoy station further comprises environment sensors for measuring third environment information of the water surface or the water body, and can transmit the third environment information together with the first environment information and the second environment information.

4. The multi-modal water-air-land cross medium cooperative perception system based on MEMS according to claim 1, characterized in that, The MEMS multi-modal sensors carried by the UAV comprise a visible light camera, an infrared imager and a micro laser radar for obtaining multi-spectral images and distance data of a wide area in the air and on the ground, and further comprise a micro magnetometer and a barometer for assisting navigation and target detection; the onboard wireless communication unit of the UAV supports a self-organizing relay function, can act as a communication relay node between the buoy station and the ground control station, and can expand communication coverage and improve bandwidth.

5. The multi-modal water-air-land cross medium cooperative perception system based on MEMS according to claim 1, characterized in that, The ground control station comprises: a data processing and fusion unit for fusion calculation of multi-source environmental information data from the underwater robot, the buoy station and the unmanned aerial vehicle; a human-computer interaction terminal for displaying the fused environmental model, target information and receiving user instructions; a communication interface module for interacting data with external systems through wireless or wired networks; the data processing and fusion unit is configured to execute a multi-sensor data fusion algorithm, including time synchronization, spatial coordinate conversion, feature extraction and weighted fusion decision steps, to generate a three-dimensional situation awareness result. 6.A multi-modal water-air-land cross-medium collaborative perception method based on MEMS, characterized in that, The application is applied to a multi-modal water-air-land cross-medium collaborative perception system based on MEMS, comprising the following steps: S1, deploying underwater robots, buoy stations, unmanned aerial vehicles and ground control station nodes in the target monitoring area, synchronously calibrating the time and coordinates of each node to have a unified time reference and spatial reference coordinate system; S2, the underwater robot collects acoustic, optical, magnetic and inertial first environmental information data in the underwater medium through its MEMS multi-modal sensor, the unmanned aerial vehicle collects image, infrared and laser point cloud second environmental information data in the air medium through its MEMS multi-modal sensor, the buoy station collects water surface environmental parameters as third environmental information data, and the ground control station receives or collects ground environmental information as fourth environmental information data; S3, the underwater robot sends the first environmental information data to the buoy station through underwater acoustic communication, the buoy station packages the received first environmental information data and sends it to the ground control station and the unmanned aerial vehicle through wireless communication, and the unmanned aerial vehicle sends the second environmental information data to the ground control station through a wireless link, thereby realizing cross-medium transmission and sharing of multi-source environmental information data; S4, the ground control station performs fusion processing on the received first, second, third and fourth environmental information data, including time alignment, coordinate registration, feature extraction, and calculating a comprehensive perception result based on a weighted fusion algorithm, wherein each sensor feature is assigned a weight coefficient, the weight coefficient is adaptively adjusted according to the signal-to-noise ratio of the sensor or the environmental condition to improve the accuracy of the fusion result, and the identification result, spatial position and motion trajectory of the target are obtained through fusion processing, and an environmental model of the monitoring area is reconstructed; S5, the ground control station evaluates the target threat level or task demand according to the fusion result, generates corresponding control instructions and sends them to the underwater robot, the unmanned aerial vehicle and the buoy station to adjust their working modes or actions, under the control of the control instructions, each module cooperatively executes further perception or operation tasks, and new environmental information data is fed back to the ground control station, entering the next round of data fusion and decision making, forming a closed-loop collaborative perception process.

7. The multi-modal water-air-land cross medium cooperative perception method based on MEMS according to claim 6, characterized in that, The weighted fusion algorithm in step S4 comprises: normalizing the features from different sensors such as acoustic, optical, magnetic and inertial, setting the weight coefficients corresponding to each sensor feature, and calculating the comprehensive perception score according to the following formula: ， wherein the feature value extracted for the kth sensor, is a weight coefficient thereof and if S exceeds a preset threshold value, the target is determined to exist and the recognition is completed.

8. The multi-modal water-air-land cross medium cooperative perception method based on MEMS according to claim 6, characterized in that, The node deployment and synchronization in step S1 comprises: The synchronization signal is sent to the underwater robot and the unmanned aerial vehicle by the buoy station, the underwater robot is calibrated by the known GPS position of the buoy station and the communication delay, or the underwater robot is navigated by the known positioning acoustic base station previously arranged in the water area; The image of the buoy station is taken by the unmanned aerial vehicle to calibrate the spatial position relationship of the unmanned aerial vehicle relative to the buoy station, thereby establishing a unified spatial coordinate reference for the heterogeneous platform.

9. The multi-modal water-air-land cross medium cooperative perception method based on MEMS according to claim 6, characterized in that, When the fusion result detects the target, the ground control station sends instructions to control the unmanned aerial vehicle to change the flight trajectory to approach the target in the air and reduce the height to obtain clearer images, control the underwater robot to adjust the route to track the underwater target or float to the water surface to continue monitoring, and control the buoy station to switch the communication mode or release additional sensor buoys, so as to ensure the continuous tracking and information acquisition of the target in different media; When the task demand changes, the ground control station dynamically allocates new sub-tasks to each module to realize intelligent cooperation among multiple platforms.

Images