A multi-modal perception method and system based on cooperation of unmanned aerial vehicle and unmanned surface vehicle

Abstract

This invention discloses a multimodal perception method and system based on UAV-UAV collaborative sensing. The method includes the following steps: the UAV performs electromagnetic spectrum scanning and edge image detection to obtain multi-source detection results; a target search area is generated based on the UAV multi-source detection results; the UAV performs a visual verification task within the target search area and collects UAV multimodal observation data; electromagnetic data from the UAV side is acquired, and the UAV multi-source detection results, UAV multimodal observation data, and UAV side electromagnetic data are spatiotemporally aligned to obtain aligned data; based on the aligned data, UAV side visual features, UAV side visual features, and electromagnetic features are constructed and fused; a target recognition result is obtained based on the fused features; and the target position, velocity, and trajectory are continuously updated based on the target recognition result. This invention enables air-sea collaborative sensing and improves the monitoring range of the sea area.

Description

Technical Field

[0001] This invention belongs to the fields of unmanned system collaborative perception, intelligent maritime monitoring and multi-source information fusion technology, specifically involving a multimodal perception method and system based on UAV-unmanned surface vessel collaboration. Background Technology

[0002] With the development of the marine economy and the increasing frequency of maritime activities, maritime security monitoring and target identification have become important technical requirements for marine management. Currently, common maritime monitoring methods mainly include shore-based radar monitoring, shipborne monitoring equipment, and manual patrols. However, these traditional methods generally suffer from limited monitoring range, insufficient real-time performance, and high labor costs.

[0003] In recent years, drone and unmanned surface vessel (USV) technologies have matured, providing new solutions for maritime monitoring. Drones offer advantages such as high mobility, wide monitoring range, and flexible deployment; USVs, on the other hand, have the ability to cruise the sea surface for extended periods, enabling them to perform continuous monitoring tasks. Therefore, the coordinated use of drones and USVs can achieve integrated air and sea monitoring.

[0004] However, existing technologies still have the following problems in practical applications: (1) Insufficient reliability of single sensing methods. For example, photoelectric camera systems have poor recognition performance under conditions of fog, night, or strong reflection on the sea surface. (2) Difficulty in data fusion from multiple sources. Electromagnetic signals, photoelectric images, and infrared data have significant differences in time, space, and data structure. (3) Lack of intelligent triggering mechanism. Most systems currently use continuous video monitoring, which leads to high consumption of computing resources. (4) Difficulty in continuously updating target information. Targets move frequently in complex marine environments, and traditional methods are difficult to achieve stable tracking. (5) Imperfect air-sea coordinated scheduling mechanism, resulting in low system efficiency.

[0005] Therefore, a new technical solution is needed to realize the collaboration between UAVs and unmanned surface vessels, multimodal data fusion, and intelligent triggering and recognition mechanisms. Summary of the Invention

[0006] This invention aims to address the shortcomings of existing technologies and provides the following solutions: A multimodal perception method based on UAV-Unmanned Surface Vessel (USV) cooperation includes the following steps: The unmanned surface vessel performs electromagnetic spectrum scanning and edge image detection to obtain multi-source detection results for the unmanned surface vessel; A target search area is generated based on the multi-source detection results of the unmanned surface vessel; The UAV performs a visual verification task in the target search area and collects multimodal observation data from the UAV. Acquire unmanned surface vessel (USV) side electromagnetic data, and perform spatiotemporal unified alignment of the USV multi-source detection results, the UAV multimodal observation data, and the USV side electromagnetic data to obtain aligned data; Based on the aligned data, unmanned surface vessel side visual features, unmanned aerial vehicle side visual features and electromagnetic features are constructed and fused. Target recognition results are obtained based on the fused features. Based on the target recognition results, the target's position, velocity, and trajectory are continuously updated.

[0007] Preferably, the method for obtaining the multi-source detection results of the unmanned surface vessel includes: The unmanned surface vessel performs electromagnetic spectrum scanning on the target sea area, extracts the frequency, power, bandwidth, duration and incident direction of candidate signals, and constructs the candidate signal feature vector; An anomaly score is calculated based on the candidate signal feature vector. When the anomaly score meets the preset anomaly triggering condition, it is determined to be an electromagnetic anomaly event, and electromagnetic anomaly event information is obtained. The unmanned surface vessel (USV) uses its onboard visible light camera and / or infrared thermal imaging equipment to acquire sea surface images, performs target detection in the edge computing unit, and outputs the target category, target bounding box, detection confidence score and timestamp to obtain the USV's side visual target detection information. The multi-source detection results of the unmanned surface vessel include the electromagnetic anomaly event information and the unmanned surface vessel side visual target detection information.

[0008] Preferably, the method for generating the target search region includes: When the multi-source detection results of the unmanned surface vessel only contain electromagnetic anomaly event information, the target position is estimated based on the position information and electromagnetic signal direction angle of the unmanned surface vessel, combined with empirical distance, and the target search area is constructed based on the azimuth error and distance error. When the multi-source detection results of the unmanned surface vessel (USV) only contain the USV's side visual target detection information, the center point of the detection box is projected onto the geographic coordinate system based on the USV's camera intrinsic parameters, camera extrinsic parameters, and the center pixel coordinates of the detection box to obtain the estimated target location and construct the target search area. When the multi-source detection results of the unmanned surface vessel (USV) simultaneously include electromagnetic anomaly event information and USV side-view target detection information, the estimated position obtained from the electromagnetic anomaly and the estimated position obtained from the visual detection are weighted and fused, and the target search area is constructed based on the fused position.

[0009] Preferably, the method for obtaining the UAV multimodal observation data includes: The drone flies to the corresponding airspace based on the received target search area; The UAV uses its onboard visible light camera and / or infrared thermal imaging device to acquire images of the target search area; The UAV performs target detection on the acquired images in the edge computing unit and outputs the target category, detection box, detection confidence and detection timestamp as the UAV's multimodal observation data.

[0010] Preferably, the method for obtaining the aligned data includes: The electromagnetic data of the unmanned surface vessel is acquired, and the multi-source detection results of the unmanned surface vessel, the multi-modal observation data of the unmanned aerial vehicle, and the electromagnetic data of the unmanned surface vessel are synchronized in time so that the time difference between the three meets the preset time synchronization tolerance. The target positions corresponding to the unmanned surface vessel's side-vision target detection information and the target positions corresponding to the UAV's multimodal observation data are respectively transformed based on the camera intrinsic parameters, camera extrinsic parameters, and scale factors of each platform, and uniformly mapped to the world coordinate system to achieve spatial registration; Based on the time-synchronized and spatially registered data, target association is performed to determine whether detection results from different sources correspond to the same target, thus obtaining the aligned data.

[0011] Preferably, the method for obtaining the target recognition result includes: Unmanned surface vessel (USV) side-view features are extracted from the aligned data. The USV side-view features include a first target shape feature, a first texture feature, a first detection feature, and a first target detection confidence level. The UAV side-view features are extracted from the aligned data, including second target shape features, second texture features, second detection features, and second target detection confidence. Electromagnetic features are extracted from the aligned data, including the frequency, power, bandwidth, duration, and incident direction characteristics of electromagnetic anomalies. Weights are assigned to the unmanned surface vessel side-view features, the unmanned aerial vehicle side-view features, and the electromagnetic features, and a fused feature vector is constructed. The probability of the target's existence is calculated based on the fused feature vector. When the probability of the target's existence meets a preset target determination threshold, the target is determined to be valid, and the target recognition result is obtained.

[0012] The present invention also provides a multimodal perception system based on UAV-UAV collaboration, the system applying the above-mentioned method, including: UAV detection module, search area generation module, UAV detection module, data alignment module, target recognition module and update module; In the unmanned surface vessel detection module, the unmanned surface vessel is used to perform electromagnetic spectrum scanning and edge image detection to obtain the multi-source detection results of the unmanned surface vessel. The search area generation module generates a target search area based on the multi-source detection results of the unmanned surface vessel. In the UAV detection module, the UAV is used to perform a visual verification task in the target search area and to collect multimodal observation data of the UAV. The data alignment module is used to acquire the electromagnetic data of the unmanned surface vessel (USV) and to perform spatiotemporal unified alignment of the USV multi-source detection results, the UAV multimodal observation data, and the USV side electromagnetic data to obtain aligned data. The target recognition module constructs unmanned surface vessel side visual features, unmanned aerial vehicle side visual features, and electromagnetic features based on the aligned data, and performs feature fusion to obtain the target recognition result based on the fused features. The update module continuously updates the target's position, speed, and trajectory based on the target recognition results.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Achieve air-sea collaborative perception and improve the monitoring range of the sea area; (2) Improve the target recognition accuracy through multimodal data fusion; (3) Reduce the system's computational burden through electromagnetic triggering visual verification mechanism; (4) Achieve continuous target tracking and status updates; (5) The system architecture of this invention has good scalability. Attached Figure Description

[0014] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 In this embodiment, as Figure 1 As shown, a multimodal perception method based on UAV-Unmanned Surface Vessel (USV) cooperation includes the following steps: S1. The unmanned surface vessel (USV) performs electromagnetic spectrum scanning and edge image detection to obtain multi-source detection results.

[0019] The method for obtaining multi-source detection results of unmanned surface vessels (USVs) includes: the USV performs electromagnetic spectrum scanning on the target sea area, extracts the frequency, power, bandwidth, duration, and incident direction of candidate signals, and constructs candidate signal feature vectors; anomaly scores are calculated based on the candidate signal feature vectors, and when the anomaly score meets the preset anomaly triggering conditions, it is determined to be an electromagnetic anomaly event, and electromagnetic anomaly event information is obtained; the USV uses its onboard visible light camera and / or infrared thermal imaging equipment to acquire sea surface images, performs target detection in the edge computing unit, and outputs the target category, target bounding box, detection confidence, and timestamp to obtain USV side-vision target detection information; the multi-source detection results of the USV include electromagnetic anomaly event information and USV side-vision target detection information.

[0020] In this embodiment, during the cruise, the unmanned surface vessel (USV) continuously scans signals in the target sea area using electromagnetic spectrum sensing equipment to extract electromagnetic characteristics such as frequency, power, bandwidth, duration, and incident direction. On the other hand, it uses the onboard visible light camera and / or infrared thermal imaging equipment to collect sea surface images and performs target detection on the USV's edge computing unit.

[0021] Let the first i The feature vector of each candidate signal is represented as: in, The center frequency of the signal. For signal power, For signal bandwidth, For signal duration, The angle of incidence of the signal. T This indicates the matrix transpose.

[0022] The results of the unmanned surface vessel side image detection are recorded as follows: in, Indicates the first j The detection results of each target Indicates the target category, Represents the target bounding box. Indicates the detection confidence level. Represents a timestamp.

[0023] To determine whether a candidate signal is an anomalous target signal, an anomaly scoring function is constructed: in, , , , and Represents the weighting coefficient, and w 1+ w 2+ w 3+ w 4+ w 5 = 1; This represents the normalized result of the power. This represents the normalized result of the bandwidth. The normalized result representing the duration. The frequency deviation is indicated by the frequency rule deviation, which characterizes whether the signal falls into a preset abnormal frequency band or deviates from a legal communication frequency band. Indicators representing directional stability or directional clustering.

[0024] The candidate signal is determined to be an abnormal electromagnetic event when the following formula is satisfied: in, This indicates the threshold for triggering an anomaly.

[0025] Meanwhile, the unmanned surface vessel's side image detection module outputs the visual target confidence score: When satisfied At that time, it was determined that the unmanned surface vessel had detected a suspected target within its current field of view; among them, This represents the anomaly trigger threshold for image detection from the perspective of an unmanned surface vessel.

[0026] Therefore, candidate anomaly events on the unmanned surface vessel (USV) side no longer originate solely from electromagnetic sources, but can also originate from image detection, or be triggered by a combination of both. The unified event representation is as follows: in, Indicates electromagnetic anomaly information. This indicates the image detection information of the unmanned surface vessel. Indicates the current position of the unmanned surface vessel. Indicates the event timestamp.

[0027] S2. Generate the target search area based on the multi-source detection results of the unmanned surface vessel.

[0028] When the unmanned surface vessel (USV) detects an abnormal event, the system generates a target search area based on the event type. Methods for generating the target search area include: (1) Scenario 1: Triggered by electromagnetic anomaly: When the multi-source detection results of the unmanned surface vessel (USV) only contain information on electromagnetic anomalies, the target position is estimated based on the USV's position information and electromagnetic signal direction angle, combined with empirical distance, and the target search area is constructed based on the azimuth error and distance error.

[0029] Specifically, if the abnormal event is mainly triggered by electromagnetic detection, then the location of the unmanned surface vessel will be used as the basis for the action. and electromagnetic angle of arrival Estimated target location: in, This represents the target distance estimated based on signal strength, historical experience, environmental models, or prior target distribution. The search area is then constructed. in, This represents the scale parameter / threshold parameter of the electromagnetic search region. p Represents the location point / spatial coordinate variable.

[0030] (2) Scenario 2: Triggered by unmanned surface vessel image detection: When the multi-source detection results of the unmanned surface vessel (USV) only contain the USV's side-view target detection information, the center point of the detection box is projected onto the geographic coordinate system based on the USV's camera intrinsic parameters, camera extrinsic parameters, and the center pixel coordinates of the detection box to obtain the estimated target location and construct the target search area.

[0031] Specifically, if the abnormal event is mainly triggered by the unmanned surface vessel's visual detection, then based on the unmanned surface vessel's image imaging model and platform pose, the center point of the detection box is projected onto the geographic coordinate system to obtain the estimated target location: in, This represents the spatial coordinates of the target point in the camera coordinate system. l This represents the scale factor, used to back-project image pixels onto their actual spatial locations in the camera coordinate system. This represents the intrinsic parameter matrix of the unmanned surface vessel's side camera. This represents the rotation matrix from the unmanned surface vessel's side camera coordinate system to the world coordinate system. This represents the translation vector from the unmanned surface vessel's side camera coordinate system to the world coordinate system. u Indicates the center pixel coordinates of the target bounding box. This indicates the estimated geographic coordinates of the target.

[0032] Constructing a visual search area: in, This indicates the visual estimation of the target location. This represents the scale parameter / threshold parameter of the visual search area.

[0033] (3) Case 3: Triggered by a combination of electromagnetic and image signals: When the multi-source detection results of the unmanned surface vessel (USV) simultaneously include electromagnetic anomaly event information and USV side-view target detection information, the estimated position obtained from the electromagnetic anomaly and the estimated position obtained from the visual detection are weighted and fused, and the target search area is constructed based on the fused position.

[0034] Specifically, when the unmanned surface vessel simultaneously obtains electromagnetic anomaly results and image detection results, the estimated positions of the two targets are fused: in, m This represents the fusion weights. This is then used to construct the joint search region. It is then sent to a drone to perform a verification task.

[0035] After detecting an abnormal electromagnetic event, the system combines the unmanned surface vessel's current position with the electromagnetic signal to estimate the area where the target might exist. Let the unmanned surface vessel's position at time... t The geographical location is: in, , These represent the two-dimensional position coordinate components.

[0036] The incident angle of the abnormal signal is The estimated target location is: Considering sea surface propagation error, direction finding error, and platform positioning error, an elliptical or sector-shaped search area is constructed. , can be represented as: Where Σ represents the target region covariance matrix, used to describe the orientation error and distance error; c This represents the regional scale parameter. The system will... This serves as the search area for the UAV visual verification task.

[0037] S3. The UAV performs visual verification tasks in the target search area and collects multimodal observation data from the UAV.

[0038] The method for obtaining UAV multimodal observation data includes: the UAV flies to the corresponding airspace based on the received target search area; the UAV uses its onboard visible light camera and / or infrared thermal imaging equipment to acquire images of the target search area; the UAV performs target detection on the acquired images in the edge computing unit and outputs the target category, detection box, detection confidence and detection timestamp as UAV multimodal observation data.

[0039] In this embodiment, the UAV receives the target search area sent by the unmanned surface vessel. Afterwards, it flies to the corresponding airspace, acquires visible light and / or infrared images of targets within the area, and performs target detection in the UAV's edge computing unit.

[0040] Let the UAV-side detection results be expressed as: in, Indicates the target category, Indicates the detection box. Indicates the detection confidence level. The timestamp indicates the detection time. The UAV is mainly used for: (1) aerial verification of suspected targets found by the UAV; (2) filling blind spots in areas where the UAV's field of view is limited; and (3) continuous observation of dynamic targets on the sea surface.

[0041] S4. Acquire the electromagnetic data of the unmanned surface vessel (USV) and perform spatiotemporal alignment of the USV's multi-source detection results, UAV's multimodal observation data, and the USV's electromagnetic data to obtain aligned data.

[0042] The method for obtaining aligned data includes: acquiring unmanned surface vessel (USV) side electromagnetic data; synchronizing the USV multi-source detection results, UAV multimodal observation data, and USV side electromagnetic data in time, ensuring that the time difference between the three meets a preset time synchronization tolerance; performing coordinate transformation on the target positions corresponding to the USV side visual target detection information and the target positions corresponding to the UAV multimodal observation data based on the camera intrinsic parameters, camera extrinsic parameters, and scale factors of each platform, and uniformly mapping them to the world coordinate system to achieve spatial registration; and performing target association based on the time-synchronized and spatially registered data to determine whether the detection results from different sources correspond to the same target, thus obtaining aligned data.

[0043] In this embodiment, since both the unmanned surface vessel (USV) and the unmanned aerial vehicle (UAV) have edge image detection capabilities, and the USV also has electromagnetic sensing capabilities, the system needs to uniformly align the following three types of data: electromagnetic data from the USV side; image detection data from the USV side; and image detection data from the UAV side.

[0044] (1) Time alignment Let the electromagnetic event time of the unmanned surface vessel be . The image detection time for the unmanned surface vessel is The drone image detection time is Then the following is required: in, , , This indicates the time synchronization tolerance.

[0045] (2) Spatial alignment By mapping the target positions corresponding to the image detection results of unmanned surface vessels and drones to the world coordinate system, a unified coordinate expression of the visual results and electromagnetic positioning information from both platforms is achieved. in, This represents the position vector of the target detected by the unmanned surface vessel in the world coordinate system. The position vector of the target detected by the UAV in the world coordinate system. This represents the rotation matrix from the UAV's camera coordinate system to the world coordinate system. This represents the translation vector from the UAV's camera coordinate system to the world coordinate system. This represents the scale factor of the unmanned surface vessel's side pixels projected onto the camera coordinate system. This represents the scale factor of the back projection of a UAV-side pixel into the camera coordinate system, used to characterize the depth or distance of the target point along the optical axis. This represents the intrinsic parameter matrix of the drone's side camera. This represents the pixel coordinates of the target point in the image of the unmanned surface vessel. This represents the pixel coordinates of the target point in the drone image.

[0046] The scale factor can be obtained through any of the following methods: (1) estimation based on the relationship between the target's prior size and the imaging; (2) acquisition based on binocular vision, ranging sensors, or laser ranging devices; (3) inverse calculation based on the constraints of platform height, pitch angle, and sea level; or (4) estimation based on the target's motion relationship and pose changes in consecutive frames.

[0047] S5. Based on the aligned data, construct the side visual features of the unmanned surface vessel, the side visual features of the unmanned aerial vehicle, and the electromagnetic features, and perform feature fusion. Based on the fused features, obtain the target recognition result.

[0048] The method for obtaining target recognition results includes: extracting unmanned surface vessel (USV) side-view features from aligned data, including first target shape features, first texture features, first detection features, and first target detection confidence; extracting unmanned aerial vehicle (UAV) side-view features from aligned data, including second target shape features, second texture features, second detection features, and second target detection confidence; extracting electromagnetic features from aligned data, including frequency, power, bandwidth, duration, and incident direction features of electromagnetic anomalies; assigning weights to USV side-view features, UAV side-view features, and electromagnetic features and constructing a fused feature vector; calculating the target existence probability based on the fused feature vector; and determining the target as valid when the target existence probability meets a preset target determination threshold, thus obtaining the target recognition result.

[0049] In this embodiment, the system constructs visual features of unmanned surface vessels, visual features of unmanned aerial vehicles, and electromagnetic features of unmanned aerial vehicles.

[0050] The side visual features of an unmanned surface vessel are represented as follows: The side-view features of the drone are represented as follows: in, This indicates the shape features of the target in the unmanned surface vessel's side image. It represents the shape features of a target in a UAV side image, used to characterize the target's outer contour, aspect ratio, area, edge distribution, or other geometric shape information; This represents the texture features of a target in an unmanned surface vessel (USV) side image. It represents the texture features of a target in a UAV-side image, used to characterize the target's surface texture, grayscale distribution, local texture changes, or other image texture information; This represents the detection features output by the unmanned surface vessel side edge image detection module. This represents the detection features output by the UAV side edge image detection module, used to characterize the semantic features, target category features, or bounding box related features extracted by the intermediate layer of the detection network; This indicates the confidence level of the unmanned surface vessel's side target detection. This represents the target detection confidence level on the UAV side, used to characterize the degree of confidence that the current detection result belongs to the target category.

[0051] in, , , All of these can be scalar features, low-dimensional feature vectors, or fixed-length feature vectors after feature dimensionality reduction. This embodiment does not limit these features.

[0052] Electromagnetic characteristics are represented as follows: in, f Indicates signal frequency. P Indicates signal power. B Indicates signal bandwidth. Indicates the signal period. i Indicates the signal phase. This indicates the confidence level of an electromagnetic signal.

[0053] Based on this, three types of feature fusion vectors are constructed: in, α Indicates the weights of the unmanned surface vessel's side visual features; β Indicates the weights of the UAV's side visual features; s This represents the electromagnetic characteristic weight.

[0054] Furthermore, the probability of the target's existence can be expressed as: in, This indicates the probability that the fused target exists. This represents the target probability obtained based on the side-view visual features of the unmanned surface vessel. This represents the target probability obtained based on the UAV's side-view visual features. This represents the target probability obtained based on electromagnetic characteristics. , and Let represent the probability fusion weights, and satisfy . .

[0055] When satisfied When the target is determined to be valid, among which The threshold is used to determine the target.

[0056] S6. Based on the target recognition results, continuously update the target position, velocity, and trajectory.

[0057] In this embodiment, the target state vector is defined as: in, Indicates time k The target state vector; px Indicates the target in the world coordinate system x Positional component of direction; py Indicates the target in the world coordinate system y Positional component of direction; vx Indicates the target in the world coordinate system x The velocity component in the direction; vy Indicates the target in the world coordinate system y The velocity component in the direction.

[0058] In one implementation, the system uses a discrete state-space model to predict the target state, and its state prediction equation is as follows: in, Indicates time k The target predicted state vector, A This represents the state transition matrix, used to describe the target from time [time]. k -1 to time k The relationship between the changes in the motion state, Indicates time k The updated state vector of the target is -1. This is the process noise vector, used to characterize the deviation between the target motion model and the actual motion.

[0059] When the target satisfies the assumption of uniform motion, the state transition matrix A It can be represented as: in, It represents the time interval between two adjacent moments.

[0060] The prediction error covariance matrix can be expressed as: in, Indicates time k The prediction error covariance matrix, Indicates time k The prediction error covariance matrix is ​​-1. Q The process noise covariance matrix is ​​used to describe the uncertainties caused by target maneuvering, sea state disturbances, and motion model simplification errors.

[0061] At that moment k When simultaneously obtaining observations from both the unmanned surface vessel (USV) and the unmanned aerial vehicle (UAV), a joint observation vector is constructed: in, Indicates time k The joint observation vector, Represents the unmanned surface vessel's side observation vector. This represents the observation vector from the UAV side.

[0062] In one implementation, both the unmanned surface vessel (USV) side observation vector and the unmanned aerial vehicle (UAV) side observation vector can be expressed in the form of target position observation, as follows: in, and Indicates the time of the unmanned surface vessel. k Observations of the target location, and Indicates the time of the drone side k Observations of the target location.

[0063] The observation equation can be expressed as: in, H This represents the observation matrix, used to establish the mapping relationship between the target state vector and the observation vector; This represents the observation noise vector, used to characterize factors such as image detection error, platform positioning error, spatiotemporal registration error, and sensor measurement error.

[0064] When the joint observation vector is in the form of a four-dimensional positional quantity At that time, the observation matrix H It can be represented as: Calculate the Kalman gain based on the Kalman filtering method: in, Indicates time k The Kalman gain matrix; R This represents the observation noise covariance matrix, used to describe the statistical characteristics of observation errors of unmanned surface vessels and unmanned aerial vehicles.

[0065] Update the predicted state using Kalman gain: in, Indicates time m The updated state vector, This represents the observation residual, used to characterize the difference between the actual observed value and the predicted observed value.

[0066] The updated error covariance matrix is: in, For a moment h The updated error covariance matrix; I It is an identity matrix.

[0067] To further evaluate the tracking reliability under dual-platform collaborative observation, this invention also constructs a trajectory reliability evaluation function: in, Indicates time k The reliability of the trajectory , , and This represents the credibility fusion weight coefficient; Indicates the time of the unmanned surface vessel. k Target detection confidence, Indicates the time of the drone side k Target detection confidence; It represents the intersection-union ratio between the target detection boxes at the current time and the previous time, and is used to characterize the temporal continuity of the target detection results; Indicates the actual observed position at the current moment; Indicates the predicted position at the current moment; This represents the Euclidean distance between the observed and predicted locations. This represents an exponential function used to map position deviation into a decay term.

[0068] in, and All of these are position vectors of the target in a unified world coordinate system, which can be represented as two-dimensional or three-dimensional position vectors. This embodiment does not limit this.

[0069] When satisfied If the current target trajectory is deemed reliable, the system continues to update the target trajectory. When satisfied If the system determines that the current target may be mismatched, obscured, interrupted, or falsely detected, it will trigger a re-search, modal compensation, task replanning, or have another platform take over the observation.

[0070] in, The threshold for determining the reliability of the trajectory.

[0071] Ultimately, the system outputs the target at time [time]. k The updated status results include target position, velocity, trajectory sequence, and corresponding confidence information.

[0072] Example 2 This embodiment constructs a maritime target perception system for collaborative operation between unmanned aerial vehicles (UAVs) and unmanned surface vessels (USVs). The USV serves as a surface patrol platform, equipped with electromagnetic spectrum sensing equipment, a visible light camera and / or infrared thermal imaging equipment, an edge computing unit, a navigation and positioning module, and a wireless communication module. The UAV serves as an aerial maneuvering platform, equipped with a visible light camera and / or infrared thermal imaging equipment, an edge computing unit, a flight control system, a navigation and positioning module, and a wireless communication module. The USV and UAV exchange mission commands, target information, and status data via a wireless link. The upper-level command and control platform receives the fused results and performs situational awareness display and mission management.

[0073] In this embodiment, the unmanned surface vessel (USV) first performs continuous inspection missions in the target sea area according to a preset cruise route. During the inspection, the USV continuously scans the surrounding sea area using electromagnetic spectrum sensing equipment to acquire electromagnetic signal information within the target area and identify abnormal frequency bands, abnormal power, abnormal duration, or abnormal direction of arrival. On the other hand, it continuously images the sea surface environment using its onboard visible light camera and / or infrared thermal imaging equipment, and uses an edge-side image detection module to analyze the acquired images in real time to identify ships, personnel, floating objects, or other targets on the sea surface.

[0074] When the unmanned surface vessel (USV) detects only an anomalous electromagnetic event, the system generates an initial search area where the target may appear based on the USV's current position, the direction of the anomalous signal, and empirical distance estimation results. When the USV detects only a suspected target in an image, the system estimates the target's corresponding sea surface spatial position based on the USV's current pose information, camera parameters, and the image region where the detection box is located, and generates a visual search area. When the USV detects both electromagnetic anomalies and a suspected target in the image, the system jointly analyzes the two types of results to generate a target search area with higher confidence. This search area can be represented as the suspected range of the target within the current sea area and serves as the guidance basis for subsequent UAV verification tasks.

[0075] After the target search area is generated, the collaborative control module selects a suitable UAV to perform the verification task based on factors such as task priority, the UAV's current position, remaining range, and current sea conditions. After receiving task information from the unmanned surface vessel (USV) or the upper platform, the UAV flies over the target area to conduct aerial observation. Compared to the USV's low-angle observation of the sea surface, the UAV has higher maneuverability and a wider field of view, enabling it to perform aerial verification of suspected targets initially detected by the USV and to fill in blind spots in areas where the USV's field of view is limited, severely obstructed, or at a distance.

[0076] After the UAV reaches the target area, it continuously acquires images of the target area using its onboard visible light camera and / or infrared thermal imaging equipment. The edge image detection module then identifies targets in the images and outputs information such as target category, bounding box, detection confidence level, and corresponding timestamp. The UAV-side detection results can be used to verify the authenticity of the UAV-side detection results and to supplement target information not observed by the UAV, thus forming a dual-platform collaborative observation mechanism.

[0077] Subsequently, the system performs unified processing on the electromagnetic sensing results from the unmanned surface vessel (USV), the image detection results from the USV, and the image detection results from the unmanned aerial vehicle (UAV). This processing includes time synchronization, spatial registration, and target association. Time synchronization ensures that data from different platforms and modes have a corresponding temporal relationship; spatial registration maps the target positions obtained from different platforms and perspectives to the same reference coordinate system; and target association determines whether the target detected by the USV and the target detected by the UAV correspond to the same sea surface target. After the above processing, a joint observation result of the same target based on multi-source data can be generated.

[0078] After completing multi-source alignment, the system fuses and determines the image detection results from unmanned surface vessels (USVs) and unmanned aerial vehicles (UAVs), as well as the electromagnetic anomaly results from USVs. Specifically, the system comprehensively considers factors such as the target detection confidence level from the USV side, the target detection confidence level from the UAV side, the degree of electromagnetic anomaly, spatial location consistency, temporal consistency, and target category consistency to comprehensively evaluate whether the current suspected target is valid. When the comprehensive evaluation result reaches a set threshold, the current target is determined to be a valid target; when the comprehensive evaluation result does not reach the threshold, it can be marked as a target to be reviewed, and continue to wait for subsequent observation data to be supplemented, or trigger a new search task.

[0079] For confirmed targets, the system proceeds to a continuous tracking phase. During this phase, the unmanned surface vessel (USV) and unmanned aerial vehicle (UAV) can conduct continuous observations of the target based on their respective platform conditions: when the USV can stably observe the target, it continuously outputs target detection results; when the USV's field of view is limited, the target is too far away, or sea conditions are strong, the UAV can continue aerial tracking; when both platforms are simultaneously capable of observation, the system can simultaneously receive the observation results from both platforms and jointly update the target's position, velocity, and trajectory. In this way, the system can continuously obtain the target's motion status on the sea surface and improve its continuous target tracking capability in complex environments.

[0080] During tracking, the system can also assess the reliability of the current target trajectory based on target detection confidence, positional continuity between consecutive frames, trajectory smoothness, and consistency of results from both platforms. When trajectory reliability is high, the system continues to update the target status; when trajectory reliability decreases, the system can trigger a re-search, modal compensation, task replanning, or have another platform take over observation to reduce the risk of mistracking caused by occlusion, surge reflection, target deflection, or temporary loss.

[0081] Finally, the system generates target status results and transmits them back to the upper-level command and control platform. These results include at least the target number, target category, current location, speed, historical trajectory, detection confidence level, and timestamp. The upper-level platform can then use these results to perform maritime situation display, anomaly warnings, trajectory analysis, and subsequent coordinated response tasks.

[0082] Example 3 In a preferred embodiment, the unmanned surface vessel (USV) serves as a front-end routine inspection platform, primarily responsible for detecting electromagnetic spectrum anomalies, initial screening of sea surface images, and reporting suspected targets; the unmanned aerial vehicle (UAV) serves as a rapid response platform, primarily responsible for target area verification, aerial gap filling, and short-to-medium-term continuous tracking.

[0083] In this implementation, the unmanned surface vessel (USV) performs low-speed cruising within the mission area for extended periods, continuously monitoring the surrounding signal environment using electromagnetic sensing devices and performing real-time analysis of sea surface images through an edge image detection module. When the USV detects a suspected abnormal target, it sends the anomaly type, suspected target location, search area range, and corresponding confidence level information to the unmanned aerial vehicle (UAV). Based on the received information, the UAV flies to the corresponding area and performs a secondary confirmation of the target from a more advantageous overhead angle, returning the confirmation result to the USV or the upper platform. If the secondary confirmation result is valid, the system enters the target continuous tracking phase; if the secondary confirmation result is invalid, the suspected event can be eliminated or transferred to a pending observation state.

[0084] This implementation method is applicable to large-scale patrol scenarios and can improve the efficiency of detecting and confirming abnormal targets on the sea surface while reducing the cost of routine drone patrols.

[0085] Example 4 In another implementation, both the unmanned surface vessel (USV) and the unmanned aerial vehicle (UAV) perform image detection on the target area simultaneously, and the USV simultaneously outputs electromagnetic sensing results. The system performs joint analysis of the three types of data to perform multi-source confirmation and continuous tracking of the same target.

[0086] In this implementation, when both the unmanned surface vessel (USV) and the unmanned aerial vehicle (UAV) detect a target, the system prioritizes comparing the temporal and spatial consistency of the detection results from the two platforms. When the two results show high consistency, they can be identified as dual-platform observations of the same target, and both can be used together for target status updates. Compared with single-platform observation, this method can improve target positioning accuracy and tracking stability, and is particularly suitable for application scenarios where the target is highly maneuverable, the sea surface background is complex, or a single platform has blind spots.

[0087] Example 5 In practical applications, a platform may be temporarily unable to continue stable detection due to factors such as wave obstruction, changes in lighting, platform vibration, target turning, or brief departure from the field of view. To address this, the present invention can also include an abnormal interruption compensation mechanism.

[0088] When the unmanned surface vessel (USV) temporarily loses the target, the system can prioritize using the detection results from the unmanned aerial vehicle (UAV) to maintain continuous target tracking. If the UAV is temporarily unable to reliably verify the target due to limitations in flight range, endurance, or observation angle, the system can revert to having the USV continue surface tracking. If both platforms experience short-term detection interruptions, the system can predict the target's possible location based on historical trajectory information and perform a re-search within the predicted area. These compensation methods enhance the system's robustness and continuous operational capability under complex sea conditions.

[0089] Example 6 The target information output by the method of this invention can be directly used for edge-side alarms or uploaded to the upper-layer business platform. After receiving the target results, the upper-layer platform can display the target's location, movement trajectory, and status changes on electronic charts, mission status interfaces, or monitoring terminals. It can also perform abnormal behavior analysis, event backtracking, and task linkage by combining historical records. For key targets, the upper-layer platform can further issue instructions for continuous tracking, accompanying surveillance, route adjustment, or other actions, realizing a complete business closed loop from target discovery to target confirmation and target handling.

[0090] Example 7 In this embodiment, a multimodal perception system based on UAV-UAV collaboration includes: an UAV detection module, a search area generation module, a UAV detection module, a data alignment module, a target recognition module, and an update module.

[0091] In the unmanned surface vessel (USV) detection module, the USV performs electromagnetic spectrum scanning and edge image detection to obtain multi-source detection results. The search region generation module generates a target search region based on the USV's multi-source detection results. In the unmanned aerial vehicle (UAV) detection module, the UAV performs visual verification within the target search region and collects multi-modal observation data. The data alignment module acquires electromagnetic data from the USV side and performs spatiotemporal alignment of the USV's multi-source detection results, the UAV's multi-modal observation data, and the USV's electromagnetic data to obtain aligned data. The target recognition module constructs USV-side visual features, UAV-side visual features, and electromagnetic features based on the aligned data and performs feature fusion to obtain the target recognition result. The update module continuously updates the target's position, velocity, and trajectory based on the target recognition result.

[0092] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal perception method based on UAV-Unmanned Surface Vessel (USV) cooperation, characterized in that, Includes the following steps: The unmanned surface vessel performs electromagnetic spectrum scanning and edge image detection to obtain multi-source detection results for the unmanned surface vessel; A target search area is generated based on the multi-source detection results of the unmanned surface vessel; The UAV performs a visual verification task in the target search area and collects multimodal observation data from the UAV. Acquire unmanned surface vessel (USV) side electromagnetic data, and perform spatiotemporal unified alignment of the USV multi-source detection results, the UAV multimodal observation data, and the USV side electromagnetic data to obtain aligned data; Based on the aligned data, unmanned surface vessel side visual features, unmanned aerial vehicle side visual features and electromagnetic features are constructed and fused. Target recognition results are obtained based on the fused features. Based on the target recognition results, the target's position, velocity, and trajectory are continuously updated.

2. The multimodal perception method based on UAV-Unmanned Surface Vessel cooperation according to claim 1, characterized in that, The methods for obtaining the multi-source detection results of the unmanned surface vessel include: The unmanned surface vessel performs electromagnetic spectrum scanning on the target sea area, extracts the frequency, power, bandwidth, duration and incident direction of candidate signals, and constructs the candidate signal feature vector; An anomaly score is calculated based on the candidate signal feature vector. When the anomaly score meets the preset anomaly triggering condition, it is determined to be an electromagnetic anomaly event, and electromagnetic anomaly event information is obtained. The unmanned surface vessel (USV) uses its onboard visible light camera and / or infrared thermal imaging equipment to acquire sea surface images, performs target detection in the edge computing unit, and outputs the target category, target bounding box, detection confidence score and timestamp to obtain the USV's side visual target detection information. The multi-source detection results of the unmanned surface vessel include the electromagnetic anomaly event information and the unmanned surface vessel side visual target detection information.

3. The multimodal perception method based on UAV-Unmanned Surface Vessel cooperation according to claim 1, characterized in that, The method for generating the target search region includes: When the multi-source detection results of the unmanned surface vessel only contain electromagnetic anomaly event information, the target position is estimated based on the position information and electromagnetic signal direction angle of the unmanned surface vessel, combined with empirical distance, and the target search area is constructed based on the azimuth error and distance error. When the multi-source detection results of the unmanned surface vessel (USV) only contain the USV's side visual target detection information, the center point of the detection box is projected onto the geographic coordinate system based on the USV's camera intrinsic parameters, camera extrinsic parameters, and the center pixel coordinates of the detection box to obtain the estimated target location and construct the target search area. When the multi-source detection results of the unmanned surface vessel (USV) simultaneously include electromagnetic anomaly event information and USV side-view target detection information, the estimated position obtained from the electromagnetic anomaly and the estimated position obtained from the visual detection are weighted and fused, and the target search area is constructed based on the fused position.

4. The multimodal perception method based on UAV-Unmanned Surface Vessel cooperation according to claim 1, characterized in that, The methods for obtaining the UAV multimodal observation data include: The drone flies to the corresponding airspace based on the received target search area; The UAV uses its onboard visible light camera and / or infrared thermal imaging device to acquire images of the target search area; The UAV performs target detection on the acquired images in the edge computing unit and outputs the target category, detection box, detection confidence and detection timestamp as the UAV's multimodal observation data.

5. The multimodal perception method based on UAV-Unmanned Surface Vessel cooperation according to claim 2, characterized in that, The methods for obtaining the aligned data include: The electromagnetic data of the unmanned surface vessel is acquired, and the multi-source detection results of the unmanned surface vessel, the multi-modal observation data of the unmanned aerial vehicle, and the electromagnetic data of the unmanned surface vessel are synchronized in time so that the time difference between the three meets the preset time synchronization tolerance. The target positions corresponding to the unmanned surface vessel's side-vision target detection information and the target positions corresponding to the UAV's multimodal observation data are respectively transformed based on the camera intrinsic parameters, camera extrinsic parameters, and scale factors of each platform, and uniformly mapped to the world coordinate system to achieve spatial registration; Based on the time-synchronized and spatially registered data, target association is performed to determine whether detection results from different sources correspond to the same target, thus obtaining the aligned data.

6. The multimodal perception method based on UAV-Unmanned Surface Vessel cooperation according to claim 1, characterized in that, The methods for obtaining the target recognition results include: Unmanned surface vessel (USV) side-view features are extracted from the aligned data. The USV side-view features include a first target shape feature, a first texture feature, a first detection feature, and a first target detection confidence level. The UAV side-view features are extracted from the aligned data, including second target shape features, second texture features, second detection features, and second target detection confidence. Electromagnetic features are extracted from the aligned data, including the frequency, power, bandwidth, duration, and incident direction characteristics of electromagnetic anomalies. Weights are assigned to the unmanned surface vessel side-view features, the unmanned aerial vehicle side-view features, and the electromagnetic features, and a fused feature vector is constructed. The probability of the target's existence is calculated based on the fused feature vector. When the probability of the target's existence meets a preset target determination threshold, the target is determined to be valid, and the target recognition result is obtained.

7. A multimodal perception system based on UAV-Unmanned Surface Vessel (USV) cooperation, wherein the system applies the method described in any one of claims 1-6, characterized in that, include: The system includes an unmanned surface vessel detection module, a search area generation module, an unmanned aerial vehicle (UAV) detection module, a data alignment module, a target recognition module, and an update module. In the unmanned surface vessel detection module, the unmanned surface vessel is used to perform electromagnetic spectrum scanning and edge image detection to obtain the multi-source detection results of the unmanned surface vessel. The search area generation module generates a target search area based on the multi-source detection results of the unmanned surface vessel. In the UAV detection module, the UAV is used to perform a visual verification task in the target search area and to collect multimodal observation data of the UAV. The data alignment module is used to acquire the electromagnetic data of the unmanned surface vessel (USV) and to perform spatiotemporal unified alignment of the USV multi-source detection results, the UAV multimodal observation data, and the USV side electromagnetic data to obtain aligned data. The target recognition module constructs unmanned surface vessel side visual features, unmanned aerial vehicle side visual features, and electromagnetic features based on the aligned data, and performs feature fusion to obtain the target recognition result based on the fused features. The update module continuously updates the target's position, speed, and trajectory based on the target recognition results.

Images