A method for processing comprehensive information of multiple unmanned platforms

Abstract

The present application relates to the field of cross-domain multi-unmanned platform command and control, and particularly relates to a multi-unmanned platform comprehensive information processing method, comprising the following steps: establishing transmission rules corresponding to data types in the multi-unmanned platform; generating a three-dimensional electronic sand table model of a target area scene; generating an unmanned platform pose mapping display under the three-dimensional electronic sand table model; transmitting the acquired environmental perception raw data by the corresponding transmission rules, and unifying the description form of the environmental perception raw data; acquiring the pose estimation value of the unmanned platform, and loading it into the three-dimensional electronic sand table model based on the environmental perception raw data, so as to achieve the purpose of cooperative positioning. The present application can solve the on-site command and dispatching problems caused by the complexity of actual combat environment and mission, realize multi-source information real-time acquisition, multi-scale information intelligent processing and actual combat situation dynamic display for cross-domain multi-platform, so as to further reduce the on-site decision burden.

Description

Technical Field

[0001] This invention relates to the field of cross-domain multi-unmanned platform command and control, and particularly to a method for integrated information processing of multiple unmanned platforms. Background Technology

[0002] In increasingly complex mission and application environments, single unmanned platforms have limitations in carrying capacity, mobility, sensor detection range, and detection accuracy. Unmanned platforms with different structures and functions need to work together to complete target tasks. Therefore, the comprehensive on-site information processing brought about by the complexity of combat environments and missions will inevitably become a thorny issue.

[0003] Information fusion technology can expand the temporal and spatial coverage of battlefield awareness, transforming single-source detection into network detection, which can improve the detection capability of battlefield targets, increase the probability of target discovery and identification level; it can improve the accuracy and reliability of synthetic information, support joint environmental situation of important targets, support joint operation decision-making and scheme formulation, improve the real-time performance and accuracy of threat assessment, and support the sharing of perception information in target areas.

[0004] Currently, most existing unmanned platforms (including aerial drones and ground-based unmanned vehicles) operate primarily as standalone systems, unable to form a cohesive system through interconnection, resulting in low overall combat effectiveness and an inability to execute complex tactics and strategies. Furthermore, the few existing integrated information processing solutions for multiple unmanned platforms largely rely on data integration methods, failing to truly fuse cross-domain, multi-level data at the feature level and pixel level, thus preventing multiple platforms from working in unison. Therefore, designing a comprehensive information processing method for multi-unmanned platforms that integrates real-time acquisition of multi-source information, intelligent processing of multi-scale information, and mission-oriented situational awareness display becomes crucial. Summary of the Invention

[0005] The purpose of this invention is to construct a method for integrated information processing across multiple unmanned platforms, aiming to solve the challenges of on-site information processing arising from the complexity of mission environments and objectives. On one hand, it designs data interaction categories and transmission methods for multiple unmanned platforms to unify the form of data returned from each platform, thereby improving the efficiency of integrated information processing. On the other hand, it designs a method for constructing a 3D sand table based on the fusion of photographic and lidar data, generating a 3D electronic sand table of the target area. Simultaneously, it combines cross-domain air-ground environmental information and collaborative positioning to achieve situational interaction based on the 3D electronic sand table, mapping the position and attitude of all unmanned platforms connected to the integrated information processing center onto the electronic sand table.

[0006] The technical solution adopted by the present invention to achieve the above objectives is: a method for integrated information processing of multiple unmanned platforms, comprising the following steps:

[0007] 1) Establish transmission rules for each data type in multiple unmanned platforms and the corresponding ground control station; send control commands to multiple unmanned platforms through the ground control station, and uniformly receive formatted data returned from multiple unmanned platforms;

[0008] 2) The ground control station generates a three-dimensional electronic sand table model of the target area scene using vertical photography;

[0009] 3) The ground control station establishes a three-dimensional unmanned platform model based on a three-dimensional electronic sand table model and has multi-degree-of-freedom pose attributes, and maps the position and attitude of all unmanned platforms connected to the ground control station into the three-dimensional electronic sand table model, that is, generates a pose mapping display of unmanned platforms under the three-dimensional electronic sand table model.

[0010] 4) The unmanned platform acquires raw environmental perception data and sends it to the ground control station for processing; the processed data is transmitted according to the corresponding transmission rules; and the description format of the raw environmental perception data is standardized.

[0011] 5) Obtain the pose estimation value of the unmanned platform based on the environmental perception raw data in a unified description form; load the pose estimation value of the unmanned platform into the three-dimensional electronic sand table model based on the environmental perception raw data to realize the cooperative pose estimation of the unmanned platform, thereby achieving the purpose of cooperative positioning.

[0012] The data types include: streaming data, sensor data, control data, task data, status report data, notification report data, situational awareness information data, and target location information;

[0013] The streaming data can be any one or more of visible light images, infrared images, and radar images;

[0014] Sensor data can be any one or more of static images, dynamic images, and sonar data monitored and transmitted by the unmanned platform as appropriate.

[0015] Control data refers to data used to manipulate unmanned platforms or their payloads.

[0016] The mission data includes planning, task, and follow-up data for unmanned platforms and their payloads.

[0017] The status report data consists of status reports of the unmanned platform before and after performing the task;

[0018] Notification reports are short message notifications sent by unmanned platforms;

[0019] The situational awareness information data is the situational awareness information transmitted back by the unmanned platform;

[0020] The target location information is the data transmitted back by the unmanned platform after it discovers the target and identifies its location.

[0021] In step 1), the establishment of transmission rules corresponding to each data type in the multi-unmanned platform and the ground control station specifically involves:

[0022] The transmission rules for streaming data adopt the UDP protocol, which does not require feedback during the transmission process;

[0023] The FTP transmission protocol, which requires a verification process during transmission, is used for the transmission of sensor data.

[0024] The TCP transmission control protocol is used for the transmission of control data, task data, notification report data, and target location information.

[0025] Status report data is sent cyclically between the TCP Transmission Control Protocol and the User Datagram Protocol.

[0026] Situational awareness information data includes two or more of the following: streaming data, sensor data, control data, mission data, notification-type report data, target location information, and status report data, and follows the transmission rules corresponding to the respective data types.

[0027] Step 2) includes the following steps:

[0028] 2-1) Obtain image files of the target ground area captured by the unmanned platform, including either video or radar images;

[0029] 2-2) Perform sparse matching on the image files of the target ground area to achieve multi-image relative positioning of the target ground area image without positioning information and radar image;

[0030] 2-3) Perform absolute orientation on the image files of the target ground area captured by the unmanned platform to obtain images with absolute coordinates. Then, perform matching point purification on the images with absolute coordinates to obtain the absolute coordinates of pixels in two adjacent frames, thus completing the absolute orientation of the target ground area.

[0031] 2-4) Using a dense point cloud matching algorithm, a three-dimensional electronic sand table model of the target area scene is generated based on the absolute coordinates of pixels in multiple adjacent frames of images.

[0032] Step 2-2) specifically involves:

[0033] a. Image files captured by an unmanned platform are convolved with a Gaussian function and then low-pass filtered to generate Gaussian pyramid images and Gaussian difference images;

[0034] b. Perform feature point detection on the Gaussian pyramid image and the difference of Gaussian image to obtain the extreme point detection and sub-pixel level localization of feature points in the scale space of the two images;

[0035] c. Obtain the main direction of the feature points in step (2);

[0036] d. Based on the main direction of the acquired feature points, construct SIFT feature descriptors and perform descriptor matching to achieve relative positioning between the unmanned platform and the target ground area.

[0037] Steps 2-3) are specifically as follows:

[0038] (1) From the image of the target ground area taken by the unmanned platform, select SIFT feature descriptors to form a matching point set, and randomly select a set of matching point pairs from the initial matching points within the set range;

[0039] (2) Using the selected matching point pairs, calculate the essential matrix model between two images with relatively adjacent frames, i.e.:

[0040] E=T R

[0041] E represents the essential matrix model, T represents the camera's movement matrix transformation when capturing two images in adjacent frames, and R represents the camera's selection matrix transformation when capturing two images in adjacent frames.

[0042] (3) Determine the distance threshold using the reference image with GPS data from the ground control station; based on the distance threshold, obtain the interior points that conform to the essential matrix model;

[0043] The distance threshold is the distance value from a pixel to the actual target ground area; that is, pixels with a distance less than the corresponding distance threshold are interior points of the essential matrix model, and pixels with a distance greater than the corresponding distance threshold are exterior points of the essential matrix model.

[0044] (4) Calculate the ratio of the number of interior points to the total number of matching points, and update the maximum number of iterations based on the ratio of the number of interior points to the total number of matching points in the overall matching pairs.

[0045]

[0046] Where, N max The maximum number of iterations is represented by 'a', the number of interior points is represented by 'm', the total number of matched points is represented by 'GSD', and the distance threshold is represented by 'GSD'.

[0047] (5) Repeat steps (1) to (3) until a matching point that meets the quantity setting value is found;

[0048] That is, the number of interior points that satisfies the distance threshold condition, i.e.:

[0049] n = S / GSD 2

[0050] Where n represents the number of inliers that satisfy the distance threshold condition, S represents the area of ​​the ground region of the target image, and GSD represents the distance threshold;

[0051] (6) Based on the essential matrix between two adjacent frames to be matched, obtain the Simpson error of all matching points, i.e.:

[0052]

[0053] Where S[f] represents the Simpson error, and x1 and x2 represent the values ​​in the two graph essence matrices, respectively;

[0054] (7) Remove all outliers in the matching pair whose Simpson error is greater than the distance threshold, retain the remaining matching point pairs, and finally purify the matching point pairs so that two adjacent images can be stitched together, thereby obtaining the absolute coordinates of pixels in multiple sets of two adjacent frames and completing the absolute orientation of the target ground area.

[0055] The process involves obtaining the absolute coordinates of pixels within multiple adjacent frames, i.e.:

[0056] M i,j =P 0,0 +GSD×W i,j

[0057] Among them, M i,j P represents the latitude and longitude coordinates of pixels i and j in two adjacent image frames. 0,0 W represents the GPS coordinates of the reference image. i,j GSD represents the distance between a pixel and the reference image.

[0058] Step 3) includes the following steps:

[0059] 3-1) Construct a TCP server that conforms to the situation data transmission method using ArcGIS service for the 3D electronic sand table model, and connect multiple unmanned platforms operating simultaneously to this server;

[0060] 3-2) Create a three-dimensional platform model with multiple degrees of freedom pose attributes, load the three-dimensional platform models of multiple unmanned platforms into the three-dimensional electronic sand table model; and perform data calculation on the pose data sent by the unmanned platforms and convert it into latitude and longitude data as well as pitch angle, roll angle and heading angle data, and attach these multiple sets of pose attribute values ​​to the unmanned platform model in the three-dimensional platform model;

[0061] 3-3) Adjust the size of the 3D platform model to complete the pose mapping display of the unmanned platform under the 3D electronic sand table model.

[0062] Step 4) includes the following steps:

[0063] The unmanned platform acquires raw environmental perception data, namely:

[0064] The aerial unmanned platform collects laser data of the target area through a laser sensor and visual data of the target area through a camera. At the same time, the ground mobile unmanned platform also collects laser data of the target area through a laser sensor and visual data of the target area through a camera. That is, the aerial unmanned platform and the ground mobile unmanned platform work together to collect laser-visual data.

[0065] Both aerial unmanned platforms and ground-based mobile unmanned platforms acquire perception perspective and orientation data through positioning devices.

[0066] The process of transmitting the processed data according to the corresponding transmission rules is as follows:

[0067] The perception perspective and orientation data of the unmanned platform are transmitted in the form of sensor data; laser-vision data are transmitted in the form of streaming data.

[0068] The unification of the description format of raw environmental perception data is specifically as follows:

[0069] The streaming data of laser and vision data from unmanned platforms are jointly described using latitude and longitude coordinates. The description format is unified as latitude and longitude coordinates f(longitude, latitude), which represents the dataset of laser and vision detection at a certain latitude and longitude.

[0070] Step 5) specifically includes:

[0071] The unmanned platform generates a 3D electronic sand table model of the target area scene from the raw environmental perception data, and subtracts it from the 3D platform model by dimension. Then, using a Kalman-type optimization filtering strategy, it obtains the pose estimate of the unmanned platform.

[0072]

[0073] Among them, P [a,b,…,n] Let W represent the pose estimate of the unmanned platform at the nth rank (a, b, ..., n). [a,b,…,n] This represents the image file of the original target ground area collected by the unmanned platform of the a, b...nth generation. This represents the Kalman fusion and normalization algorithm, F. [a,b,…,n] The method for generating a three-dimensional electronic sand table in step 2) is denoted by D, where D represents the three-dimensional platform model.

[0074] The present invention has the following beneficial effects and advantages:

[0075] 1. This invention combines the characteristics of multi-scale information generated by cross-domain multi-unmanned platform collaboration, and gives full play to the advantages of centralized data processing, effectively improving the ability of on-site command and decision-making personnel to control the mission area;

[0076] 2. The integrated information processing method of the present invention effectively improves the efficiency of integrated information processing and fusion by designing data interaction categories and transmission methods for multiple unmanned platforms;

[0077] 3. This invention generates an interactive 3D electronic sand table with pose mapping of unmanned platforms by using a 3D sand table construction method based on the fusion of photography and LiDAR data, which intuitively displays the situational information of multiple unmanned platforms and the environment;

[0078] 4. This invention designs a cross-domain environmental information loading process into a three-dimensional electronic sand table and a collaborative positioning process, which improves the cross-domain system's ability to perceive and recognize its environment. Attached Figure Description

[0079] Figure 1 This is a schematic diagram illustrating the principle of the present invention based on integrated information processing across multiple unmanned platforms.

[0080] Figure 2 This is a flowchart of the method for constructing a three-dimensional electronic sand table model based on the fusion of photographic and lidar data according to the present invention.

[0081] Figure 3 Flowchart of a method for constructing a 3D sand table based on the fusion of photographic and LiDAR data;

[0082] Figure 4 Flowchart of GPU-based sparse matching algorithm for images;

[0083] Figure 5 Flowchart of situational interaction method based on 3D electronic sand table;

[0084] Figure 6 Flowchart of cross-domain environmental information fusion and collaborative positioning between air and ground;

[0085] Figure 7 Flowchart of cross-domain environmental information fusion between air and ground; Detailed Implementation

[0086] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0087] like Figure 1 The diagram shown is a schematic of the principle of the present invention based on the integrated information processing of multiple unmanned platforms. The present invention includes multiple unmanned platforms, including multiple aerial unmanned platforms (drones) and multiple ground unmanned platforms (unmanned vehicles).

[0088] There is one ground control station, which receives data transmitted by the unmanned platform and unifies the data format returned by the unmanned platform according to the transmission rules;

[0089] The integrated information processing system is the overall implementation of the integrated information processing method of the present invention;

[0090] This invention discloses a method for integrated information processing across multiple unmanned platforms, comprising the following steps:

[0091] 1) Establish transmission rules for each data type in multiple unmanned platforms and the corresponding ground control station; send control commands to multiple unmanned platforms through the ground control station, and uniformly receive formatted data returned from multiple unmanned platforms;

[0092] 2) The ground control station generates a three-dimensional electronic sand table model of the target area scene using vertical photography;

[0093] 3) The ground control station establishes a three-dimensional unmanned platform model based on a three-dimensional electronic sand table model and has multi-degree-of-freedom pose attributes, and maps the position and attitude of all unmanned platforms connected to the ground control station into the three-dimensional electronic sand table model, that is, generates a pose mapping display of unmanned platforms under the three-dimensional electronic sand table model.

[0094] 4) The unmanned platform acquires raw environmental perception data and sends it to the ground control station for processing; the processed data is transmitted according to the corresponding transmission rules; and the description format of the raw environmental perception data is standardized.

[0095] 5) Obtain the pose estimation value of the unmanned platform based on the environmental perception raw data in a unified description form; load the pose estimation value of the unmanned platform into the three-dimensional electronic sand table model based on the environmental perception raw data to realize the cooperative pose estimation of the unmanned platform, thereby achieving the purpose of cooperative positioning.

[0096] (1) First, design the data interaction categories and transmission methods for multiple unmanned platforms, as shown in Table 1:

[0097] All data that can be transmitted back from the unmanned platform is classified according to information type and data characteristics, and the corresponding transmission methods are specified. Information types are divided into streaming data, sensor data, control data, mission data, status reports, notification reports, situational awareness information data, and target location information data. The detailed description, main characteristics, and transmission method specifications for each information type are shown in the table below.

[0098] Table 1

[0099]

[0100]

[0101] All data that can be transmitted back from the unmanned platform is classified according to information type and data characteristics, and the corresponding transmission methods are specified:

[0102] Streaming data includes real-time images such as visible light images, infrared images, and radar images. Its characteristics are that the data is continuously pushed in a streaming manner, the data volume is large, it can realize one-to-many data transmission, and no feedback is required during the transmission process. This type of data transmission method is set as User Datagram Protocol (UDP).

[0103] Sensor data refers to reconnaissance data that can be collected by the platform and transmitted at an opportune time, including static images, dynamic images, and sonar data. Its characteristics are large data volume, support only one-to-one transmission, and the need for verification during transmission. This type of data transmission method is set as the File Transfer Protocol (FTP).

[0104] Control data refers to data that manipulates the platform or platform payload. Its characteristics include data being sent in the form of messages, small data volume, the need for one-to-one transmission, and the need to ensure reliability during transmission. This type of data transmission method is set as Transmission Control Protocol (TCP).

[0105] Task data consists of planning, task, and follow-up data for the platform and its payloads. Its characteristics include data being sent in the form of messages, data volume varying, the need for one-to-one transmission, and the requirement to ensure reliability during transmission. This type of data transmission method is set as Transmission Control Protocol (TCP).

[0106] Status report data refers to the status report before and after the platform task is executed. Its characteristics are that the data is sent in the form of messages, the data volume is small, it needs to be transmitted one-to-one, and the transmission process needs to be verified. This type of data transmission method is set to be sent cyclically by Transmission Control Protocol or User Datagram Protocol.

[0107] Notification-type report data refers to short message notifications sent by the platform, such as reports after a target is discovered. Its characteristics are that the data is sent in the form of messages, the data volume is small, it requires one-to-one transmission, and the transmission process needs to be verified. This type of data transmission method is set as the transmission control protocol.

[0108] Situational awareness information data refers to the situational awareness information returned by the platform. This type of data is diverse and is generally a combination of several other data types. It can be sent in accordance with other data types.

[0109] Target location information refers to the data transmitted back by the platform after discovering the target and identifying its location. Its characteristics are that the data is sent in the form of messages, the data volume is small, it requires one-to-one transmission, and the transmission process needs to be verified. This type of data transmission method is set as the transmission control protocol.

[0110] (2) A method for constructing a 3D electronic sand table model based on the fusion of photographic and lidar data was designed. The flowchart of the method can be found in [link to flowchart]. Figure 2As shown, firstly, images and radar data captured by multiple unmanned platforms are acquired. Then, sparse matching of images based on GPU is used to perform relative positioning of multiple images without positioning information. Next, GPS data from the camera sites is used to perform absolute qualitative analysis of the images. Finally, dense matching of point clouds is used to generate a 3D electronic sand table model of the target scene.

[0111] The construction of a three-dimensional electronic sand table model includes the following steps:

[0112] 2-1) Obtain image files of the target ground area captured by the unmanned platform, including either video or radar images;

[0113] 2-2) Perform sparse matching on the image files of the target ground area to achieve multi-image relative positioning between the target ground area image without positioning information and the radar image;

[0114] 2-3) Perform absolute orientation on the image files of the target ground area captured by the unmanned platform to obtain images with absolute coordinates. Then, perform matching point purification on the images with absolute coordinates to obtain the absolute coordinates of pixels in two adjacent frames, thus completing the absolute orientation of the target ground area.

[0115] 2-4) Using a dense point cloud matching algorithm, a three-dimensional electronic sand table model of the target area scene is generated based on the absolute coordinates of pixels in multiple adjacent frames of images.

[0116] The steps of the GPU-based sparse matching algorithm for images are as follows: Figure 3 As shown, it includes the following steps:

[0117] a. Image files captured by an unmanned platform are convolved with a Gaussian function and then low-pass filtered to generate Gaussian pyramid images and Gaussian difference images;

[0118] b. Perform feature point detection on the Gaussian pyramid image and the difference of Gaussian image to obtain the extreme point detection and sub-pixel level localization of feature points in the scale space of the two images;

[0119] c. Obtain the main direction of the feature points in step (2);

[0120] d. Based on the main direction of the acquired feature points, construct SIFT feature descriptors and perform descriptor matching to achieve relative positioning between the unmanned platform and the target ground area.

[0121] The steps of the point cloud dense matching algorithm are as follows: Figure 4 As shown:

[0122] (1) From the image of the target ground area taken by the unmanned platform, select SIFT feature descriptors to form a matching point set, and randomly select a set of matching point pairs from the initial matching points within the set range;

[0123] (2) Using the selected matching point pairs, calculate the essential matrix model between two images with relatively adjacent frames, i.e.:

[0124] E=T R

[0125] E represents the essential matrix model, T represents the camera's movement matrix transformation when capturing two images in adjacent frames, and R represents the camera's selection matrix transformation when capturing two images in adjacent frames.

[0126] (3) Determine the distance threshold using the reference image with GPS data from the ground control station; based on the distance threshold, obtain the interior points that conform to the essential matrix model;

[0127] The distance threshold is the distance value from a pixel to the actual target ground area; that is, pixels with a distance less than the corresponding distance threshold are interior points of the essential matrix model, and pixels with a distance greater than the corresponding distance threshold are exterior points of the essential matrix model.

[0128] (4) Calculate the ratio of the number of interior points to the total number of matching points, and update the maximum number of iterations based on the ratio of the number of interior points to the total number of matching points in the overall matching pairs.

[0129]

[0130] Where, N max The maximum number of iterations is represented by 'a', the number of interior points is represented by 'm', the total number of matched points is represented by 'GSD', and the distance threshold is represented by 'GSD'.

[0131] (5) Repeat steps (1) to (3) until a matching point that meets the quantity setting value is found;

[0132] That is, the number of interior points that satisfies the distance threshold condition, i.e.:

[0133] n = S / GSD 2

[0134] Where n represents the number of inliers that satisfy the distance threshold condition, S represents the area of ​​the ground region of the target image, and GSD represents the distance threshold;

[0135] (6) Based on the essential matrix between two adjacent frames to be matched, obtain the Simpson error of all matching points, i.e.:

[0136]

[0137] Where S[f] represents the Simpson error, and x1 and x2 represent the values ​​in the two graph essence matrices, respectively;

[0138] (7) Remove all outliers in the matching pair whose Simpson error is greater than the distance threshold, retain the remaining matching point pairs, and finally purify the matching point pairs so that two adjacent images can be stitched together, thereby obtaining the absolute coordinates of pixels in multiple sets of two adjacent frames and completing the absolute orientation of the target ground area.

[0139] The process involves obtaining the absolute coordinates of pixels within multiple adjacent frames, i.e.:

[0140] M i,j =P 0,0 +GSD×W i,j

[0141] Among them, M i,j P represents the latitude and longitude coordinates of pixels i and j in two adjacent image frames. 0,0 W represents the GPS coordinates of the reference image. i,j GSD represents the distance between a pixel and the reference image.

[0142] The essential matrix between the two images is determined through the above process. Based on this essential matrix, all outliers with an error greater than the threshold in the matching pair set are removed, and the inliers of the remaining matching point pairs are retained, thus completing the purification of the matching point pairs.

[0143] (3) Figure 5 The diagram shown illustrates the flowchart design of the situation interaction method based on a 3D electronic sand table according to the present invention. Specifically, the situation interaction method based on a 3D electronic sand table is as follows:

[0144] First, the 3D electronic sand table file (slpk format) is imported into the ArcGIS-based program. Second, a TCP server conforming to the situational data transmission method is constructed, and multiple unmanned platforms operating simultaneously are connected to this server. Then, a 3D platform model with 6 degrees of freedom pose attributes is created, and the pose data sent by the unmanned platforms is processed and converted into latitude, longitude, altitude, pitch, roll, and heading angle data. These 6 sets of values ​​are then attached to the unmanned platform model on the map. Finally, the model size is adjusted to complete the pose mapping display of the unmanned platforms under the 3D electronic sand table.

[0145] (4) Design a process for cross-domain environmental information fusion and collaborative positioning, such as... Figures 6-7 As shown:

[0146] The unmanned platform acquires raw environmental perception data, namely:

[0147] The aerial unmanned platform collects laser data of the target area through a laser sensor and visual data of the target area through a camera. At the same time, the ground mobile unmanned platform also collects laser data of the target area through a laser sensor and visual data of the target area through a camera. That is, the aerial unmanned platform and the ground mobile unmanned platform work together to collect laser-visual data.

[0148] Both aerial unmanned platforms and ground-based mobile unmanned platforms acquire perception perspective and orientation data through positioning devices.

[0149] The processed data will be transmitted according to the corresponding transmission rules:

[0150] The perception perspective and orientation data of the unmanned platform are transmitted in the form of sensor data; laser-vision data are transmitted in the form of streaming data.

[0151] The unification of the description format of raw environmental perception data is specifically as follows:

[0152] The streaming data of laser and vision data from unmanned platforms are jointly described using latitude and longitude coordinates. The description format is unified as latitude and longitude coordinates f(longitude, latitude), which represents the dataset of laser and vision detection at a certain latitude and longitude.

[0153] The latitude and longitude coordinates of the laser-vision data collected by the aerial unmanned platform and the ground mobile unmanned platform were unified, and the error was calibrated, namely:

[0154] |D| 2 =δ 2 +ε 2

[0155] Where D represents the calibration error, δ represents the laser latitude and longitude error, and ε represents the visual data positioning error;

[0156] Step 5), specifically as follows: Figure 7 The diagram shows the flowchart for achieving collaboration between raw environmental perception data and a 3D electronic sand table model:

[0157] The unmanned platform generates a 3D electronic sand table model of the target area scene from the raw environmental perception data, and subtracts it from the 3D platform model by dimension. Then, using a Kalman-type optimization filtering strategy, it obtains the pose estimate of the unmanned platform.

[0158]

[0159] Among them, P [a,b,…,n] Let W represent the pose estimate of the unmanned platform at the nth rank (a, b, ..., n). [a,b,…,n] This represents the image file of the original target ground area collected by the unmanned platform of the a, b...nth generation. This represents the Kalman fusion and normalization algorithm, F. [a,b,…,n] The method for generating a three-dimensional electronic sand table in step 2) is denoted by D, where D represents the three-dimensional platform model.

[0160] The above are merely specific steps of the present invention and do not constitute any limitation on the scope of protection of the present invention; it can be extended to various fields of autonomous take-off and landing of UAVs in various sea areas, and all technical solutions formed by equivalent transformation or equivalent substitution fall within the scope of protection of the present invention; the parts of the present invention not described in detail belong to the well-known technology in the field.

Claims

1. A method for integrated information processing across multiple unmanned platforms, characterized in that, Includes the following steps: 1) Establish transmission rules for each data type in multiple unmanned platforms and the corresponding ground control station; send control commands to multiple unmanned platforms through the ground control station, and uniformly receive formatted data returned from multiple unmanned platforms; 2) The ground control station generates a three-dimensional electronic sand table model of the target area scene using vertical photography; 3) The ground control station establishes a three-dimensional unmanned platform model based on a three-dimensional electronic sand table model and has multi-degree-of-freedom pose attributes, and maps the position and attitude of all unmanned platforms connected to the ground control station into the three-dimensional electronic sand table model, that is, generates a pose mapping display of unmanned platforms under the three-dimensional electronic sand table model. 4) The unmanned platform acquires raw environmental perception data and sends it to the ground control station for processing; the processed data is transmitted according to the corresponding transmission rules; and the description format of the raw environmental perception data is standardized. 5) Obtain the pose estimation value of the unmanned platform based on the environmental perception raw data in a unified description form; load the pose estimation value of the unmanned platform into the three-dimensional electronic sand table model based on the environmental perception raw data to realize the cooperative pose estimation of the unmanned platform, thereby achieving the purpose of cooperative positioning.

2. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, The data types include: streaming data, sensor data, control data, task data, status report data, notification report data, situational awareness information data, and target location information; The streaming data can be any one or more of visible light images, infrared images, and radar images; Sensor data can be any one or more of static images, dynamic images, and sonar data monitored and transmitted by the unmanned platform as appropriate. Control data refers to data used to manipulate unmanned platforms or their payloads. The mission data includes planning, task, and follow-up data for unmanned platforms and their payloads. The status report data consists of status reports of the unmanned platform before and after performing the task; Notification reports are short message notifications sent by unmanned platforms; The situational awareness information data is the situational awareness information transmitted back by the unmanned platform; The target location information is the data transmitted back by the unmanned platform after it discovers the target and identifies its location.

3. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, Step 1), establishing the transmission rules for each data type in the multi-unmanned platform and the corresponding ground control station, specifically involves: The transmission rules for streaming data adopt the UDP protocol, which does not require feedback during the transmission process; The FTP transmission protocol, which requires a verification process during transmission, is used for the transmission of sensor data. The TCP transmission control protocol is used for the transmission of control data, task data, notification report data, and target location information. Status report data is sent cyclically between the TCP Transmission Control Protocol and the User Datagram Protocol. Situational awareness information data includes two or more of the following: streaming data, sensor data, control data, mission data, notification-type report data, target location information, and status report data, and follows the transmission rules corresponding to the respective data types.

4. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, Step 2) includes the following steps: 2-1) Acquire image files of the target ground area captured by the unmanned platform, including either video or radar images; 2-2) Perform sparse matching on the image files of the target ground area to achieve multi-image relative positioning of the target ground area image without positioning information and radar image; 2-3) Perform absolute orientation on the image files of the target ground area captured by the unmanned platform to obtain images with absolute coordinates. Then, perform matching point purification on the images with absolute coordinates to obtain the absolute coordinates of pixels in two adjacent frames, thus completing the absolute orientation of the target ground area. 2-4) Using a dense point cloud matching algorithm, a three-dimensional electronic sand table model of the target area scene is generated based on the absolute coordinates of pixels in multiple adjacent frames of images.

5. The method for integrated information processing of multiple unmanned platforms according to claim 4, characterized in that, Step 2-2) specifically involves: a. Image files captured by an unmanned platform are convolved with a Gaussian function and then low-pass filtered to generate Gaussian pyramid images and Gaussian difference images; b. Perform feature point detection on the Gaussian pyramid image and the difference of Gaussian image to obtain the extreme point detection and sub-pixel level localization of feature points in the scale space of the two images; c. Obtain the main direction of the feature points in step (2); d. Based on the main direction of the acquired feature points, construct SIFT feature descriptors and perform descriptor matching to achieve relative positioning between the unmanned platform and the target ground area.

6. The method for integrated information processing of multiple unmanned platforms according to claim 4, characterized in that, Steps 2-3 are specifically as follows: (1) From the image of the target ground area taken by the unmanned platform, select SIFT feature descriptors to form a matching point set, and randomly select a set of matching point pairs from the initial matching points within the set range; (2) Using the selected matching point pairs, calculate the essential matrix model between two images with relatively adjacent frames, i.e.: ； E represents the essential matrix model, T represents the camera's movement matrix transformation when capturing two images in adjacent frames, and R represents the camera's selection matrix transformation when capturing two images in adjacent frames. (3) Determine the distance threshold using the reference image with GPS data from the ground control station; based on the distance threshold, obtain the interior points that conform to the essential matrix model; The distance threshold is the distance value from a pixel to the actual target ground area; that is, pixels with a distance less than the corresponding distance threshold are interior points of the essential matrix model, and pixels with a distance greater than the corresponding distance threshold are exterior points of the essential matrix model. (4) Calculate the ratio of the number of inliers to the total number of matched points, and update the maximum number of iterations based on the ratio of the number of inliers to the total number of matched points in the overall matching pairs. ； in, This represents the maximum number of iterations, 'a' represents the number of interior points, and 'm' represents the total number of matched points. Indicates the distance threshold; (5) Repeat steps (1) to (3) until a matching point that meets the quantity setting value is found; That is, the number of interior points that satisfies the distance threshold condition, i.e.: ； Where n represents the number of inliers that satisfy the distance threshold condition, S represents the area of ​​the ground region of the target image, and GSD represents the distance threshold; (6) Based on the essential matrix between two adjacent frames to be matched, obtain the Simpson error of all matching points, i.e.: ； in, Indicates Simpson's error. , These represent the values ​​within the essential matrices of the two graphs, respectively. (7) Remove all outliers in the matching pair whose Simpson error is greater than the distance threshold, retain the remaining matching point pairs, and finally purify the matching point pairs so that two adjacent images can be stitched together, thereby obtaining the absolute coordinates of pixels in multiple sets of two adjacent frames and completing the absolute orientation of the target ground area. The process involves obtaining the absolute coordinates of pixels within multiple adjacent frames, i.e.: ； in, This represents the latitude and longitude coordinates of pixels i and j in two adjacent image frames. This refers to the GPS coordinates of the reference image. This represents the pixel distance from the reference image. This indicates the distance threshold.

7. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, Step 3) includes the following steps: 3-1) Construct a TCP server that conforms to the situation data transmission method using ArcGIS service for the 3D electronic sand table model, and connect multiple unmanned platforms operating simultaneously to this server; 3-2) Create a 3D platform model with multiple degrees of freedom pose attributes, load the 3D platform models of multiple unmanned platforms into the 3D electronic sand table model; and perform data calculation on the pose data sent by the unmanned platforms and convert it into latitude and longitude data as well as pitch angle, roll angle and heading angle data, and attach multiple sets of pose attribute values ​​to the unmanned platform model in the 3D platform model; 3-3) Adjust the size of the 3D platform model to complete the pose mapping display of the unmanned platform under the 3D electronic sand table model.

8. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, Step 4) includes the following steps: The unmanned platform acquires raw environmental perception data, namely: The aerial unmanned platform collects laser data of the target area through a laser sensor and visual data of the target area through a camera. At the same time, the ground mobile unmanned platform also collects laser data of the target area through a laser sensor and visual data of the target area through a camera. That is, the aerial unmanned platform and the ground mobile unmanned platform work together to collect laser-visual data. Both aerial unmanned platforms and ground-based mobile unmanned platforms acquire perception perspective and orientation data through positioning devices.

9. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, The process of transmitting the processed data according to the corresponding transmission rules is as follows: The perception perspective and orientation data of the unmanned platform are transmitted in the form of sensor data; laser-vision data are transmitted in the form of streaming data. The unification of the description format of raw environmental perception data is specifically as follows: The streaming data of laser and vision data from the unmanned platform are jointly described using latitude and longitude coordinates, with the description format unified as latitude and longitude coordinates. This value represents the dataset of laser and visual detection at a certain latitude and longitude.

10. The method for integrated information processing of multiple unmanned platforms according to claim 1, characterized in that, Step 5) specifically involves: The unmanned platform generates a 3D electronic sand table model of the target area scene from the raw environmental perception data, and subtracts it from the 3D platform model by dimension. Then, using a Kalman-type optimization filtering strategy, it obtains the pose estimate of the unmanned platform. ； in, Let represent the pose estimate of the unmanned platform at the ith (a, b, ..., n) position. This represents the image file of the original target ground area collected by the unmanned platform of the a, b...nth generation. This refers to the Kalman fusion and normalization algorithm. This describes the method for generating a three-dimensional electronic sand table in step 2). This represents a three-dimensional platform model.

Images