Unattended multi-sensor collaborative operation system and method

Abstract

The invention belongs to the field of data mining, and particularly relates to an unattended multi-sensor collaborative operation system and method. The system mainly comprises ten hardware modules, a radar processing equipment module mainly performs data fusion on radar equipment, optoelectronic equipment and AIS equipment, and provides fused data to perform data fusion with sonar processing equipment and radio processing equipment; the sonar equipment module is mainly used for generating sonar data; the sonar processing device receives and processes sonar data and supports data fusion with the radar processing device and the radio processing device. The radio equipment module mainly generates radio data; the radio processing equipment receives and processes radio data and supports data fusion with the radar processing equipment and the sonar processing equipment; the cooperative control server completes fusion work of data generated by the radar processing device, the sonar processing device and the radio processing device; and the database server completes storage of fusion data generated by the cooperative control server.

Description

Technical Field

[0001] The invention belongs to the field of data mining, and in particular relates to an unattended multi-sensor collaborative operation system and method. Background Art

[0002] With the rapid development of artificial intelligence technology, navigation technology is mainly used in communication, control, positioning and other related systems, which can be called navigation automation. The more perfect the automation function of navigation technology is, the more help it will provide for navigation. At present, navigation planning is generally based on navigation experience. If you encounter emergencies such as bad weather during navigation, you can only change the original navigation route according to the actual navigation conditions, but this will bring certain risks to the navigation and prolong the navigation progress. In view of this hidden danger, it is difficult to meet the needs of modern navigation if you only rely on a single sensor to obtain information.

[0003] At present, large ocean-going vessels are not only equipped with navigation radars, but also with sonar equipment, radio equipment and optoelectronic equipment. They are also equipped with AIS (Automatic Identification System), ADS-B signal receivers, GPS receivers and other navigation support equipment, as well as hull attitude, wind force and direction, waves, current speed, temperature and salinity sensor equipment. The comprehensive application of multiple sensors in the ship can effectively improve the navigation safety of the ship.

[0004] Various sensor devices have their own advantages in different aspects such as detection accuracy, breadth, depth, and timeliness. Combining multi-source information fusion technology and integrating the measurement information of multiple sensors can make the estimation of target state information more accurate, timely, and comprehensive, ensuring navigation safety and information effectiveness, and realizing information sharing and complementary advantages of heterogeneous sensors. This has become a very important research topic in the field of navigation.

[0005] The present invention aims to integrate multi-source heterogeneous data such as navigation radar, ship-borne optoelectronic equipment, sonar detection equipment, and radio detection equipment, and to identify, fuse, organize, store, and report the same target when the information capacity is huge and the relationship between the information is complex. Summary of the invention

[0006] The purpose of the present invention is to provide an unattended multi-sensor collaborative operation system to complete the automatic collaborative operation of radar equipment, optoelectronics, AIS, sonar equipment, and radio equipment for key targets and the matching data fusion and storage.

[0007] The technical solution adopted by the present invention is:

[0008] An unattended multi-sensor collaborative operation system, including radar equipment, optoelectronic equipment, AIS equipment, radar processing equipment, sonar equipment, radio equipment, sonar processing equipment, radio processing equipment, collaborative control server, and database server;

[0009] Radar equipment mainly generates raw radar scan data to support radar, optoelectronics, and AIS fusion within radar processing equipment;

[0010] Optoelectronic equipment generates raw visible light and infrared video data to support radar and optoelectronic fusion within radar processing equipment;

[0011] AIS equipment receives surrounding AIS information and generates AIS data to support radar and AIS fusion inside radar processing equipment;

[0012] Radar processing equipment mainly performs data fusion on radar equipment, optoelectronic equipment, and AIS equipment, and provides fused data to support data fusion with sonar processing equipment and radio processing equipment;

[0013] Sonar equipment mainly produces sonar data;

[0014] The sonar processing equipment receives and processes the sonar data to support data fusion with the radar processing equipment and the radio processing equipment;

[0015] Radio equipment primarily generates radio data;

[0016] The radio processing equipment receives and processes radio data, and supports data fusion with radar processing equipment and sonar processing equipment;

[0017] The collaborative control server completes the integration of data generated by radar processing equipment, sonar processing equipment, and radio processing equipment;

[0018] The database server completes the storage of the fusion data generated by the collaborative control server.

[0019] Furthermore, in the unattended multi-sensor collaborative operation system as mentioned above, the radar processing equipment completes the data fusion of the radar equipment, the optoelectronic equipment, and the AIS equipment, specifically including: position matching of AIS data and radar data; azimuth matching and distance evaluation of radar data and optoelectronic video data; re-fusion of the fused data of AIS data and radar data with the fused data of optoelectronic video data and radar data; and quality evaluation of the fusion results.

[0020] Furthermore, the collaborative control server mainly completes the coordinated scheduling of the above-mentioned devices, and the scheduling process mainly includes:

[0021] Radar processing equipment, sonar processing equipment, and radio processing equipment trigger a collaborative mode;

[0022] The collaborative control server responds to the collaboration and requests the radar processing device, the sonar processing device, and the radio processing device to feedback data;

[0023] The collaborative control server performs preliminary fusion after receiving the feedback data;

[0024] Evaluate the fusion results, remove redundant, invalid, and abnormal data, and retain high-quality data to store in the database server;

[0025] During the warehousing process, a local data set is generated according to the predefined format.

[0026] An unattended multi-sensor collaborative operation method comprises the following steps:

[0027] Radar equipment generates raw radar scanning data, optoelectronic equipment generates raw visible light and infrared video data, and AIS equipment receives surrounding AIS information and generates AIS data;

[0028] The radar processing equipment performs data fusion on the data generated by the radar equipment, optoelectronic equipment, and AIS equipment;

[0029] The sonar equipment generates sonar data, and the sonar processing equipment receives and processes the sonar data to support data fusion with the radar processing equipment and the radio processing equipment;

[0030] The radio equipment generates radio data, and the radio processing equipment receives and processes the radio data to support data fusion with the radar processing equipment and the sonar processing equipment;

[0031] The collaborative control server fuses the data generated by the radar processing equipment, sonar processing equipment, and radio processing equipment;

[0032] The database server completes the storage of the fusion data generated by the collaborative control server.

[0033] The beneficial effects of the present invention are:

[0034] The present invention realizes an unattended multi-sensor collaborative operation system, which can collect, fuse and store multi-source data of key targets of surrounding situations spontaneously by sensors without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the overall flow chart of the unattended multi-sensor collaborative operation system;

[0036] Figure 2 It is the radar, AIS, and optoelectronic fusion flow chart;

[0037] Figure 3 Flowchart of the unattended multi-sensor collaborative operation method. DETAILED DESCRIPTION

[0038] The following is an introduction to an unattended multi-sensor collaborative operation system provided by the present invention in conjunction with the accompanying drawings and embodiments:

[0039] An embodiment of the present invention provides an unattended multi-sensor collaborative operation system. It mainly includes ten hardware components, such as Figure 1 , respectively, they are radar equipment 1, optoelectronic equipment 2, AIS equipment 3, radar processing equipment 4, sonar equipment 5, sonar processing equipment 6, radio equipment 7, radio processing equipment 8, collaborative control server 9, and database server 10.

[0040] The radar device 1 mainly generates raw radar scanning data to support the radar, optoelectronics, and AIS fusion inside the radar processing device 4 .

[0041] The optoelectronic device 2 generates raw visible light and infrared video data to support radar and optoelectronic fusion inside the radar processing device 4 .

[0042] The AIS device 3 receives surrounding AIS information and generates AIS data to support radar and AIS fusion inside the radar processing device 4 .

[0043] The radar processing device 4 mainly performs data fusion on the radar device 1, the optoelectronic device 2, and the AIS device 3, including position matching of the AIS data and the radar data; azimuth matching and distance evaluation of the radar data and the optoelectronic video data; re-fusion of the AIS data, the radar fusion data, and the optoelectronic and radar fusion data; and quality evaluation of the fusion results. The fused data is then provided to the sonar processing device 6 and the radio processing device 8 for data fusion.

[0044] The sonar device 5 mainly generates sonar data.

[0045] The sonar processing device 6 receives and processes the sonar data, and supports data fusion with the radar processing device 4 and the radio processing device 8 .

[0046] The radio device 7 mainly generates radio data.

[0047] The radio processing device 8 receives and processes radio data, and supports data fusion with the radar processing device 4 and the sonar processing device 6 .

[0048] The collaborative control server 9 completes the fusion of data generated by the radar processing device 4, the sonar processing device 6, and the radio processing device 8. Specifically, it includes responding to the collaborative request of the radar processing device 4, the sonar processing device 6, and the radio processing device 8 and receiving the feedback data of the radar processing device 4, the sonar processing device 6, and the radio processing device 8; the collaborative control server 9 performs preliminary fusion after receiving the feedback data; evaluates the fusion results, removes timeout, invalid, and abnormal data, and retains high-quality data to store in the database server 10; during the storage process, generates a local data set according to the predefined format.

[0049] The database server 10 completes the storage of the fusion data generated by the collaborative control server 9 .

[0050] By using the unattended multi-sensor collaborative operation system as described above, an unattended multi-sensor collaborative operation method can be implemented, which can be divided into the following three parts:

[0051] (I) Fusion of AIS data, radar data, and optoelectronic data

[0052] like Figure 2 As shown, the fusion data of AIS data, radar data, and optoelectronic data received by the collaborative control server 9 consists of two stages, namely, the fusion of AIS data and radar data; and the fusion of radar data and optoelectronic video data.

[0053] 1. Fusion of AIS data and radar data

[0054] The radar data generated by the radar device 1 is firstly subjected to target detection by the radar processing device 4, and the detection result is subjected to coordinate transformation, specifically, polar coordinates are transformed into rectangular coordinates, and then the rectangular coordinates are transformed into longitude and latitude coordinates;

[0055] The moment when the radar target is successfully detected is taken as the reference time of the target, and after being fused with the AIS data, it is taken as the time of the fused data.

[0056] Compared with radar data, the AIS data reception interval is longer, so the received AIS data needs to be classified and cached according to the unique identifier, and then interpolated at fixed intervals, mainly for position interpolation.

[0057] Based on the interpolated AIS and the radar target after coordinate transformation, position matching is performed. If it fails, the radar target is abandoned. If it succeeds, the fusion result 1 of the AIS device 3 and the radar device 1 is generated, and the fusion result 2 of the optoelectronic device 2 and the radar device 1 is waited.

[0058] 2. Fusion of radar data and optoelectronic video data

[0059] The radar data generated by the radar device 1 is first detected by the radar processing device 4, and this process is the same as the fusion of AIS data and radar data. The coordinate system of the detected target coordinates is transformed, specifically: the polar coordinates centered on the radar are transformed into a rectangular coordinate system, and then the rectangular coordinate system relative to the center of the radar is transformed into a rectangular coordinate system centered on the optoelectronic device 2. In this process, it is necessary to consider that the 0-degree direction and actual position of the radar device 1 are different from the 0-degree direction and actual position of the optoelectronic device 2, and the coordinate system needs to be translated and rotated.

[0060] The moment when the radar target is successfully detected is used as the reference time of the target, and after being fused with the optoelectronic device 2, it is used as the time of the fused data.

[0061] When the optoelectronic device 2 is turned on, the optoelectronic video data generated is firstly processed by the radar processing device 4 for target detection.

[0062] When the radar device 1 is turned on, the azimuth and distance of the target detected by the radar are converted to the coordinate system of the optoelectronic device 2 after the coordinate system transformation. The PTZ parameters of the optoelectronic device 2, specifically the horizontal offset angle, vertical offset angle, and zoom factor, are converted, and the optoelectronic device 2 is called to align with the converted area.

[0063] After the optoelectronic device 2 completes the alignment operation, the radar detection target and the detection target in the current optoelectronic field of view are first matched in orientation (both are in the coordinate system of the optoelectronic device 2). If the match fails, it is abandoned. After the match is successful, the distance between the two is evaluated. If successful, a fusion result 2 is generated. If it fails, the radar target is abandoned.

[0064] In summary, the above two steps generate fusion data 1 of AIS data and radar data, and fusion data 2 of radar data and optoelectronic video data. The two are subject to time constraints. When the time constraints are met, fusion data 3 of the radar processing device 4 is generated and enters the collaborative control server 9, which is used to support data fusion with the sonar processing device 6 and the radio processing device 8 (i.e., collaborative scheduling).

[0065] 2. Collaborative Scheduling

[0066] The following description will be made by taking the radar processing device 4 triggering collaboration as an example. The process of triggering collaboration by the radio processing device 8 and the sonar processing device 6 is consistent with the process of triggering collaboration by the radar processing device 4.

[0067] The process of triggering the collaboration by the radar processing device 4 is as follows: Figure 3 As shown, the specific steps include:

[0068] The radar processing device 4 detects a key target, triggers collaboration, and sends collaboration information to the collaborative response module of the collaborative control server 9;

[0069] The coordinated response module sends the coordinated instruction (including the position and orientation information carried in the coordinated information) to the radar processing device 4, the sonar processing device 6, and the radio processing device 8;

[0070] The radar processing device 4, the sonar processing device 6, and the radio processing device 8 send feedback data to the collaborative processing module of the collaborative control server 9. In this process, the feedback data 3 generated by the radar processing device 4 is Figure 2 The fusion data 3 in the figure is consistent, the feedback data 1 and 2 generated by the sonar processing device 6 and the radio processing device 8 are single data, which can be directly generated by the processing device;

[0071] The collaborative processing module waits for feedback data 1, 2, and 3 to be received. First, it passes the time constraint. If it is within the constraint range, it proceeds to the next step. If a timeout occurs, it means that the data is missing, and the next step is only performed on the feedback data that meets the time constraint.

[0072] Azimuth matching is performed on the feedback data that meets the time constraint. In this process, the azimuth value is based on the azimuth value in the feedback data 3 generated by the radar processing device 4 as the standard value, and the feedback data with abnormal azimuth is eliminated;

[0073] The feedback data that meets the azimuth matching is evaluated for distance. In this process, the distance in the feedback data 3 generated by the radar processing device 4 is used as the standard value, and the feedback data with abnormal distance is eliminated;

[0074] The data after distance assessment meets the constraints in time, azimuth and distance. The feedback data is directly spliced ​​and stored in the database. The fields include radar target position, radio signal azimuth, listening sonar signal azimuth, imaging sonar target position and other information. The position difference, azimuth difference, etc. can be calculated. At the same time, local files are generated in a predefined format to accumulate data sets.

[0075] In the above process, the key target triggering method of processing devices 4, 6, and 8 is as follows:

[0076] 1) How the radar processing device 4 detects key targets

[0077] Mode 1: Passively receive AIS data. When the target type identifier (MMSI) is a target of interest, call the optoelectronic device 2 to aim at the target and trigger collaboration.

[0078] Method 2: Radar active detection. According to the radar target detection results, the latest radar targets are traversed in sequence, and the optoelectronic device 2 is called to aim at the target to trigger coordination.

[0079] 2) How the radio processing device 8 discovers key targets

[0080] Method 1: Use radar reconnaissance means of radio detection equipment to detect radar targets in key frequency bands and key frequency points, and trigger coordination.

[0081] Method 2: TACAN reconnaissance by radio detection equipment. Once a suspected TACAN station signal is found in the open sea, coordination will be triggered.

[0082] Method 3: Radio detection equipment IFF signal reconnaissance means, once the enemy IFF signal is detected in the open sea, coordination will be triggered.

[0083] 3) How the sonar processing device 6 discovers key targets

[0084] Method 1: The signal listening sonar receives the hydroacoustic signal and combines it with the GPS data sent by the submersible to locate the underwater submersible target in real time, predict the water exit position based on the target track, and trigger coordination before the target emerges from the water.

[0085] Method 2: Signal listening sonar receives underwater acoustic signals, locates underwater targets in real time, and performs signal classification and modulation pattern recognition. Once a target signal is detected in the open sea, coordination is triggered.

[0086] Method 3: Active imaging sonar scans the underwater environment and classifies and identifies seabed targets and suspended targets. Once a target signal is detected in the open sea, coordination is triggered.

[0087] (III) Data Collection

[0088] All the above data, including fused data, original data, etc., are stored in the database. The system supports multiple databases such as MySQL, Sqllite, and self-developed memory databases. At the same time, according to the needs, the local files can be stored in a predetermined format while being pre-set for storage, which can form a multi-field, multi-source, multi-dimensional, and multi-modal data set for subsequent research and use. The fused data of the present invention supports multiple types of databases when being stored in the database, and supports the export of information in multiple types of data formats.

[0089] The present invention does not elaborate on some of the well-known technologies that belong to those skilled in the art.

[0090] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. An unattended multi-sensor collaborative operation system, characterized in that: Including radar equipment, optoelectronic equipment, AIS equipment, radar processing equipment, sonar equipment, radio equipment, sonar processing equipment, radio processing equipment, collaborative control server, database server; Radar equipment is used to generate raw radar scan data; Optoelectronic equipment is used to generate raw visible light and infrared video data; AIS equipment is used to receive surrounding AIS information and generate AIS data; Radar processing equipment is used to fuse data generated by radar equipment, optoelectronic equipment, and AIS equipment, and provide fused data to support data fusion with sonar processing equipment and radio processing equipment; Sonar equipment is used to generate sonar data; Sonar processing equipment is used to receive and process sonar data to support data fusion with radar processing equipment and radio processing equipment; Radio equipment for generating radio data; Radio processing equipment is used to receive and process radio data, and supports data fusion with radar processing equipment and sonar processing equipment; The collaborative control server is used to complete the integration of data generated by radar processing equipment, sonar processing equipment, and radio processing equipment; The database server is used to store the fusion data generated by the collaborative control server.

2. The unmanned multi-sensor collaborative operation system according to claim 1, characterized in that: The radar processing device performs data fusion on the data generated by the radar device, the optoelectronic device, and the AIS device, including: fusing AIS data with radar data; fusing radar data with optoelectronic video data; fusing the fused data of AIS data and radar data with the fused data of optoelectronic video data and radar data again; and performing quality assessment on the fusion results.

3. The unmanned multi-sensor collaborative operation system according to claim 2, characterized in that: The fusion of AIS data and radar data includes position matching of AIS data and radar data; the fusion of radar data and optoelectronic video data includes azimuth matching and distance evaluation of radar data and optoelectronic video data.

4. The unmanned multi-sensor collaborative operation system according to claim 1, characterized in that: The collaborative control server completes the coordinated scheduling of radar processing equipment, sonar processing equipment, and radio processing equipment. The scheduling process includes: Radar processing equipment, sonar processing equipment, and radio processing equipment trigger a collaborative mode; The collaborative control server responds to the collaboration and requests feedback data from the radar processing device, the sonar processing device, and the radio processing device; The collaborative control server receives the feedback data and performs integration; Evaluate the fusion results, eliminate redundant, invalid, and abnormal data, and retain high-quality data to store in the database server; During the warehousing process, a local data set is generated according to the predefined format.

5. The method according to claim 4, characterized in that The collaborative control server receives the feedback data and performs integration, including: First, a time constraint is set. If it is within the constraint range, the next step is continued. If a timeout occurs, it means that the data is missing, and the next step is only performed on the feedback data that meets the time constraint. Performing azimuth matching on feedback data that meets the time constraint, wherein the azimuth value is based on the azimuth value in the feedback data generated by the radar processing device, and rejecting feedback data with abnormal azimuth; Perform distance evaluation on feedback data that meets the azimuth matching requirement, using the distance in the feedback data generated by the radar processing device as the standard value, and discard feedback data with abnormal distances. The data that meets the distance assessment and meets the constraints in time, direction and distance are directly spliced ​​and stored in the database.

6. The unmanned multi-sensor collaborative operation system according to claim 1, characterized in that: The fusion data generated by the collaborative control server supports multiple types of databases when it is stored in the database, and also supports the export of information in multiple types of data formats.

7. An unattended multi-sensor collaborative operation method, characterized in that: The following steps are involved: Radar equipment generates raw radar scanning data, optoelectronic equipment generates raw visible light and infrared video data, and AIS equipment receives surrounding AIS information and generates AIS data; Radar processing equipment performs data fusion on data generated by radar equipment, optoelectronic equipment, and AIS equipment; Sonar equipment generates sonar data, and sonar processing equipment receives and processes the sonar data to support data fusion with radar processing equipment and radio processing equipment; Radio equipment generates radio data, and radio processing equipment receives and processes the radio data to support data fusion with radar processing equipment and sonar processing equipment; The collaborative control server integrates the data generated by radar processing equipment, sonar processing equipment, and radio processing equipment; The database server completes the storage of the fusion data generated by the collaborative control server.

8. The method according to claim 7, characterized in that The radar processing device performs data fusion on the data generated by the radar device, the optoelectronic device, and the AIS device, including: fusing AIS data with radar data; fusing radar data with optoelectronic video data; fusing the fused data of AIS data and radar data with the fused data of optoelectronic video data and radar data again; and performing quality assessment on the fusion results.

9. The method according to claim 7, characterized in that The collaborative control server completes the coordinated scheduling of radar processing equipment, sonar processing equipment, and radio processing equipment. The scheduling process includes: Radar processing equipment, sonar processing equipment, and radio processing equipment trigger a collaborative mode; The collaborative control server responds to the collaboration and requests feedback data from the radar processing device, the sonar processing device, and the radio processing device; The collaborative control server receives the feedback data and performs integration; Evaluate the fusion results, eliminate redundant, invalid, and abnormal data, and retain high-quality data to store in the database server; During the warehousing process, a local data set is generated according to the predefined format.

10. The method according to claim 9, characterized in that The collaborative control server receives the feedback data and performs integration, including: First, a time constraint is set. If it is within the constraint range, the next step is continued. If a timeout occurs, it means that the data is missing, and the next step is only performed on the feedback data that meets the time constraint. Performing azimuth matching on feedback data that meets the time constraint, wherein the azimuth value is based on the azimuth value in the feedback data generated by the radar processing device, and rejecting feedback data with abnormal azimuth; Perform distance evaluation on feedback data that meets the azimuth matching requirement, using the distance in the feedback data generated by the radar processing device as the standard value, and discard feedback data with abnormal distances. The data that meets the distance assessment and meets the constraints in time, direction and distance are directly spliced ​​and stored in the database.

Images