Sea surface target radar information processing method and system based on integrated machine learning

Abstract

The invention provides a sea surface target information processing system and method based on integrated deep learning, and the system comprises a preprocessing module which is used for carrying out the echo processing of radar echo data, and obtaining the processed radar echo data; the target detection module is used for performing target detection by using an end-to-end convolutional neural network model of target detection based on the preprocessed radar echo data to obtain a target trace point; the target tracking module is used for carrying out target trace point tracking by utilizing an end-to-end convolutional neural network model of target tracking based on the obtained target trace point, so as to obtain multi-target track information; the related fusion module is used for performing target fusion by utilizing a fused end-to-end convolutional neural network model based on the multi-channel target track information to obtain fused target track information; and the display control module is used for carrying out display control by utilizing a display control end-to-end convolutional neural network model based on the fused target track information.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-module radar information processing, and in particular to a method and system for processing sea surface target radar information based on integrated machine learning.

[0002] This patent relates to a multi-module radar information processing method. The system input is: navigation radar echo. The information processing module adopts an improved machine learning method and combines the characteristics of radar echo to realize an integrated new multi-module deep learning structure. It completes the target detection, tracking, correlation, fusion, display and control of radar echo in layers. It adopts information feedback structure and AIS target information supervised learning. The system realizes intelligent information processing of targets in various sea environments and realizes system functions such as target detection, tracking, fusion, display and control in an integrated manner. Background Art

[0003] Traditional radar detection algorithms face many challenges, such as weak target detection under strong clutter and parameter adaptation in complex and changing environments. However, in theory, these problems are not unsolvable. Take parameter adaptation in complex and changing environments as an example: for an environment with poor detection effect, adjust the algorithm parameters to the optimal effect for this environment; record the algorithm parameters and the characteristics of the environment at this time; when encountering such an environment in the future, call the recorded algorithm parameters; in theory, as long as enough types of environments are recorded and processed in a targeted manner, parameter adaptation in complex and changing environments can be achieved. Parameter adaptation is still a major problem today because environmental conditions are ever-changing and the above process is too cumbersome and time-consuming to achieve theoretical exhaustion.

[0004] The root of the problem is that the essence of traditional radar detection algorithms is based on artificial feature operators (for example, the clutter map extraction algorithm in the detection algorithm is a feature operator extracted artificially based on empirical rules). It usually performs poorly outside the theoretical scope of application of the artificial feature operator used. In other words, traditional radar detection algorithms do not have generalization capabilities. In addition, even within the theoretical scope of application, it is very difficult to accurately estimate the distribution model of noise and clutter and grasp their dynamic characteristics due to the limitations of simulation technology and theoretical level, and the designed artificial feature operators may not necessarily achieve good results.

[0005] Radar small target detection in a strong sea clutter environment and target behavior estimation with incomplete information are global problems. After decades of research, various clutter statistical models have been proposed, and even nonlinear models, chaos models, fractal theory and other methods have been applied. All models and methods seem to have answers in shallow functional relationships, and the actual target detection performance has not been greatly improved. These technical improvements have basically reached the technical top.

[0006] With the application of artificial intelligence and machine learning methods in game theory, image recognition and other aspects, especially the deep learning architecture using multi-layer convolutional neural networks, extremely complex (deep) functional relationships have been achieved, and some even contain "latent knowledge" or "dark knowledge" that humans cannot understand. Machine learning is a brand-new cognitive methodology that humans cannot really understand, but can be tested in practice. People's belief that "practice is the criterion for testing truth" has once again been greatly improved in wisdom. Humans will face new choices: whether to build a community with a shared future through human-machine complementarity or to insist that machines must explain to people the deductive and inductive process to be the truth.

[0007] Deep learning is a collection of algorithms that use multi-layer nonlinear transformations to model highly complex data. It has the advantages of automatically extracting data features and strong migration and generalization capabilities. Deep learning has achieved excellent practical results in many fields such as search technology, data mining, optical image recognition, speech recognition, natural language processing, and personalized recommendations, far exceeding previous traditional technologies. Radar target detection is essentially a binary classification task of radar targets and clutter backgrounds, and classification and recognition tasks are exactly what deep learning excels at.

[0008] Machine learning has made great progress in image processing, but there are many differences between radar information and image information, such as pulse detection mode, sector resolution area, frame scanning, accuracy changes with distance, azimuth cycle, unstable distance boundary, huge target volume, etc. The causes of sea clutter are very complex, and intelligent improvement is needed in suppressing sea clutter. Target track tracking reflects the front-to-back correlation between the target contour and the center, and whether it conforms to the motion characteristics of the target. The fusion of multi-station and multi-sensor target tracks is an effective method for comprehensive processing based on compensation of the errors of each system. Display control and selective information feedback are the display output of the system and the overall optimization of the system. Each function of the system can be implemented individually and then integrated. Since the relevant information output by each module needs to be fed back to the previous level information, it is impossible to achieve system optimization, and it is difficult to achieve effective solutions to various difficult problems. First of all, the amount of calculation is too large to be real-time, which may be fatal to radar information processing. Since deep learning calculations need to run on parallel computers, this brings huge cost pressure to the entire system. It needs to be run on ordinary industrial computers. It is very necessary to compress the depth of each deep network, adopt cascade form and supervised training of AIS target tracks, which can greatly save learning calculations. Then, we refine the traditional information processing methods and optimize the network structure in a direction during learning and training. In short, it will be a very meaningful development direction to reduce the amount of calculation and optimize the system structure in a targeted manner and further improve the system performance.

[0009] Based on radar echoes (including various clutter, reflections, noise, interference, various targets, etc.), the convolutional neural network and deep learning are used to perform preliminary screening of radar echoes, preliminarily separate targets (focus of attention) and other categories, introduce the (target) attention mechanism (Attent ion), extract the characteristics of echoes that are somewhat similar to targets in image processing, and refine quasi-target echoes; then use the middle-level convolutional neural network and deep learning, refer to the motion characteristics of multi-frame echoes, the degree of echo aggregation, etc., filter out quasi-target echoes that do not belong to the target, and splice the quasi-target echoes of the previous and next frames into the target track; use the upper-level convolutional neural network and deep learning, refer to other (such as AIS, optoelectronics, infrared, laser ranging, other radars' own layer information, etc.) target track information and sea lanes and regional information, effectively separate approaching targets, state transitions (movement to stillness, or stillness to movement, sudden acceleration or deceleration, turning, etc.) Special targets such as targets can be correlated and fused to realize the output of various target tracks (small boats, large ships, tugboats, speedboats, buoys, anchorage targets, etc.). For target tracks that cannot be effectively correlated and fused, and target tracks with particularly complex tracks and too close correlation to be distinguished, feedback information is needed to continue learning, and the information is fed back to the previous learning network for reference; some parts of each learning module may introduce other external information, such as AIS, optoelectronics, feedback from the back of this processing module, other radar target information, etc., as machine learning with reference, the results are annotated; the learning results are automatically inferred according to the network structure for the parts without reference. The entire processing system can automatically complete the system structure and parameter optimization under display control and human intervention to keep the system in the best state.

[0010] Patent document CN109934088A (application number: 201910024532.1) discloses a method for identifying sea vessels based on deep learning, comprising the following steps: step one, using a drone to shoot a video of a sea vessel as a training video; step two, performing frame decoding on the training video obtained in step one to obtain multiple ship images to form a training data set; step three, training the Faster R-CNN algorithm to obtain the final Faster R-CNN model; step four, using a drone to shoot a video of a sea vessel as a test video; step five, using the Faster R-CNN model obtained in step three to detect the detection data set obtained in step four, and finally outputting the corresponding ship detection result. Summary of the invention

[0011] In view of the defects in the prior art, the object of the present invention is to provide a sea surface target radar information processing method and system based on integrated machine learning.

[0012] A sea surface target information processing system based on integrated deep learning provided by the present invention includes: a preprocessing module, a target detection module, a target tracking module, a correlation fusion module and a display control module;

[0013] The preprocessing module is used to perform echo processing on the radar echo data to obtain processed radar echo data, and use the preprocessed radar echo data as the first feedback information;

[0014] The target detection module is used to perform target detection based on the preprocessed radar echo data using an end-to-end convolutional neural network model for target detection to obtain target traces, and use the obtained target traces as second feedback information;

[0015] The target tracking module is used to track the target point traces based on the acquired target point traces using an end-to-end convolutional neural network model for target tracking, obtain multi-target track information, and use the multi-target track information as the third feedback information;

[0016] The related fusion module is used to perform target fusion based on multi-channel target track information using a fused end-to-end convolutional neural network model to obtain fused target track information, and use the fused target track information as fourth feedback information;

[0017] The display control module is used to perform display control based on the fused target track information using the end-to-end convolutional neural network model of the display control; extract part of the target display control information as feedback information, and feed back the current feedback information to the first feedback information, the second feedback information, the third feedback information and the fourth feedback information respectively;

[0018] The fourth feedback information is used as a partial input of the fused end-to-end convolutional neural network model;

[0019] Introducing the fourth feedback information into the third feedback information as a partial input of an end-to-end convolutional neural network model for target tracking;

[0020] Introducing the third feedback information into the second feedback information as a partial input of an end-to-end convolutional neural network model for target detection;

[0021] The second feedback information is introduced into the first feedback information and used as a part of the input of a preprocessing module.

[0022] Preferably, each feedback information is used as a partial input of each end-to-end convolutional neural network model, and each end-to-end convolutional neural network model is optimized by introducing an attention mechanism.

[0023] Preferably, a radar echo database is constructed, and an end-to-end convolutional neural network model for target detection is trained using the radar echo database;

[0024] The construction of the radar echo database comprises: obtaining radar echo raw data, detecting and processing the radar echo raw data, and recording the detection result and the radar original echo map; based on the detection result, the radar original echo map is marked using the photoelectric image information and / or AIS target information as a target reference to obtain the radar echo database.

[0025] Preferably, the end-to-end convolutional neural network model for target detection adopts a single neural network structure, and the end-to-end convolutional neural network model for target detection is compressed.

[0026] Preferably, the end-to-end convolutional neural network model for target detection using a radar echo database is trained, comprising:

[0027] Construct training and test sets based on radar echo database;

[0028] The end-to-end convolutional neural network model for target detection is trained using the training set to obtain the trained end-to-end convolutional neural network model for target detection, and the performance of the end-to-end convolutional neural network model for target detection is tested using the test set. When the preset requirements are met, the trained end-to-end convolutional neural network model for target detection is obtained; when the preset requirements are not met, the end-to-end convolutional neural network model for target detection is optimized and triggered repeatedly until the preset requirements are met;

[0029] The currently trained end-to-end convolutional neural network model for target detection is regarded as a teacher model, and a student model is constructed. In transfer learning, the spatial neuron activation distribution between the teacher model and the student model is matched by minimizing the MMD distance between the two, thereby realizing knowledge distillation from the classroom model to the student model, and finally obtaining an end-to-end convolutional neural network model for target detection that meets the preset requirements.

[0030] Preferably, the target tracking module includes:

[0031] The end-to-end convolutional neural network model for target tracking is combined with AIS target data to obtain multi-target track information.

[0032] Preferably, the fused end-to-end convolutional neural network model includes:

[0033] Acquire photoelectric and infrared images, perform target detection and target tracking based on the photoelectric and infrared images, obtain target track information, and perform multi-target track correlation between the current target track information and the target track information obtained by the end-to-end convolutional neural network model based on target tracking to obtain multi-channel target track information.

[0034] A method for processing sea surface target information based on integrated deep learning provided by the present invention includes:

[0035] Step S1: the preprocessing module performs echo processing on the radar echo data to obtain processed radar echo data, and uses the preprocessed radar echo data as first feedback information;

[0036] Step S2: The target detection module performs target detection based on the preprocessed radar echo data using an end-to-end convolutional neural network model for target detection to obtain target point traces, and uses the obtained target point traces as second feedback information;

[0037] Step S3: The target tracking module tracks the target points based on the acquired target points using an end-to-end convolutional neural network model for target tracking, obtains multi-target track information, and uses the multi-target track information as the third feedback information;

[0038] Step S4: the relevant fusion module uses the fused end-to-end convolutional neural network model to perform target fusion based on the multi-channel target track information to obtain fused target track information, and uses the fused target track information as the fourth feedback information;

[0039] Step S5: The display control module performs display control based on the fused target track information using the end-to-end convolutional neural network model of the display control; extracts part of the target display control information as feedback information, and feeds back the current feedback information to the first feedback information, the second feedback information, the third feedback information, and the fourth feedback information respectively;

[0040] The fourth feedback information is used as a partial input of the fused end-to-end convolutional neural network model;

[0041] Introducing the fourth feedback information into the third feedback information as a partial input of an end-to-end convolutional neural network model for target tracking;

[0042] Introducing the third feedback information into the second feedback information as a partial input of an end-to-end convolutional neural network model for target detection;

[0043] The second feedback information is introduced into the first feedback information and used as a part of the input of a preprocessing module.

[0044] Preferably, each feedback information is used as a partial input of each end-to-end convolutional neural network model, and each end-to-end convolutional neural network model is optimized by introducing an attention mechanism.

[0045] Preferably, a radar echo database is constructed, and an end-to-end convolutional neural network model for target detection is trained using the radar echo database;

[0046] The construction of the radar echo database comprises: acquiring radar echo raw data, performing detection processing on the radar echo raw data, and recording the detection result and the radar original echo map; based on the detection result, the radar original echo map is annotated using the photoelectric image information and / or the AIS target information as a target reference to obtain the radar echo database;

[0047] The end-to-end convolutional neural network model for target detection adopts a single neural network structure, and compresses the end-to-end convolutional neural network model for target detection;

[0048] The target tracking module comprises:

[0049] Combine the end-to-end convolutional neural network model for target tracking with AIS target data to obtain multi-target track information;

[0050] The fused end-to-end convolutional neural network model includes:

[0051] Acquire photoelectric and infrared images, perform target detection and target tracking based on the photoelectric and infrared images, obtain target track information, and perform multi-target track correlation between the current target track information and the target track information obtained by the end-to-end convolutional neural network model based on target tracking to obtain multi-channel target track information.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention solves the technical problems of information processing such as sea surface radar target tracking and fusion through the design and implementation of an integrated machine learning architecture, comprehensively realizes the deep and complex functional relationship and technical characteristics of each processing module, realizes the effective separation of radar targets and the environment, strengthens the overall information technology effect, improves the overall intelligent level of sea surface radar information processing, and enhances the robustness and real-time performance of the system.

[0054] 2. The present invention can effectively solve various system problems caused by the separate learning of each information processing module, optimize the system globally, overcome certain shortcomings of the system, enhance the effectiveness and real-time performance of the system, and comprehensively improve the level of intelligent learning with a learning architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0056] Figure 1 It is a schematic diagram of multi-station information processing modules and information connection relationships.

[0057] Figure 2 It is an integrated learning module and architecture for the platform.

[0058] Figure 3 Layered architecture for integrated machines.

[0059] Figure 4 To build a flow chart of the automatic annotation system.

[0060] Figure 5 Flowchart for designing and training deep neural network models.

[0061] Figure 6 Schematic diagram of the acceleration process for model optimization.

[0062] Figure 7 Various noise interferences in radar monitoring images.

[0063] Figure 8 This is a measured map of the ship tracking trajectory. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0065] Example 1

[0066] According to a sea surface target information processing system and method based on integrated deep learning provided by the present invention, Figures 1 to 8 As shown, it includes: cascade processing of modules such as radar echo preprocessing, target detection, target tracking, multi-target correlation, track fusion, display and command. Each module uses convolutional neural network for integrated machine learning to achieve integrated optimization of each module at the system level.

[0067] The sea surface target information processing system based on integrated deep learning includes:

[0068] Preprocessing module, target detection module, target tracking module, correlation fusion module and display control module;

[0069] The preprocessing module is used to perform echo processing on the radar echo data to obtain processed radar echo data, and use the preprocessed radar echo data as the first feedback information;

[0070] The target detection module is used to perform target detection based on the preprocessed radar echo data using an end-to-end convolutional neural network model for target detection to obtain target traces, and use the obtained target traces as second feedback information;

[0071] The target tracking module is used to track the target point traces based on the acquired target point traces using an end-to-end convolutional neural network model for target tracking, obtain multi-target track information, and use the multi-target track information as the third feedback information;

[0072] The related fusion module is used to perform target fusion based on multi-channel target track information using a fused end-to-end convolutional neural network model to obtain fused target track information, and use the fused target track information as fourth feedback information;

[0073] The display control module is used to perform display control based on the fused target track information using the end-to-end convolutional neural network model of the display control; extract part of the target display control information as feedback information, and feed back the current feedback information to the first feedback information, the second feedback information, the third feedback information and the fourth feedback information respectively;

[0074] The fourth feedback information is used as a partial input of the fused end-to-end convolutional neural network model;

[0075] Introducing the fourth feedback information into the third feedback information as a partial input of an end-to-end convolutional neural network model for target tracking;

[0076] Introducing the third feedback information into the second feedback information as a partial input of an end-to-end convolutional neural network model for target detection;

[0077] The second feedback information is introduced into the first feedback information and used as a part of the input of a preprocessing module.

[0078] Specifically, each feedback information is used as a partial input of each end-to-end convolutional neural network model, and each end-to-end convolutional neural network model is optimized by introducing an attention mechanism.

[0079] Specifically, a radar echo database is constructed and used to train an end-to-end convolutional neural network model for target detection;

[0080] The construction of the radar echo database comprises: obtaining radar echo raw data, detecting and processing the radar echo raw data, and recording the detection result and the radar original echo map; based on the detection result, the radar original echo map is marked using the photoelectric image information and / or AIS target information as a target reference to obtain the radar echo database.

[0081] Specifically, the end-to-end convolutional neural network model for target detection adopts a single neural network structure, and compresses the end-to-end convolutional neural network model for target detection.

[0082] Specifically, the end-to-end convolutional neural network model for target detection using a radar echo database is trained, including:

[0083] Construct training and test sets based on radar echo database;

[0084] The end-to-end convolutional neural network model for target detection is trained using the training set to obtain the trained end-to-end convolutional neural network model for target detection, and the performance of the end-to-end convolutional neural network model for target detection is tested using the test set. When the preset requirements are met, the trained end-to-end convolutional neural network model for target detection is obtained; when the preset requirements are not met, the end-to-end convolutional neural network model for target detection is optimized and triggered repeatedly until the preset requirements are met;

[0085] The currently trained end-to-end convolutional neural network model for target detection is regarded as a teacher model, and a student model is constructed. In transfer learning, the spatial neuron activation distribution between the teacher model and the student model is matched by minimizing the MMD distance between the two, thereby realizing knowledge distillation from the classroom model to the student model, and finally obtaining an end-to-end convolutional neural network model for target detection that meets the preset requirements.

[0086] Specifically, the target tracking module includes:

[0087] The end-to-end convolutional neural network model for target tracking is combined with AIS target data to obtain multi-target track information.

[0088] Specifically, the fused end-to-end convolutional neural network model includes:

[0089] Acquire photoelectric and infrared images, perform target detection and target tracking based on the photoelectric and infrared images, obtain target track information, and perform multi-target track correlation between the current target track information and the target track information obtained by the end-to-end convolutional neural network model based on target tracking to obtain multi-channel target track information.

[0090] The present invention also provides a sea surface target information processing system based on integrated deep learning. The sea surface target information processing system based on integrated deep learning can be realized by executing the process steps of the sea surface target information processing method based on integrated deep learning, that is, those skilled in the art can understand the sea surface target information processing method based on integrated deep learning as a preferred implementation mode of the sea surface target information processing system based on integrated deep learning.

[0091] Example 2

[0092] Embodiment 2 is a preferred embodiment of Embodiment 1

[0093] According to the present invention, a sea surface target information processing system based on integrated deep learning is provided, combined with the characteristics of radar sea surface echoes, referring to the application scenarios, and needs to be completed in stages; including: the system has (later-stage) information hierarchical feedback, hierarchical information addition, reference to the main learning direction (AIS target), and controllability of system global optimization. The three stages of research and development include: theoretical realization simulation of overall deep learning, combination of background and system application improvement, and application improvement and perfection. System learning is open, and on-site adaptive learning can be performed based on historical learning rules to gradually improve the overall performance of the system.

[0094] Among them, the end-to-end model neural network has fewer hidden layers and is relatively simple, but less flexible. It mainly considers information feedback, prior information, and reference information at all levels, because these information cannot directly access the hidden layer of a large end-to-end network. The entire radar data processing system is divided into four end-to-end network models, an end-to-end convolutional neural network model for echo classification and target detection, an end-to-end convolutional neural network model for target tracking; an end-to-end convolutional neural network model for target fusion; an end-to-end convolutional neural network model for display and control; an external comprehensive optimization display and control module; each end-to-end network structure is basically the same, the knowledge convolution and learning objectives are different, and the reference information model is also different.

[0095] Since the end-to-end neural network model used for echo classification and target detection is the basic model, the learning degree of this part directly affects the subsequent network models at all levels. This part can be learned in advance based on the previous radar data to establish a basic architecture. When the actual system is working, only a small number of parts need to be iterated, which can greatly reduce the huge amount of calculation and is very beneficial to the real-time learning of radar target.

[0096] Among them, in the echo preprocessing part, an automatic annotation system for radar echo images is constructed;

[0097] like Figure 4As shown in the figure, firstly, a large amount of radar echo raw data covering various environmental conditions is collected, and the existing radar detection algorithm is used to process and detect it, and the detection results and radar raw echo map, such as the size and position of the target, are recorded. At this time, the radar detection algorithm should be adjusted to the optimal state under the condition based on experience. Then the radar echo map is divided into a large number of sub-maps with the same specifications (for example, 32*32 specifications). At this time, some sub-maps contain one or more radar targets, and some sub-maps do not contain any radar targets. In the sub-map containing the radar target, according to the position information and size of the target recorded before, the minimum rectangular frame containing this target is calculated, and the horizontal and vertical coordinates of the upper left vertex of the rectangular frame and the length and width of the rectangular frame are recorded. The sub-map that does not contain the radar target is regarded as empty target information. All sub-maps and their corresponding target information constitute the radar echo database. Next, the labeled database is cleaned. If conditions permit, other information sources such as AIS can be used to assist in obtaining target information, improve data quality, and expand data dimensions.

[0098] So far, the automatic annotation of radar echo images under various environmental conditions has been completed, and a general radar echo data set (more than 100,000) in the radar field has been established. In addition, based on this data set, a standardized and automated radar detection effect evaluation system can also be established.

[0099] Design and train deep neural network models suitable for radar target detection, such as Figure 5-6 As shown;

[0100] Due to its complex network structure and numerous computing nodes, deep neural networks have the disadvantage of slow operation speed. However, radar detection tasks have high real-time requirements, so a "one-stage" type of detection algorithm is used. Compared with the "two-stage" detection algorithm, this type of algorithm abandons the stage of generating candidate areas, directly identifies objects and calculates position coordinate values, so it has a faster detection speed. The network structure is roughly composed of a basic feature extraction network, a multi-scale feature fusion layer, and an output layer.

[0101] The feature fusion layer considers using multiple feature maps of different scales for target detection. The specific scale should be determined according to the actual range represented by a single pixel in the database. Since radar targets are relatively small, multiple scales such as 3*3, 7*7, and 13*13 can be selected to detect small, medium, and large radar surface targets. The feature maps of each scale are then fused through a series of convolutional layers and upsampling. The output layer considers using a full convolution structure to summarize the predicted content of each rectangular area to obtain the target category and location information. The data set can be divided into a training set and a test set at a ratio of 5:1. The training set is used to train the model, and the test set is used to evaluate the radar target detection effect, thereby obtaining a deep neural network model that performs well on both the training set and the test set.

[0102] Optimize and accelerate the trained deep neural network model

[0103] Although a "one-stage" type detection algorithm is used, in order to obtain the best possible radar target detection capability under environmental variations, the complexity of the designed deep neural network should be relatively high so as to have the ability to learn more complex functions. This means that there may be many parts in the model that are rarely activated. These parts have little effect on the output results, but they consume more computing resources. Therefore, it is necessary to further improve the running speed of the model through model compression. Referring to the subsequent feedback target information, with the help of target features and partially successfully applied target models, the well-trained deep neural network model is regarded as a teacher network; and a relatively low-complexity shallow network is constructed (this network still meets the aforementioned model design ideas to be suitable for the field of radar detection) as a student network.

[0104] In transfer learning, the distribution of spatial neuron activation between the teacher model and the student model is matched by minimizing the MMD (Maximum mean discrepancy) distance between the two, thereby realizing knowledge distillation from the teacher network (high complexity) to the student network (low complexity), and finally obtaining a radar target detection network model with a processing speed that meets real-time requirements.

[0105] The radar detection effect benchmark platform developed based on this database can facilitate the evaluation of detection effects. Usually, radar detection effect evaluation relies on manual judgment of indicators such as false alarm rate and missed alarm rate, which has the disadvantages of being time-consuming and labor-intensive, inconsistent scales, and insufficient sample size. The radar detection effect benchmark platform directly compares the output results of the radar detection algorithm to be tested with the data annotations, and can quickly and accurately calculate various indicators under different environmental conditions without the problem of inconsistent scales or insufficient sample size.

[0106] During the information processing, the target track information may be fed back to the detection part during tracking, correlation, and fusion. The multi-sensor (including radar, optoelectronics, infrared, AIS) target information processing module may also feed back to the previous module. In this way, there is a deep coupling relationship between the modules. No single module of machine learning can make the system optimal. The preprocessing, detection, tracking, correlation, fusion, display and control modules should be considered as a whole. Deep learning is to establish a hierarchical learning framework for a problem (temporarily called fine) hierarchical learning. The modules are considered to be coarse layers, and the coarse and fine layers are combined for system deep learning. End-to-end network learning is fine layer learning, and deep functional relationships are learned for representation. Information layer transmission and on-demand feedback belong to the coarse layer. The global optimization of the integrated system realizes the deep coupling of the coarse and fine layers of the system.

[0107] In order to ensure the stable tracking of small targets, it is necessary to feed back the tracking information to the detection learning module, introduce the attention mechanism, and make full use of the local optimality of the deep learning network (which was originally a defect, but now is an advantage for specific targets); the multi-level deep learning network uses feedback and reference information, and the parameters of each layer of modules are constantly learning and optimizing the process, and the structure and parameters are constantly adjusted to gradually achieve the global optimality of the processing system. Make full use of information reverse feedback to solve problems such as clutter, noise, interference, reflection, etc., and solve them together with intelligent machine learning.

[0108] Example 3

[0109] Embodiment 3 is a preferred embodiment of Embodiment 1

[0110] Using image thinking to process radar information

[0111] Traditionally, radar echo signals have been treated as one-dimensional signals, which reduces the correlation between signals at different times, different directions, and different frames. Directly treating radar echoes as images can bring out the correlation between signals and improve the system processing effect.

[0112] In radar images, no distinction is made between backgrounds such as noise, clutter, reflection, and interference. All backgrounds, including targets, are considered echoes. The target is gradually separated from the background based on target characteristics (motion, contour, track, reference, prior, etc.), and the target track is established and stable tracking is achieved. Through the target's characteristics, references, correlations, and feedback information, further correlation fusion is performed to confirm and extend the target track and improve the reliability and accuracy of the target track, thereby achieving global optimization of system performance.

[0113] Introducing deep learning into the field of ship radar information

[0114] Since there are obvious differences between optical images and radar images, such as the scanning mode, resolution, granularity, etc. of the radar, the method can be used for reference. According to the scanning mode adopted by the radar, a circular convolutional neural network of azimuth lines is used in the circular azimuth direction, and a convolutional neural network is used in the distance direction. Calculated by frame, the radar image detected and processed does not exceed 200 frames of radar scanning (to adapt to the rapid changes of sea clutter and environment), and its impact decays frame by frame; the target track information for tracking and subsequent processing is not limited to 200 frames. The feedback mechanism of subsequent information is introduced as part of the input of the convolutional network, and (part of) the reference information can be used as the hidden layer parameters of the convolutional neural network for supervised training.

[0115] Use classifiers to classify ships, and directly use classifiers to separate interference signals such as sea clutter from targets;

[0116] To distinguish the types of ships, we use a variety of typical target characteristics and motion characteristics as the feature extraction images of the convolution layer of the neural network, and use a large sample size training method to make high-quality distinctions, thereby providing high-quality services to meet user needs. The traditional method is to filter sea clutter by capturing features such as texture, but the feature expression ability is not strong, and many detailed information cannot be captured. Deep learning relies on its powerful analytical ability to gradually discover details in sea clutter signals that many human eyes cannot notice.

[0117] Reference tracking method, combined with deep learning for target track tracking;

[0118] Traditional tracking problems have matching and searching ideas. The "filtering + prediction + correlation" method, the new idea of ​​deep learning and "prediction", can greatly improve the learning speed and efficiency. The target information of the previous frames (multi-step recursion) is sent to the deep convolutional neural network (of the tracking correlation module) to predict the position of the target track of this frame (multi-step). In the processing of this frame, the correlation and filtering method is adopted, and the M / N detector is used as the correlation reference to realize the frame-by-frame tracking of the target track. The target information that is not correlated is retained for continued observation. The target track after correlation and filtering is fed back to the previous convolutional neural network to reduce the learning time of the previous stage, and is sent to the next convolutional neural network as the learning input layer of the next stage. This is a great innovation in the traditional target tracking idea.

[0119] Refer to relevant fusion methods and combine deep learning to perform target track fusion;

[0120] Traditional fusion problems have the idea of ​​correlation plus fusion, combined with other sensors (AIS, photoelectric, infrared, laser ranging, other radars, etc.) target track information (these information is incomplete or the measured coordinates are different, such as radar target position is distance and azimuth; photoelectric and infrared are: azimuth and elevation; laser ranging is distance; the size, profile, echo amplitude, measurement time, sampling period, target track number, etc. of each sensor target may be different), using deep convolutional neural network learning and the new idea of ​​"correlation", it can explore the subtle features and deep functional relationships between the different coordinate characteristics of each sensor target track information. Target tracks with several (different) coordinate characteristics can be considered as related targets. Through the correlation of multiple tracks, the fusion is achieved through the related target tracks to further confirm the target track and improve the (fusion) track performance indicators. Target tracks that have not passed the correlation can continue to learn. When there is evidence that the track cannot be correlated, the subsequent network learning will not be carried out. The fused track can be fed back to the previous network learning module to reduce the amount of calculation at the previous level; the fusion layer network information is sent to the next level convolutional neural network as input. This is a great innovation in traditional related integration ideas.

[0121] Refer to the system display and control method and combine it with deep learning to optimize the system;

[0122] The display and control of the traditional system mainly include display and control. The display may include the target echo (shape and amplitude) sent by the previous levels and the track information including history and status (track ID number, fusion ID number, target position, speed and heading, captain, target discovery station and fusion station, tracking time, etc.). The display of the target has special areas such as strong clutter area, multi-reflection area, radar blind area, echo shielding area, key focus area, multi-target channel, sea anchorage, alarm sea area, etc. The control part includes three parts: evidence collection; the user (clicks with the mouse) focuses on the target, and uses the radar target position to guide the optoelectronic and infrared to obtain long-distance image evidence of the target; or uses the optoelectronic target position to guide the radar to measure and track the target at a long distance; it can guide the drone to approach the target for evidence collection. Alarm; whether the AIS information of the radar target (not open, intermittent, AIS number changed) is normal, and it is necessary to call available (other optoelectronic, other radar, etc.) resources for evidence collection or authentication. Whether the target deviates from the course, departs, docks, and whether the behavior trajectory is normal, etc., call the historical track and record to comprehensively judge whether it is abnormal; when a target enters the alarm sea area, and the previous evidence collection exceeds the alarm threshold (further learning later), an alarm is issued so that the user action team can take corresponding measures, and can guide drones and unmanned ships to shout or drive away when approaching the target. Optimization; The system interface adjusts the system parameters of the entire system (including radar, optoelectronics, infrared and other equipment) according to user needs. The display and control information also includes the type and characteristics of the key targets that the user is concerned about, the location and area, and the system performance required by the user.

[0123] These display and control information are used as input (display and control level) convolutional neural networks, which mainly learn the system optimization structure and parameters, control the input scale and information feedback structure of each learning module, optimize the learning depth and learning model parameters of each module, optimize the control process, etc. The ultimate goal is to optimize the system and achieve intelligent system performance within a limited time; mainly learn the characteristics that users care about, various information of the target, alarm areas and thresholds, etc. This is related to the user, and different users need to learn again. The ultimate goal is to meet user needs. This is a great innovation in traditional display and control ideas.

[0124] Replacing (one-level deep) convolutional neural networks with integrated processing learned by multi-level convolutional neural networks

[0125] Modern first-level deep learning convolutional neural networks can have more than one hundred hidden layers. Although they can learn extremely complex functional relationships, the computing time and system processing flexibility will be greatly reduced. This is not suitable for radar information processing systems because the sea surface environment changes very quickly, that is, the background echo changes very quickly (calculated in 15 minutes), and the radar scanning speed (2.5s per frame) is also fast. The system needs to track a large number of targets in real time (up to more than 2,000 batches of targets), which greatly compresses the learning time and computing power. In addition, there is feedback between the previous and next information, and the intermediate input may add other sensor target track information. The goal pursued by the entire system is to optimize the recursive processing by (scanning) frame cycle, rather than by batch (multiple frames in one batch) processing optimization (with non-internal adjustment and intermediate information addition, etc.). Taking into account the flexibility of the system, information feedback, global optimization, time requirements and other issues, it is not appropriate to use a single-level (deep) convolutional network for deep learning. Instead, a multi-level (deeper, about 7 levels) convolutional network is used for all-in-one (integrated) deep learning. Comprehensive processing can combine the prior information and reference information of the radar target, adapt to the feedback structure, and realize a globally optimized adaptive processing system, so that the system parameters set in the application are minimized, which greatly facilitates the system application. This is a great innovation in the idea of ​​machine learning.

[0126] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0127] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A sea surface target information processing system based on integrated deep learning, characterized in that: include: Preprocessing module, target detection module, target tracking module, correlation fusion module and display control module; The preprocessing module is used to perform echo processing on the radar echo data to obtain processed radar echo data, and use the preprocessed radar echo data as the first feedback information; The target detection module is used to perform target detection based on the preprocessed radar echo data using an end-to-end convolutional neural network model for target detection to obtain target traces, and use the obtained target traces as second feedback information; The target tracking module is used to track the target point traces based on the acquired target point traces using an end-to-end convolutional neural network model for target tracking, obtain multi-target track information, and use the multi-target track information as the third feedback information; The related fusion module is used to perform target fusion based on multi-channel target track information using a fused end-to-end convolutional neural network model to obtain fused target track information, and use the fused target track information as fourth feedback information; The display control module is used to perform display control based on the fused target track information using the end-to-end convolutional neural network model of the display control; extract part of the target display control information as feedback information, and feed back the current feedback information to the first feedback information, the second feedback information, the third feedback information and the fourth feedback information respectively; The fourth feedback information is used as a partial input of the fused end-to-end convolutional neural network model; Introducing the fourth feedback information into the third feedback information as a partial input of an end-to-end convolutional neural network model for target tracking; Introducing the third feedback information into the second feedback information as a partial input of an end-to-end convolutional neural network model for target detection; The second feedback information is introduced into the first feedback information and used as a part of the input of a preprocessing module.

2. The sea surface target information processing system based on integrated deep learning according to claim 1 is characterized in that: Each feedback information is used as a partial input of each end-to-end convolutional neural network model, and each end-to-end convolutional neural network model is optimized by introducing an attention mechanism.

3. The sea surface target information processing system based on integrated deep learning according to claim 1 is characterized in that: Build a radar echo database and use it to train an end-to-end convolutional neural network model for target detection; The construction of the radar echo database comprises: obtaining radar echo raw data, detecting and processing the radar echo raw data, and recording the detection result and the radar original echo map; based on the detection result, the radar original echo map is marked using the photoelectric image information and / or AIS target information as a target reference to obtain the radar echo database.

4. The sea surface target information processing system based on integrated deep learning according to claim 1 is characterized in that: The end-to-end convolutional neural network model for target detection adopts a single neural network structure and performs compression processing on the end-to-end convolutional neural network model for target detection.

5. The sea surface target information processing system based on integrated deep learning according to claim 1 is characterized in that: The end-to-end convolutional neural network model for target detection using the radar echo database is trained, including: Construct training and test sets based on radar echo database; The end-to-end convolutional neural network model for target detection is trained using the training set to obtain the trained end-to-end convolutional neural network model for target detection, and the performance of the end-to-end convolutional neural network model for target detection is tested using the test set. When the preset requirements are met, the trained end-to-end convolutional neural network model for target detection is obtained; when the preset requirements are not met, the end-to-end convolutional neural network model for target detection is optimized and triggered repeatedly until the preset requirements are met; The currently trained end-to-end convolutional neural network model for target detection is regarded as a teacher model, and a student model is constructed. In transfer learning, the spatial neuron activation distribution between the teacher model and the student model is matched by minimizing the MMD distance between the two, thereby realizing knowledge distillation from the classroom model to the student model, and finally obtaining an end-to-end convolutional neural network model for target detection that meets the preset requirements.

6. The sea surface target information processing system based on integrated deep learning according to claim 1 is characterized in that: The target tracking module comprises: The end-to-end convolutional neural network model for target tracking is combined with AIS target data to obtain multi-target track information.

7. The sea surface target information processing system based on integrated deep learning according to claim 1 is characterized in that: The fused end-to-end convolutional neural network model includes: Acquire photoelectric and infrared images, perform target detection and target tracking based on the photoelectric and infrared images, obtain target track information, and perform multi-target track correlation between the current target track information and the target track information obtained by the end-to-end convolutional neural network model based on target tracking to obtain multi-channel target track information.

8. A method for processing sea surface target information based on integrated deep learning, characterized in that: include: Step S1: the preprocessing module performs echo processing on the radar echo data to obtain processed radar echo data, and uses the preprocessed radar echo data as first feedback information; Step S2: The target detection module performs target detection based on the preprocessed radar echo data using an end-to-end convolutional neural network model for target detection to obtain target point traces, and uses the obtained target point traces as second feedback information; Step S3: The target tracking module tracks the target points based on the acquired target points using an end-to-end convolutional neural network model for target tracking, obtains multi-target track information, and uses the multi-target track information as the third feedback information; Step S4: the relevant fusion module uses the fused end-to-end convolutional neural network model to perform target fusion based on the multi-channel target track information to obtain fused target track information, and uses the fused target track information as the fourth feedback information; Step S5: The display control module performs display control based on the fused target track information using the end-to-end convolutional neural network model of the display control; extracts part of the target display control information as feedback information, and feeds back the current feedback information to the first feedback information, the second feedback information, the third feedback information, and the fourth feedback information respectively; The fourth feedback information is used as a partial input of the fused end-to-end convolutional neural network model; Introducing the fourth feedback information into the third feedback information as a partial input of an end-to-end convolutional neural network model for target tracking; Introducing the third feedback information into the second feedback information as a partial input of an end-to-end convolutional neural network model for target detection; The second feedback information is introduced into the first feedback information and used as a part of the input of a preprocessing module.

9. The method for processing sea surface target information based on integrated deep learning according to claim 8, characterized in that: Each feedback information is used as a partial input of each end-to-end convolutional neural network model, and each end-to-end convolutional neural network model is optimized by introducing an attention mechanism.

10. The method for processing sea surface target information based on integrated deep learning according to claim 8, characterized in that: Build a radar echo database and use it to train an end-to-end convolutional neural network model for target detection; The construction of the radar echo database comprises: acquiring radar echo raw data, performing detection processing on the radar echo raw data, and recording the detection result and the radar original echo map; based on the detection result, the radar original echo map is annotated using the photoelectric image information and / or the AIS target information as a target reference to obtain the radar echo database; The end-to-end convolutional neural network model for target detection adopts a single neural network structure, and compresses the end-to-end convolutional neural network model for target detection; The target tracking module comprises: Combine the end-to-end convolutional neural network model for target tracking with AIS target data to obtain multi-target track information; The fused end-to-end convolutional neural network model includes: Acquire photoelectric and infrared images, perform target detection and target tracking based on the photoelectric and infrared images, obtain target track information, and perform multi-target track correlation between the current target track information and the target track information obtained by the end-to-end convolutional neural network model based on target tracking to obtain multi-channel target track information.

Images