Multi-node multi-source heterogeneous information fusion and situation information generation method and system

Abstract

The present invention provides a multi-node, multi-source, heterogeneous information fusion and situation information generation method and system, comprising: step S1: performing target detection on visible light images and SAR images respectively to generate visible light image slices and SAR image slices; step S2: inputting the visible light image slices and SAR image slices into a multi-branch deep feature fusion network to perform target recognition based on optical modal features, SAR modal features, and fused modal features; step S3: using the D-S evidence theory to perform decision-level fusion of the recognition results with electronic reconnaissance data to further obtain a high-confidence recognition result; step S4: generating an entire situation base map, extracting the target heading and speed based on the situation base map according to the recognition results, generating a track based on the target heading and speed, and mapping the track onto the situation base map to generate situation information. The present invention extracts multi-dimensional target information by fusing multi-source, heterogeneous image data and performing intelligent association analysis.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing methods and systems, and in particular to a multi-node multi-source heterogeneous information fusion and situation information generation method and system. Background Art

[0002] With the rapid development of the global marine economy and port transportation industry, ocean shipping has become a vital pillar of international trade. However, the complexity of maritime activities continues to increase, especially in busy port areas and important sea lanes, where the types and numbers of vessels continue to increase. Satellite payload imaging has gradually become an important method for obtaining vessel information and assisting in maritime traffic management.

[0003] Under the conditions of high-resolution data sources, ship targets occupy a large number of pixels and generally have clear outlines and rich detailed features. Single payload target detection and recognition can detect and identify ships with high confidence based on various obvious ship features.

[0004] However, in complex ocean scenarios, factors such as islands and reefs on the sea surface and sea waves interfere with ship detection. Meteorological factors such as clouds, fog, rain, and snow have a serious impact on payload imaging. It is impossible to achieve high-confidence detection and identification of ships by relying on a single payload.

[0005] The information element that requires a high update rate for ships is information related to target movement. The key means of obtaining this information is to use multiple payloads to continuously observe key targets, fuse multi-source information, and quickly generate relevant situation information. Therefore, it is urgent to design a multi-node, multi-source, heterogeneous information fusion and situation information generation method and system. By fusing and processing multi-dimensional, heterogeneous monitoring information and performing intelligent correlation analysis, it is possible to extract important information from a large amount of marine vessel information, achieve intelligent perception of maritime activity status, and assist ports in making efficient management decisions. Summary of the Invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a multi-node multi-source heterogeneous information fusion and situation information generation method and system.

[0007] A method for multi-node multi-source heterogeneous information fusion and situation information generation provided by the present invention includes:

[0008] Step S1: performing target detection on the visible light image and the SAR image to generate visible light image slices and SAR image slices respectively;

[0009] Step S2: Input the visible light image slice and the SAR image slice into a multi-branch deep feature fusion network for target recognition;

[0010] Step S3: Use DS evidence theory to perform decision-level fusion of the recognition results and electronic reconnaissance data to further obtain high-confidence recognition results;

[0011] Step S4: Generate the entire situation base map, extract the target heading and speed based on the situation base map according to the target recognition result, generate a track based on the target heading and speed, and map the track onto the situation base map to generate situation information.

[0012] Preferably, the step S1 includes:

[0013] Step S1.1: Use the improved YOLOV8 model based on the C2f-PKI module to perform target detection on the visible light image to generate visible light image slices;

[0014] Step S1.2: Use the improved YOLOV8 model based on the CBAM-SPPF module to perform target detection on the SAR image and generate SAR image slices;

[0015] The improved YOLOV8 model based on the C2f-PKI module combines the C2f module for extracting and transforming input features with the PKI module for capturing texture features of various scales to form a C2f-PKI module. The C2f-PKI module performs feature extraction through global average pooling and multiple deep convolution kernels of different sizes to achieve the purpose of detecting and capturing texture features of targets of different sizes.

[0016] The improved YOLOV8 model based on the CBAM-SPPF module is formed by adding a CBAM module that enhances the feature representation capability of the convolutional neural network to the SPPF module that integrates large-scale global information. The CBAM-SPPF module enhances the feature extraction capability of the network from both spatial and channel dimensions, thereby learning more target feature information and position information.

[0017] Preferably, the multi-branch deep feature fusion network includes: two independent feature extractors, a feature fusion module and three independent classifiers;

[0018] Two independent feature extractors are used to extract features from visible light image slices and SAR image slices respectively to obtain optical modal features and SAR modal features;

[0019] The optical modal features and SAR modal features are fused into modal features through a feature fusion module;

[0020] Three independent classifiers are used to identify targets based on optical modal features, SAR modal features, and fusion modal features.

[0021] The feature extractor is an improved ResNet-50; the improved ResNet-50 is obtained by removing the last two layers of ResNet-50, namely the global average pooling layer and the fully connected layer, and adjusting the output to a high-dimensional feature map that meets the preset requirements, thereby retaining the spatial information of the image;

[0022] The feature fusion module combines the optical modal features and the SAR modal features, fuses them using a linear layer, and ultimately reduces the dimensionality to a single modality.

[0023] The classifier uses a softmax activation function to convert the output into probabilities.

[0024] Preferably, the method further comprises: obtaining a total loss through weighted calculation based on the cross entropy loss function of each branch, and evaluating the performance of the multi-branch deep feature fusion network using the total loss;

[0025] The total loss includes:

[0026]

[0027] in, represents the optical modality cross entropy loss; represents the SAR modality cross entropy loss; represents the fused modality cross entropy loss.

[0028] Preferably, step S3 includes:

[0029] Set the decision space as target, non-target, and uncertain, represented by (1, 0, -1); the a types of information sources involved in the fusion are: e1, e2, ...e a ; The detection accuracy of target corresponding to type a signal source: d1, d2, ...d a , its practical meaning is the probability of being confirmed as a target among the detected suspicious targets, and is defined as:

[0030] Target detection accuracy = number of correctly detected targets / total number of detected targets = detection rate / (detection rate + false alarm rate);

[0031] Detection rate = number of correct targets detected / total number of correct targets;

[0032] False alarm rate = number of wrongly detected targets / total number of correct targets;

[0033] Target recognition confidence of a source: c1, c2, ...c a , then the basic probability assignment function of the i-th source is:

[0034]

[0035]

[0036]

[0037] The mixed basic probability assignment function is:

[0038]

[0039]

[0040]

[0041] Where K is the conflict coefficient;

[0042] Through the above process, the mixed basic probability assignment function is calculated using the visible light image target information, SAR image target information and electronic reconnaissance image data, and the decision branch with the largest value is selected as the decision result of whether the target exists.

[0043] Preferably, step S4 includes:

[0044] When the target appears on the image at the first moment, its speed is set to 0 by default; for the i-th moment, i>1, the speed is calculated according to the speed calculation formula;

[0045] First, determine the distance d traveled by the target between time i and time i-1. The specific calculation method is as follows:

[0046]

[0047] Where R is the radius of the Earth, and are the latitudes at the i-1th moment and the i-th moment respectively, and are the longitudes at the i-1th moment and the i-th moment respectively;

[0048] After calculating the distance, calculate the speed of the current target. The specific calculation formula is:

[0049]

[0050] Where v is the speed, d is the distance traveled between time i and time i−1, is the time interval between the i-th moment and the i−1-th moment;

[0051] Calculate the coordinate difference of the connecting line: Let the coordinate of the i-1th moment be ( , ), the coordinates of the i-th moment are ( , The specific calculation formula is as follows:

[0052]

[0053]

[0054] Calculate the heading angle: Use the inverse tangent function to calculate the angle between the connecting line and the positive direction of the y-axis. Since the heading is the angle relative to the north, use Δx and Δy to calculate the target heading θ.

[0055]

[0056] The heading angle calculation result is in degrees and needs to be converted to a preset range. The specific conversion formula is:

[0057] Heading = (θ + 360) mod 360

[0058] The targets at different times are connected according to the direction of the required heading and mapped onto the situation base map according to the coordinates to form situation information.

[0059] A multi-node, multi-source, heterogeneous information fusion and situation information generation system provided by the present invention includes:

[0060] Module M1: performs target detection on visible light images and SAR images to generate visible light image slices and SAR image slices respectively;

[0061] Module M2: Input visible light image slices and SAR image slices into a multi-branch deep feature fusion network for target recognition;

[0062] Module M3: Use DS evidence theory to fuse recognition results with electronic reconnaissance data at the decision level to further obtain high-confidence recognition results;

[0063] Module M4: Generate the entire situation base map, extract the target heading and speed based on the target recognition results based on the situation base map, generate the track based on the target heading and speed, and map the track onto the situation base map to generate situation information.

[0064] Preferably, the module M1 includes:

[0065] Module M1.1: Use the improved YOLOV8 model based on the C2f-PKI module to perform target detection on visible light images and generate visible light image slices;

[0066] Module M1.2: Use the improved YOLOV8 model based on the CBAM-SPPF module to perform target detection on SAR images and generate SAR image slices;

[0067] The improved YOLOV8 model based on the C2f-PKI module combines the C2f module for extracting and transforming input features with the PKI module for capturing texture features of various scales to form a C2f-PKI module. The C2f-PKI module performs feature extraction through global average pooling and multiple deep convolution kernels of different sizes to achieve the purpose of detecting and capturing texture features of targets of different sizes.

[0068] Preferably, the multi-branch deep feature fusion network includes: two independent feature extractors, a feature fusion module and three independent classifiers;

[0069] Two independent feature extractors are used to extract features from visible light image slices and SAR image slices respectively to obtain optical modal features and SAR modal features;

[0070] The optical modal features and SAR modal features are fused into modal features through a feature fusion module;

[0071] Three independent classifiers are used to classify optical modal features, SAR modal features and fusion modal features respectively;

[0072] The feature extractor is an improved ResNet-50; the improved ResNet-50 is obtained by removing the last two layers of ResNet-50, namely the global average pooling layer and the fully connected layer, and adjusting the output to a high-dimensional feature map that meets the preset requirements, thereby retaining the spatial information of the image;

[0073] The feature fusion module combines the optical modal features and the SAR modal features, fuses them using a linear layer, and ultimately reduces the dimensionality to a single modality.

[0074] The classifier uses a softmax activation function to convert the output into a probability;

[0075] The total loss is obtained by weighted calculation based on the cross entropy loss function of each branch, and the total loss is used to evaluate the performance of the multi-branch deep feature fusion network;

[0076] The total loss includes:

[0077]

[0078] in, represents the optical modality cross entropy loss; represents the SAR modality cross entropy loss; represents the fused modality cross entropy loss.

[0079] Preferably, the module M4 includes:

[0080] When the target appears on the image at the first moment, its speed is set to 0 by default; for the i-th moment, i>1, the speed is calculated according to the speed calculation formula;

[0081] First, determine the distance d traveled by the target between time i and time i-1. The specific calculation method is as follows:

[0082]

[0083] Where R is the radius of the Earth, and are the latitudes at the i-1th moment and the i-th moment respectively, and are the longitudes at the i-1th moment and the i-th moment respectively;

[0084] After calculating the distance, calculate the speed of the current target. The specific calculation formula is:

[0085]

[0086] Where v is the speed, d is the distance traveled between time i and time i−1, is the time interval between the i-th moment and the i−1-th moment;

[0087] Calculate the coordinate difference of the connecting line: Let the coordinate of the i-1th moment be ( , ), the coordinates of the i-th moment are ( , The specific calculation formula is as follows:

[0088]

[0089]

[0090] Calculate the heading angle: Use the inverse tangent function to calculate the angle between the connecting line and the positive direction of the y-axis. Since the heading is the angle relative to the north, use Δx and Δy to calculate the target heading θ.

[0091]

[0092] The heading angle calculation result is in degrees and needs to be converted to a preset range. The specific conversion formula is:

[0093] Heading = (θ + 360) mod 360

[0094] The targets at different times are connected according to the direction of the required heading and mapped onto the situation base map according to the coordinates to form situation information.

[0095] Compared with the prior art, the present invention has the following beneficial effects:

[0096] 1. The present invention simultaneously performs end-to-end rapid extraction of multi-modal data to form target data with high information density, thus solving the problem that multi-source heterogeneous payload data contains a large amount of interference information;

[0097] 2. This invention addresses the problem of low target recognition rate caused by meteorological, geographical, and environmental factors in single payload acquisition. It combines deep feature fusion and decision fusion to significantly improve the confidence of the target.

[0098] 3. Since ships in ports and on the sea are strongly correlated with motion information and require high-frequency updates of target positions, the present invention achieves the purpose of quickly extracting key information from a large amount of target data to assist management decisions by extracting the target's speed, heading, and track. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0100] Figure 1 This is a flowchart of the multi-node multi-source heterogeneous information fusion and situation information generation method.

[0101] Figure 2 This is the structure diagram of the YOLOV8 model improved based on the C2f-PKI module.

[0102] Figure 3 This is the detection result of the YOLOV8 model improved based on the C2f-PKI module for visible light images.

[0103] Figure 4 This is the structure diagram of the YOLOV8 model improved based on the CBAM-SPPF module.

[0104] Figure 5 This is the detection result of the YOLOV8 model improved based on the CBAM-SPPF module for SAR images.

[0105] Figure 6 This is a multi-branch deep feature fusion network structure diagram.

[0106] Figure 7 It is a decision fusion flowchart based on DS evidence theory.

[0107] Figure 8 This is a partial screenshot of the decision fusion results based on DS evidence theory.

[0108] Figure 9 Generate a result graph for a situation information.

[0109] Figure 10Schematic diagram of a multi-node, multi-source, heterogeneous information fusion and situation information generation system. DETAILED DESCRIPTION

[0110] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0111] Example 1

[0112] A method for multi-node multi-source heterogeneous information fusion and situation information generation provided by the present invention includes:

[0113] Step 1: Use the improved YOLOV8 model based on the C2f-PKI module to perform target detection on visible light images to generate visible light image slices; use the improved YOLOV8 model based on the CBAM-SPPF module to perform target detection on SAR images to generate SAR image slices;

[0114] Step 2: Input the visible light image slices and SAR image slices into the multi-branch deep feature fusion network to identify the target fine type;

[0115] Step 3: Use DS evidence theory to fuse the recognition results with electronic reconnaissance data at the decision level to further obtain high-confidence recognition results;

[0116] Step 4: Generate the entire situation base map, extract the target heading and speed based on the recognition results, generate the track, and map it onto the situation base map to generate situation information.

[0117] Specifically, the improved YOLOV8 model based on the C2f-PKI module combines the C2f module for extracting conversion input features with the PKI module for capturing texture features of various scales to form a C2f-PKI module.

[0118] Specifically, the improved YOLOV8 model based on the CBAM-SPPF module adds a CBAM module to enhance the feature representation capability of the convolutional neural network after the SPPF module that integrates large-scale global information.

[0119] Specifically, the multi-branch deep feature fusion network independently obtains the features of visible light image slices and SAR image slices through two feature extractors, fuses the multimodal features to obtain a joint representation, and jointly calculates the loss through a multi-task classifier to independently classify the optical modality, SAR modality and the fused features.

[0120] Specifically, the two feature extractors have the same structure, both of which are obtained by removing the last two global average pooling layers and fully connected layers of ResNet-50. The multimodal feature fusion concatenates the features of the optical modality and the SAR modality and fuses them using a linear layer.

[0121] Specifically, the multi-task classifier independently processes the optical modality, SAR modality and fusion modality features, uses the softmax activation function to convert the output of the model into probability, and the cross-entropy loss function evaluates the performance of the multi-branch deep feature fusion network, and finally calculates the total loss through weighted calculation.

[0122] Specifically, the DS evidence theory calculates a mixed basic probability assignment function, selects a decision branch with the largest value, and determines whether the target exists.

[0123] Specifically, the speed is obtained by using the difference between the longitude and latitude of the target at two fixed times to obtain the actual distance the target moves, and then divided by the fixed time. The heading is calculated by using the inverse tangent function to calculate the angle between the line connecting the position of the target within the fixed time and the north direction. The track connects the targets at different times according to the direction of the heading and maps the coordinates to the situation base map composed of optical images.

[0124] The present invention also provides a multi-node multi-source heterogeneous information fusion and situation information generation system. The multi-node multi-source heterogeneous information fusion and situation information generation system can be implemented by executing the process steps of the multi-node multi-source heterogeneous information fusion and situation information generation method. That is, those skilled in the art can understand the multi-node multi-source heterogeneous information fusion and situation information generation method as an optimal implementation method of the multi-node multi-source heterogeneous information fusion and situation information generation system.

[0125] Example 2

[0126] Example 2 is a preferred example of Example 1

[0127] like Figure 1 As shown, a multi-node multi-source heterogeneous information fusion and situation information generation method provided by the present invention includes:

[0128] Step 1: Use the improved YOLOV8 model based on the C2f-PKI module to perform target detection on visible light images to generate visible light image slices; use the improved YOLOV8 model based on the CBAM-SPPF module to perform target detection on SAR images to generate SAR image slices;

[0129] Step 2: Input the visible light image slices and SAR image slices into the multi-branch deep feature fusion network to identify the target fine type;

[0130] Step 3: Use DS evidence theory to fuse the recognition results with electronic reconnaissance data at the decision level to further obtain high-confidence recognition results;

[0131] Step 4: Generate the entire situation base map, extract the target heading and speed based on the recognition results, generate the track, and map it onto the situation base map to generate situation information.

[0132] Furthermore, if Figure 2 As shown, this embodiment provides an improved YOLOV8 model structure diagram based on the C2f-PKI module.

[0133] The improved YOLOV8 model based on the C2f-PKI module combines a C2f module for extracting and transforming input features with a PKI module for capturing texture features of various scales to form a C2f-PKI module.

[0134] The PKI module is an Inception-style module that captures local information through a small kernel convolution and then uses a series of parallel depth-wise convolutions to capture contextual information across multiple scales.

[0135] The C2f-PKI module uses global average pooling and 1×1 strip convolution to enhance the features of the central area and capture long-range contextual information. Unlike methods that rely on large convolution kernels or dilated convolutions, it uses multiple deep convolution kernels of different sizes to extract multi-scale texture features in different receptive fields without dilation.

[0136] Furthermore, if Figure 3 As shown, this embodiment provides a detection result of a visible light image using an improved YOLOV8 model based on the C2f-PKI module. A meter-level visible light remote sensing image dataset is used to verify the improved YOLOV8 model based on the C2f-PKI module. The detection results of 16 images are randomly selected, and the ship targets are marked with bounding boxes of different colors. The type of ship and the confidence level of the target detection result are indicated above.

[0137] Furthermore, if Figure 4 As shown, this embodiment provides an improved YOLOV8 model structure diagram based on the CBAM-SPPF module.

[0138] The improved YOLOV8 model based on the CBAM-SPPF module adds a CBAM module to the SPPF module that integrates large-scale global information to enhance the feature representation capability of the convolutional neural network. It enhances the network's feature extraction capability from both spatial and channel dimensions, suppresses the interference of background clutter, and strengthens the target feature extraction capability in SAR images, thereby enabling the network to learn more target feature information and location information, and improving the detection algorithm performance.

[0139] The CBAM-SPPF module is an attention mechanism module designed to enhance the feature representation capability in convolutional neural networks; it focuses on more informative features by integrating spatial and channel attention mechanisms, thereby improving model performance.

[0140] The channel attention mechanism aims to analyze the correlation between feature channels, increase the weight of valid channels, and reduce the weight of invalid channels, thereby improving the target's feature representation. The CBAM-SPPF module aggregates spatial information in the feature map through global average pooling and global maximum pooling operations. The two pooling results are then processed by a multi-layer perceptron (MLP) with shared weights. Finally, the two signals are added together and passed through a sigmoid function to obtain the attention weight for each channel.

[0141] The spatial attention mechanism focuses on valid information in the feature map by analyzing internal spatial relationships. While channel attention focuses on the correlation between channels to determine the weight of each channel, spatial attention focuses on which spatial locations in the image are most informative. Following channel attention, the CBAM-SPPF module applies max pooling and average pooling (along the channels) to the feature map, stacking the results. This is then passed through a convolutional layer and a sigmoid activation function to generate a spatial attention map that highlights which spatial locations are more important.

[0142] Furthermore, if Figure 5 As shown, this embodiment provides a detection result of a SAR image using an improved YOLOV8 model based on the CBAM-SPPF module. The RSDD-SAR public dataset is used to verify the improved YOLOV8 model based on the CBAM-SPPF module. The detection results of 16 pictures are randomly selected, and the ship target is marked with a blue bounding box. The word "ship" and the confidence level of the target detection result are marked above.

[0143] Furthermore, if Figure 6 As shown in the figure, this embodiment provides a multi-branch deep feature fusion network architecture. Given visible light target slices and SAR target slices, the network first extracts features from the two different modalities of target image slices using two branches. It then fuses the features of the optical and SAR modalities to obtain a joint representation. Finally, a multi-task classifier jointly calculates the loss and independently classifies the optical, SAR, and fused features, thereby enabling precise target type recognition.

[0144] The network architecture can support simultaneous training of fusion classification tasks and single-modal classification tasks. When the network receives input data of only one modality, it outputs the classification results of the single-modal image. When the network simultaneously inputs multi-modal images, it extracts features separately and performs deep feature fusion.

[0145] The network's feature extractor uses a modified ResNet-50, removing the last two layers of ResNet-50: the global average pooling layer and the fully connected layer. This adjustment results in a high-dimensional feature map that preserves the spatial information of the image. Each modality is configured with a separate feature extractor. Although the architecture is the same, it is trained independently for data from different modalities.

[0146] The output high-dimensional feature maps are batch normalized to speed up the training convergence of the model, alleviate the gradient vanishing and gradient exploding problems to a certain extent, and process the value range differences of high-dimensional feature maps on the same scale.

[0147] Feature fusion combines the features of the optical and SAR modes, fuses them using a linear layer, and ultimately reduces the dimensionality to that of a single modality. The linear layer is a lightweight method that does not introduce additional model complexity and performs feature transformation only by learning a matrix.

[0148] The multi-branch deep feature fusion network processes the optical modality, SAR modality and fusion modality features through independent classifiers to perform single modality classification and fusion modality classification respectively. Each classifier uses a softmax activation function to convert the model output into a probability, and finally calculates the total loss.

[0149] Softmax activation function: Converts the output of each classifier into a probability distribution:

[0150]

[0151] Among them, zi is the output value of the i-th category, is the normalization factor for all categories.

[0152] Cross entropy loss function: Cross entropy is used to measure the difference between the predicted probability and the true label.

[0153]

[0154] in, is the one-hot encoding of the true label, is the probability that sample i is predicted to be class j, N represents the sample size, and C represents the total number of target types.

[0155] Multi-classifier loss design: The optical modality loss LH and the SAR modality loss LS are optimized separately, and the fusion modality loss LF is used to optimize the classification after the two modalities are combined. The total loss is the weighted sum of the single modality loss and the fusion modality loss:

[0156]

[0157] This dataset, constructed using SpaceNet6's MSAW remote sensing data, uses optical and SAR dual-modality civilian vessel images. This dataset contains 11 different types of civilian vessels. The dataset was randomly split into a training set and a test set in a 2:8 ratio. The images were uniformly resized to 256×128, and the output dimension of the feature converter was set to 2028. SGD was selected as the optimizer with an initial learning rate of 0.003, 50 training epochs, and a batch size of 1. Finally, the recognition accuracy was measured.

[0158] Table 1 Results of multi-branch deep feature fusion network

[0159]

[0160] It can be seen that in the single-task mode with one of the branches blocked, the recognition accuracy of the optical single modality is 91.84%, and the accuracy of the SAR single modality is 75.84%; in the multi-task mode with dual-modal input, the recognition accuracy of the dual-modal fusion branch is 93.12%, the recognition accuracy of the optical modality branch is 92.32%, and the recognition accuracy of the SAR modality branch is 80.48%. The use of the multi-branch deep feature fusion network improves the recognition accuracy of SAR and visible light images.

[0161] Furthermore, if Figure 7 As shown, this embodiment provides a decision fusion flowchart based on DS evidence theory.

[0162] Set the decision space (target, non-target, uncertain), represented by (1,0,-1); a types of information sources involved in fusion: e1, e2, ...e a ; The detection accuracy of target corresponding to type a signal source: d1, d2, ...d a , its practical meaning is the probability of being confirmed as a target among the detected suspicious targets, and is defined as:

[0163] Target detection accuracy = number of correctly detected targets / total number of detected targets = detection rate / (detection rate + false alarm rate);

[0164] Detection rate = number of correct targets detected / total number of correct targets;

[0165] False alarm rate = number of wrongly detected targets / total number of correct targets;

[0166] Target recognition confidence of a source: c1, c2, ...c a , then the basic probability assignment function of the i-th source is:

[0167]

[0168]

[0169]

[0170] The mixed basic probability assignment function is:

[0171]

[0172]

[0173]

[0174] Where K is the conflict coefficient.

[0175] Through the above process, the mixed basic probability assignment function is calculated using the visible light image target information, SAR image target information and electronic reconnaissance image data, and the decision branch with the largest value is selected as the decision result of whether the target exists.

[0176] Furthermore, if Figure 8 As shown in the figure, this embodiment provides a diagram of the decision fusion results based on the DS evidence theory. The results of deep feature fusion of the optical-SAR dual-modality civilian vessel image dataset were fused on the Ascend 310, and the recognition rate of 95.08% and the classification results of different types of civilian vessels were printed.

[0177] Furthermore, if Figure 9 As shown, this embodiment provides a situation information generation result diagram.

[0178] A 1024×22551 pixel situation base map is formed by stitching visible light images of the port, and the longitude and latitude information of the lower left and upper right corners of the base map are recorded.

[0179] When an object appears in the image at the first moment, its speed is set to 0 by default, as there is no position information from the previous moment to calculate the actual speed. For all other moments (i.e., moment i, where i>1), the speed is inferred according to the speed calculation formula. The calculation method first determines the distance d traveled by the object between moment i and moment i-1. That is, based on the difference in longitude and latitude between the two consecutive moments, the actual distance the object traveled during this period is calculated. The specific calculation method is as follows:

[0180]

[0181] Where R is the radius of the earth, usually 6371 km. and are the latitudes at time i-1 and time i (in radians), and are the longitudes (in radians) at time i-1 and time i, respectively.

[0182] After calculating the distance, we can calculate the speed of the current target. The specific calculation formula is:

[0183]

[0184] Where v is the speed, d is the distance traveled between two consecutive moments (time i and time i−1), is the time interval between the i-th moment and the i-1-th moment. Substituting into the speed formula we can get the speed of the ith element.

[0185] Calculate the coordinate difference of the connecting line: Let the coordinate of the i-1th moment be ( , ), the coordinates of the i-th moment are ( , The specific calculation formula is as follows:

[0186]

[0187]

[0188] Calculate the heading angle: Use the inverse tangent function to calculate the angle between the line and the positive direction of the y-axis (due north). Since the heading is the angle relative to north, we use Δx and Δy to calculate the target heading θ.

[0189]

[0190] The calculated result of the heading angle is in degrees, and the result must be converted to a suitable range (for example, 0° to 360°) to represent different directions. The specific conversion formula is:

[0191] Heading = (θ + 360) mod 360

[0192] The targets at different times are connected according to the direction of the required heading and mapped according to the coordinates to the situation base map composed of optical images to form situation information.

[0193] like Figure 10 As shown, this embodiment also provides a multi-node multi-source heterogeneous information fusion and situation information generation system, which mainly includes:

[0194] The visible light image processing module is used to deploy the improved YOLOV8 model based on the C2f-PKI module to perform target detection on visible light images and generate visible light image slices;

[0195] SAR image processing module, used to deploy the YOLOV8 model improved based on the CBAM-SPPF module to perform target detection on SAR images and generate SAR image slices;

[0196] Fusion processing module, used for feature fusion of visible light image slices, SAR image slices and electronic reconnaissance data;

[0197] The situation information generation module is used to extract the target heading and speed to generate a track and map it onto the situation base map to generate situation information.

[0198] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0199] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for multi-node multi-source heterogeneous information fusion and situation information generation, characterized in that: include: Step S1: Use the improved YOLOV8 model based on the C2f-PKI module to perform target detection on the visible light image to generate visible light image slices; Step S2: Use the improved YOLOV8 model based on the CBAM-SPPF module to perform target detection on the SAR image to generate SAR image slices; Step S3: Input the visible light image slice and the SAR image slice into a multi-branch deep feature fusion network for target recognition; Step S4: Use DS evidence theory to perform decision-level fusion of the recognition results and electronic reconnaissance data to further obtain high-confidence recognition results; Step S5: Generate the entire situation base map, extract the target heading and speed based on the situation base map according to the target recognition result, generate a track based on the target heading and speed, and map the track onto the situation base map to generate situation information; The improved YOLOV8 model based on the C2f-PKI module combines the C2f module for extracting and transforming input features with the PKI module for capturing texture features of various scales to form a C2f-PKI module. The C2f-PKI module performs feature extraction through global average pooling and multiple deep convolution kernels of different sizes to achieve the purpose of detecting and capturing texture features of targets of different sizes. The improved YOLOV8 model based on the CBAM-SPPF module is formed by adding a CBAM module to the SPPF module that integrates large-scale global information to enhance the feature representation capability of the convolutional neural network. The CBAM-SPPF module enhances the feature extraction capability of the network from both spatial and channel dimensions, thereby learning more target feature information and position information. The multi-branch deep feature fusion network includes: two independent feature extractors, a feature fusion module and three independent classifiers; Two independent feature extractors are used to extract features from visible light image slices and SAR image slices respectively to obtain optical modal features and SAR modal features; The optical modal features and SAR modal features are fused into modal features through a feature fusion module; Three independent classifiers are used to identify targets based on optical modal features, SAR modal features, and fusion modal features. The feature extractor is an improved ResNet-50; the improved ResNet-50 is obtained by removing the last two layers of ResNet-50, namely the global average pooling layer and the fully connected layer, and adjusting the output to a high-dimensional feature map that meets the preset requirements, thereby retaining the spatial information of the image; The feature fusion module combines the optical modal features and the SAR modal features, fuses them using a linear layer, and ultimately reduces the dimensionality to a single modality. The classifier uses a softmax activation function to convert the output into a probability; The step S5 comprises: When the target appears on the image at the first moment, its speed is set to 0 by default; for the i-th moment, i>1, the speed is calculated according to the speed calculation formula; First, determine the distance d traveled by the target between time i and time i-1. The specific calculation method is as follows: Where R is the radius of the Earth, and are the latitudes at the i-1th moment and the i-th moment respectively, and are the longitudes at the i-1th moment and the i-th moment respectively; After calculating the distance, calculate the speed of the current target. The specific calculation formula is: Where v is the velocity, d is the time between the i-th moment and the The distance traveled between moments, is the time between the i-th moment and the the time interval between moments; Calculate the coordinate difference of the connecting line: Let the coordinate of the i-1th moment be ( , ), the coordinates of the i-th moment are ( , The specific calculation formula is as follows: Calculate the heading angle: Use the inverse tangent function to calculate the angle between the connecting line and the positive direction of the y-axis. Since the heading is the angle relative to the north, use Δx and Δy to calculate the target heading θ. The heading angle calculation result is in degrees and needs to be converted to a preset range. The specific conversion formula is: Heading = (θ + 360) mod 360 The targets at different times are connected according to the direction of the required heading and mapped onto the situation base map according to the coordinates to form situation information.

2. The multi-node multi-source heterogeneous information fusion and situation information generation method according to claim 1 is characterized in that: The method further includes: obtaining a total loss through weighted calculation based on the cross entropy loss function of each branch, and evaluating the performance of the multi-branch deep feature fusion network using the total loss; The total loss includes: in, represents the optical modality cross entropy loss; represents the SAR modality cross entropy loss; represents the fused modality cross entropy loss.

3. The multi-node multi-source heterogeneous information fusion and situation information generation method according to claim 1 is characterized in that: The step S4 comprises: Set the decision space as target, non-target, and uncertain, represented by (1, 0, -1); the a types of information sources involved in the fusion are: e1, e2, ...e a ; The detection accuracy of target corresponding to type a signal source: d1, d2, ...d a , its practical meaning is the probability of being confirmed as a target among the detected suspicious targets, and is defined as: Target detection accuracy = number of correctly detected targets / total number of detected targets = detection rate / (detection rate + false alarm rate); Detection rate = number of correct targets detected / total number of correct targets; False alarm rate = number of wrongly detected targets / total number of correct targets; Target recognition confidence of a source: c1, c2, ...c a , then the basic probability assignment function of the i-th source is: The mixed basic probability assignment function is: Where K is the conflict coefficient; Through the above process, the mixed basic probability assignment function is calculated using the visible light image target information, SAR image target information and electronic reconnaissance image data, and the decision branch with the largest value is selected as the decision result of whether the target exists.

4. A multi-node multi-source heterogeneous information fusion and situation information generation system, characterized by: include: Module M1: Use the improved YOLOV8 model based on the C2f-PKI module to perform target detection on visible light images and generate visible light image slices; Module M2: Use the improved YOLOV8 model based on the CBAM-SPPF module to perform target detection on SAR images and generate SAR image slices; Module M3: Input visible light image slices and SAR image slices into a multi-branch deep feature fusion network for target recognition; Module M4: Use DS evidence theory to fuse recognition results with electronic reconnaissance data at the decision level to further obtain high-confidence recognition results; Module M5: Generates the entire situation base map, extracts the target heading and speed based on the target recognition results based on the situation base map, generates a track based on the target heading and speed, and maps the track to the situation base map to generate situation information; The improved YOLOV8 model based on the C2f-PKI module combines the C2f module for extracting and transforming input features with the PKI module for capturing texture features of various scales to form a C2f-PKI module. The C2f-PKI module performs feature extraction through global average pooling and multiple deep convolution kernels of different sizes to achieve the purpose of detecting and capturing texture features of targets of different sizes. The improved YOLOV8 model based on the CBAM-SPPF module is formed by adding a CBAM module to the SPPF module that integrates large-scale global information to enhance the feature representation capability of the convolutional neural network. The CBAM-SPPF module enhances the feature extraction capability of the network from both spatial and channel dimensions, thereby learning more target feature information and position information. The multi-branch deep feature fusion network includes: two independent feature extractors, a feature fusion module and three independent classifiers; Two independent feature extractors are used to extract features from visible light image slices and SAR image slices respectively to obtain optical modal features and SAR modal features; The optical modal features and SAR modal features are fused into modal features through a feature fusion module; Three independent classifiers are used to classify optical modal features, SAR modal features and fusion modal features respectively; The feature extractor is an improved ResNet-50; the improved ResNet-50 is obtained by removing the last two layers of ResNet-50, namely the global average pooling layer and the fully connected layer, and adjusting the output to a high-dimensional feature map that meets the preset requirements, thereby retaining the spatial information of the image; The feature fusion module combines the optical modal features and the SAR modal features, fuses them using a linear layer, and ultimately reduces the dimensionality to a single modality. The classifier uses a softmax activation function to convert the output into a probability; The module M5 includes: When the target appears on the image at the first moment, its speed is set to 0 by default; for the i-th moment, i>1, the speed is calculated according to the speed calculation formula; First, determine the distance d traveled by the target between time i and time i-1. The specific calculation method is as follows: Where R is the radius of the Earth, and are the latitudes at the i-1th moment and the i-th moment respectively, and are the longitudes at the i-1th moment and the i-th moment respectively; After calculating the distance, calculate the speed of the current target. The specific calculation formula is: Where v is the velocity, d is the time between the i-th moment and the The distance traveled between moments, is the time between the i-th moment and the the time interval between moments; Calculate the coordinate difference of the connecting line: Let the coordinate of the i-1th moment be ( , ), the coordinates of the i-th moment are ( , The specific calculation formula is as follows: Calculate the heading angle: Use the inverse tangent function to calculate the angle between the connecting line and the positive direction of the y-axis. Since the heading is the angle relative to the north, use Δx and Δy to calculate the target heading θ. The heading angle calculation result is in degrees and needs to be converted to a preset range. The specific conversion formula is: Heading = (θ + 360) mod 360 The targets at different times are connected according to the direction of the required heading and mapped onto the situation base map according to the coordinates to form situation information.

5. The multi-node multi-source heterogeneous information fusion and situation information generation system according to claim 4 is characterized in that: The system further includes: obtaining a total loss through weighted calculation based on the cross entropy loss function of each branch, and evaluating the performance of the multi-branch deep feature fusion network using the total loss; The total loss includes: in, represents the optical modality cross entropy loss; represents the SAR modality cross entropy loss; represents the fused modality cross entropy loss.

6. The multi-node multi-source heterogeneous information fusion and situation information generation system according to claim 4, characterized in that: The module M4 includes: Set the decision space as target, non-target, and uncertain, represented by (1, 0, -1); the a types of information sources involved in the fusion are: e1, e2, ...e a ; The detection accuracy of target corresponding to type a signal source: d1, d2, ...d a , its practical meaning is the probability of being confirmed as a target among the detected suspicious targets, and is defined as: Target detection accuracy = number of correctly detected targets / total number of detected targets = detection rate / (detection rate + false alarm rate); Detection rate = number of correct targets detected / total number of correct targets; False alarm rate = number of wrongly detected targets / total number of correct targets; Target recognition confidence of a source: c1, c2, ...c a , then the basic probability assignment function of the i-th source is: The mixed basic probability assignment function is: Where K is the conflict coefficient; Through the above process, the mixed basic probability assignment function is calculated using the visible light image target information, SAR image target information and electronic reconnaissance image data, and the decision branch with the largest value is selected as the decision result of whether the target exists.

Images