Marine target heterogenous data matching method based on spatial topology and multiple features

Abstract

The invention relates to a marine target heterogenous data matching method based on spatial topology and multiple features, and belongs to the technical field of target detection. In order to solve the problem that target correspondence in visible light and infrared images cannot be carried out by using traditional feature matching, the method comprises the following steps: step 1, acquiring a marine photoelectric image data set and carrying out preprocessing; 2, an improved Yolov9 target detection model is built; step 3, training and optimizing the Yolov9 target detection model improved in the step 2; step 4, inputting test data into the improved Yov9 target detection model for matching to obtain four angular coordinate values of the target block diagram; and 5, constructing a topological relation by using the four angular coordinate values of the target block diagram obtained in the step 4, and determining a spatial connection relation between the targets. The method improves the detection precision, and reduces the calculation amount and parameter amount.

Description

Technical Field

[0001] The invention relates to a method for matching heterogeneous data of marine targets, in particular to a method for matching heterogeneous data of marine targets based on spatial topology and multiple features, and belongs to the technical field of target detection. Background Art

[0002] Maritime target recognition plays an important role in promoting maritime trade and maintaining maritime transportation. There are usually dense routes and a large number of ships in offshore areas, which poses a great safety hazard. Therefore, it is necessary to monitor and identify cargo ships, fishing boats, ferries and other ships in the sea area. Traditional methods of maritime target detection include radar, AIS, visible light images, infrared images and other technologies. However, a single sensor is easily affected by harsh environments and it is difficult to give a comprehensive and detailed description of the scene. The comprehensive use of multiple sensors has become the key to solving the problem. Visible light images present rich target details and surrounding environment by capturing light in the visible light range, while infrared images can detect the temperature difference between the target and the surrounding environment in insufficient light to gain insight into the target. Therefore, how to comprehensively use visible light and infrared images to obtain more accurate and richer target scene information is an important topic for automatic labeling of maritime targets at this stage.

[0003] The offshore environment is complex and ships are close to the shore. The traditional image matching process is easily affected by the complex shore background, the mutual occlusion of ships, and the change of attitude angles. It is difficult to separate the ship from the complex background. The target detection method based on deep learning has become a current research hotspot. In order to improve the detection performance of the model, most of them start from the perspective of reducing the number of parameters, improving detection accuracy and speeding up image processing. Improving detection accuracy while ensuring lightweight has become an important consideration for model improvement.

[0004] The focal lengths of the photoelectric image data sets obtained during automatic annotation of marine targets are different. The focal length of the visible light image is larger and the field of view is smaller, while the focal length of the infrared image is smaller and the field of view is larger. This will result in different sizes of targets in the image, and the number of targets cannot form a one-to-one correspondence. At the same time, the photoelectric images have different pixel resolutions. When performing image registration, the image to be registered needs to be scaled to match the resolution of the reference image. However, in the case of large differences in the field of view of the photoelectric images, excessive scaling will cause the target feature points to change, so it is impossible to use traditional feature matching to match the targets in the visible light and infrared images. Summary of the invention

[0005] The purpose of the present invention is to solve the problem in the prior art that traditional feature matching cannot be used to correspond targets in visible light and infrared images, and to provide a method for matching heterogeneous data of marine targets based on spatial topology and multiple features. The method uses an improved Yolov9 target detection model to identify the specific location of the target, establishes a topological relationship based on the decision level for the multi-target data detection target to match the photoelectric image target, and realizes the use of topological structure to establish a correspondence between the same target in infrared and visible light images.

[0006] In order to solve the above problems, the present application is implemented through the following technical solutions: The method for matching heterogeneous data of marine targets based on spatial topology and multiple features is special in that it includes the following steps: Step 1: Obtain the marine optoelectronic image dataset and preprocess it: Step 2: By replacing the backbone network of the Yolov9 model with the Mobilenetv4 module and setting four detection heads in the head network part to fuse the neck network, an improved Yolov9 target detection model is built; wherein the backbone network is responsible for feature extraction, The neck network performs feature fusion on the extracted features. The head network outputs target detection results of different sizes; Step 3: Train and optimize the improved Yolov9 target detection model in step 2; Step 4: Input the test data into the improved Yolov9 target detection model in step 3 for matching, and obtain the four corner coordinate values ​​of the target frame diagram; Step 5: Use the four corner coordinate values ​​of the target frame diagram obtained in step 4 to build a topological relationship and determine the spatial connection relationship between the targets.

[0007] Furthermore, the step 1 specifically includes the following sub-steps: Step 1.1, collect image-text pair datasets of visible light and infrared images, wherein the image-text pair datasets include a marine target image dataset and a corresponding text dataset, and the annotations are in Yolo format, and meet the standards of balanced category distribution, accurate annotation, and scene diversity; Step 1.2, collecting visible light and infrared image data, wherein the visible light and infrared image data include multi-target images covering different time, weather, lighting conditions and sea conditions; Step 1.3: Preprocess the image-text data set of the visible light and infrared images collected in step 1.1 so that both the target image data and the text data meet the requirements of the Yolov9 target detection algorithm. At the same time, divide the data set into a training set, a validation set, and a test set, of which 80% is a training set, 10% is a validation set, and 10% is a test set. Further, the step 2 is specifically as follows: the core module of the MobileNetV4 module is the inverted bottleneck search block UIB, and the inverted bottleneck search block UIB integrates the inverted bottleneck IB, ConvNext, the feedforward network FFN and the additional depth-separable convolution ExtraDW variant to form a unified structure; In the inverted bottleneck search block UIB, the number of input feature channels is first expanded to a higher dimension through the 1×1 convolution of the expansion layer; secondly, a depthwise separable convolution DW Conv is added before the expansion layer or between the expansion layer and the projection layer to enhance spatial feature extraction. The addition of the depthwise separable convolution layer is automatically determined by the neural network architecture search optimization process; then, features are further extracted through traditional 3×3 depthwise separable convolution; finally, the projection layer uses 1×1 convolution to map high-dimensional features back to low dimensions.

[0008] Furthermore, the MobileNetV4 module in step 2 uses residual connection to optimize information flow when the input and output shapes are consistent; Furthermore, in step 2, four detection heads are set in the head network part for fusing the neck network as follows: each of the P2, P3, P4 and P5 feature layers of the neck network is provided with a detection head; Furthermore, the step 3 specifically includes the following steps: Step 3.1, use the training set preprocessed in step 1.3 to train the improved Yolov9 target detection model, and adjust the hyperparameters of the improved Yolov9 target detection model, including learning rate, batch size, weight decay, etc.; Step 3.2: Use the validation set to tune the improved Yolov9 target detection model, and optimize the training effect through grid search and learning rate tuning; Step 3.3, use the pre-trained weight file for transfer learning to optimize the detection effect of the improved Yolov9 target detection model in multi-target scenarios; Further, the step 4 is specifically as follows: the improved Yolov9 target detection model identifies the target in the visible light and infrared images and determines the position of the target by regressing the center point coordinates and category confidence of the target bounding box, generates a bounding box adapted to the target and adjusts it to obtain the four corner coordinate values ​​of the target frame diagram; Furthermore, the step 5 is specific: determining the matching relationship by combining the aspect ratio of the target itself and the length ratio of the same target in the two images after matching, and the specific steps are as follows: Step 5.1, calculate the aspect ratio of the target itself in the visible light image, calculate the aspect ratio of each target in the infrared image, traverse and search for targets with the same aspect ratio as the target in the visible light image, and eliminate targets with different aspect ratios from the target in the visible light image; Step 5.2, take the coordinates of the remaining four corner points of the target in the visible light image in clockwise order, and calculate the slope of the corresponding corner points of any two targets; Find the coordinates of the four corner points of target k in the infrared image in clockwise order, and traverse to find the target with the same slope as the corresponding corner points of the two targets in the visible light pattern; Step 5.3: For the targets with equal slopes of corresponding corner points obtained in step 5.2, calculate the length ratio of the same target after matching to further verify, and remove the targets with inconsistent length ratios as erroneous targets.

[0009] Furthermore, the step 5.2 is specifically as follows: The coordinates of the four corner points of target 1 in the visible light image are (a11, b11), (a12, b12), (a13, b13), and (a14, b14) clockwise from the upper left corner. The coordinates of the four corner points of the nth target are (an1, bn1), (an2, bn2), (an3, bn3), and (an4, bn4) clockwise from the upper left corner. The slopes of the corresponding corner points of the two targets are calculated. The slopes of the four corner points of targets 1 and 2 are: k1=(b21-b11) / (a21-a11), k2=(b22-b12) / (a22-a12), k3=(b23-b13) / (a23-a13), k4=(b24-b14) / (a24-a14). The calculation method of the slopes of the four corner points of any other two targets is the same as the calculation method of the slopes of the four corner points of target 1 and target 2.

[0010] This application improves the YOLOv9 model and designs a method for matching targets in marine optoelectronic images. Specifically: a P2 detection head is added to improve the detection accuracy of small targets, and the shallow feature maps connected in series are detected to alleviate the problem of poor fusion effect of ordinary feature fusion layers; the backbone network is replaced with a Mobilenetv4 module that extracts more refined and lightweight features, while not changing the feature extraction capability of the backbone network, the reasoning speed of the network model is improved while the model parameters and computational complexity are reduced; the topological structure is used for decision-level optoelectronic image target correspondence. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of the present invention; Figure 2 This is the framework diagram of the improved Yolov9 target detection model; Figure 3Improved renderings for the head network part; Figure 4 Schematic diagram of the target in the visible light image; Figure 5 It is a schematic diagram of the target in the infrared image; Figure 6 This is a schematic diagram of the matching results of scenario one; Figure 7 This is a schematic diagram of the matching results of scenario 2. DETAILED DESCRIPTION

[0012] The following is a specific implementation of the present invention with reference to the accompanying drawings to further illustrate the composition of the present invention.

[0013] Embodiment 1. A method for matching marine target heterogeneous data based on spatial topology and multiple features comprises the following steps: Step 1: Obtain and preprocess the marine optoelectronic image dataset; The marine electro-optical image dataset contains visible light and infrared images, covering a variety of marine target types and complex environmental conditions to support tasks such as target detection and recognition. The dataset includes multi-target images under different weather conditions (sunny, foggy), lighting conditions (day, night), and sea conditions (calm, wavy); the target types are diverse, including fishing boats, cargo ships, buoys, ferries, and tugboats. Step 2: Build an improved Yolov9 target detection model by replacing the backbone network of the Yolov9 model with the Mobilenetv4 module and setting four detection heads in the head network part to fuse the neck network, such as Figure 2 As shown; wherein the backbone network is responsible for feature extraction, The neck network performs feature fusion on the extracted features. The head network outputs target detection results of different sizes; Step 3: Train and optimize the improved Yolov9 target detection model in step 2; Step 4: Input the test data into the improved Yolov9 target detection model in step 3 for matching, and obtain the four corner coordinate values ​​of the target frame diagram; Step 5: Use the four corner coordinate values ​​of the target frame diagram obtained in step 4 to build a topological relationship and determine the spatial connection relationship between the targets.

[0014] Furthermore, the step 1 specifically includes the following sub-steps: Step 1.1, collect image-text pair datasets of visible light and infrared images, wherein the image-text pair datasets include a marine target image dataset and a corresponding text dataset, and the annotations are in Yolo format, and meet the standards of balanced category distribution, accurate annotation, and scene diversity; Step 1.2, collecting visible light and infrared image data, wherein the visible light and infrared image data include multi-target images covering different time, weather, lighting conditions and sea conditions; Step 1.3: Preprocess the image and text data sets of the visible light and infrared images collected in step 1.1 so that both the target image data and the text data meet the requirements of the Yolov9 target detection algorithm. At the same time, divide the data set into a training set, a validation set, and a test set, of which 80% is a training set, 10% is a validation set, and 10% is a test set.

[0015] Further, the step 2 is specifically as follows: the core module of the MobileNetV4 module is the inverted bottleneck search block UIB, and the inverted bottleneck search block UIB integrates the inverted bottleneck IB, ConvNext, the feedforward network FFN and the extra depth separable convolution ExtraDW variant to form a unified structure; In the inverted bottleneck search block UIB, the number of input feature channels is first expanded to a higher dimension through the 1×1 convolution of the expansion layer; secondly, a depthwise separable convolution DW Conv is added before the expansion layer or between the expansion layer and the projection layer to enhance spatial feature extraction. The addition of the depthwise separable convolution layer is automatically determined by the neural network architecture search optimization process; then, features are further extracted through traditional 3×3 depthwise separable convolution; finally, the projection layer uses 1×1 convolution to map high-dimensional features back to low dimensions.

[0016] Furthermore, the MobileNetV4 module in step 2 uses residual connection to optimize information flow when the input and output shapes are consistent.

[0017] Furthermore, in step 2, four detection heads are set in the head network part for fusing the neck network as follows: each of the P2, P3, P4 and P5 feature layers of the neck network is provided with a detection head. For small targets, since they are usually blurred and small in size in the image, the original detection head often finds it difficult to accurately capture their position and features. Therefore, this application introduces an additional detection head based on the original model. By performing detection on the lower feature layer P2, it makes full use of its characteristics of having strong position information, and can more accurately locate small targets. The improvement effect is as follows: Figure 3 shown.

[0018] Furthermore, the step 3 specifically includes the following steps: Step 3.1, use the training set preprocessed in step 1.3 to train the improved Yolov9 target detection model, and adjust the hyperparameters of the improved Yolov9 target detection model, including learning rate, batch size, weight decay, etc.; Step 3.2: Use the validation set to tune the improved Yolov9 target detection model, and optimize the training effect through grid search and learning rate tuning; Step 3.3: Use the pre-trained weight file for transfer learning to optimize the detection effect of the improved Yolov9 target detection model in multi-target scenarios.

[0019] Furthermore, step 4 is specifically as follows: the improved Yolov9 target detection model identifies the targets in the visible light and infrared images and determines the position of the target by regressing the center point coordinates and category confidence of the target bounding box, generates a bounding box that fits the target and adjusts it to obtain the four corner coordinate values ​​of the target frame diagram.

[0020] Furthermore, the step 5 is specific: determining the matching relationship by combining the aspect ratio of the target itself and the length ratio of the same target in the two images after matching, and the specific steps are as follows: Step 5.1, calculate the aspect ratio of the target itself in the visible light image, calculate the aspect ratio of each target in the infrared image, traverse and search for targets with the same aspect ratio as the target in the visible light image, and eliminate targets with different aspect ratios from the target in the visible light image; like Figure 4-Figure 5 As shown, the aspect ratio of target 1 in the visible light image is calculated, the aspect ratio of each ship in the infrared image is calculated, and the aspect ratio of target 1 is searched. Similarly, the aspect ratio of targets 2 and 3 is searched. Target 8 has a different aspect ratio from targets 1, 2, and 3, so it can be eliminated. Step 5.2, take the coordinates of the remaining four corner points of the target in the visible light image in clockwise order, and calculate the slope of the corresponding corner points of any two targets; Find the coordinates of the four corner points of target k in the infrared image in clockwise order, and traverse to find the target with the same slope as the corresponding corner points of the two targets in the visible light pattern; Step 5.3: For the targets with equal slopes of corresponding corner points obtained in step 5.2, calculate the length ratio of the same target after matching to further verify, and remove the targets with inconsistent length ratios as wrong targets. The specific matching results are as follows: Figure 6-Figure 7 shown.

[0021] Furthermore, the step 5.2 is specifically as follows: The coordinates of the four corner points of target 1 in the visible light image are (a11, b11), (a12, b12), (a13, b13), and (a14, b14) clockwise from the upper left corner. The coordinates of the four corner points of the nth target are (an1, bn1), (an2, bn2), (an3, bn3), and (an4, bn4) clockwise from the upper left corner. The slopes of the corresponding corner points of the two targets are calculated. The slopes of the four corner points of targets 1 and 2 are: k1=(b21-b11) / (a21-a11), k2=(b22-b12) / (a22-a12), k3=(b23-b13) / (a23-a13), k4=(b24-b14) / (a24-a14). The calculation method of the slopes of the four corner points of any other two targets is the same as the calculation method of the slopes of the four corner points of target 1 and target 2.

[0022] Using the same method to find k in the infrared image, we can find the values ​​that are equal to k1-k4. Figure 5 The slopes of the corner points of target 4 and target 5 are equal; To determine the visible light Figure 4 Targets 1, 2, and 3 in the infrared image are Figure 5 Targets 4, 5, and 6 match each other, while the corner slopes of targets 4, 5, 6 and 7 are obviously different.

[0023] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the defined scope.

Claims

1. A method for matching marine target heterogeneous data based on spatial topology and multiple features, characterized by: The following steps are involved: Step 1: Obtain and preprocess the marine optoelectronic image dataset; Step 2: Build an improved Yolov9 target detection model by replacing the backbone network of the Yolov9 model with the Mobilenetv4 module and setting four detection heads in the head network part to fuse the neck network; Among them, the backbone network is responsible for feature extraction, The neck network performs feature fusion on the extracted features. The head network outputs target detection results of different sizes; Step 3: Train and optimize the improved Yolov9 target detection model in step 2; Step 4: Input the test data into the improved Yolov9 target detection model in step 3 for matching, and obtain the four corner coordinate values ​​of the target frame diagram; Step 5: Use the four corner coordinate values ​​of the target frame diagram obtained in step 4 to build a topological relationship and determine the spatial connection relationship between the targets.

2. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 1 is characterized in that: The step 1 specifically includes the following sub-steps: Step 1.1, collect image-text pair datasets of visible light and infrared images, wherein the image-text pair datasets include a marine target image dataset and a corresponding text dataset, and the annotations are in Yolo format, and meet the standards of balanced category distribution, accurate annotation, and scene diversity; Step 1.2, collecting visible light and infrared image data, wherein the visible light and infrared image data include multi-target images covering different time, weather, lighting conditions and sea conditions; Step 1.3: Preprocess the image and text data sets of the visible light and infrared images collected in step 1.1 so that both the target image data and the text data meet the requirements of the Yolov9 target detection algorithm. At the same time, divide the data set into a training set, a validation set, and a test set, of which 80% is a training set, 10% is a validation set, and 10% is a test set.

3. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 1 or 2, characterized in that: The step 2 is specifically as follows: the core module of the MobileNetV4 module is the inverted bottleneck search block UIB, and the inverted bottleneck search block UIB integrates the inverted bottleneck IB, ConvNext, the feedforward network FFN and the extra depth-separable convolution ExtraDW variant to form a unified structure; In the inverted bottleneck search block UIB, the number of input feature channels is first expanded to a higher dimension through a 1×1 convolution of the expansion layer; Secondly, a depthwise separable convolution DW Conv is added before the expansion layer or between the expansion layer and the projection layer to enhance spatial feature extraction. The addition of the depthwise separable convolution layer is automatically determined by the neural network architecture search optimization process. Then, features are further extracted through traditional 3×3 depthwise separable convolution. Finally, the projection layer uses 1×1 convolution to map high-dimensional features back to low dimensions.

4. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 3 is characterized in that: The MobileNetV4 module in step 2 uses residual connection to optimize information flow when the input and output shapes are consistent.

5. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 3 is characterized in that: In step 2, four detection heads are set in the head network part for fusing the neck network as follows: each of the P2, P3, P4 and P5 feature layers of the neck network is equipped with a detection head.

6. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 3 is characterized in that: The step 3 specifically includes the following steps: Step 3.1, use the training set preprocessed in step 1.3 to train the improved Yolov9 target detection model, and adjust the hyperparameters of the improved Yolov9 target detection model, including learning rate, batch size, and weight decay; Step 3.2: Use the validation set to tune the improved Yolov9 target detection model, and optimize the training effect through grid search and learning rate tuning; Step 3.3: Use the pre-trained weight file for transfer learning to optimize the detection effect of the improved Yolov9 target detection model in multi-target scenarios.

7. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 3 is characterized in that: The specific details of step 4 are as follows: the improved Yolov9 target detection model identifies the targets in the visible light and infrared images and determines the position of the target by regressing the center point coordinates and category confidence of the target bounding box, generates a bounding box that fits the target and adjusts it to obtain the four corner coordinate values ​​of the target frame diagram.

8. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 3 is characterized in that: The specific steps of step 5 are as follows: determining the matching relationship by combining the length-width ratio of the target itself and the length ratio of the same target in the two images after matching. Step 5.1, calculate the aspect ratio of the target itself in the visible light image, calculate the aspect ratio of each target in the infrared image, traverse and search for targets with the same aspect ratio as the target in the visible light image, and eliminate targets with different aspect ratios from the target in the visible light image; Step 5.2, take the coordinates of the remaining four corner points of the target in the visible light image in clockwise order, and calculate the slope of the corresponding corner points of any two targets; Find the coordinates of the four corner points of target k in the infrared image in clockwise order, and traverse to find the target with the same slope as the corresponding corner points of the two targets in the visible light pattern; Step 5.3: For the targets with equal slopes of corresponding corner points obtained in step 5.2, calculate the length ratio of the same target after matching to further verify, and remove the targets with inconsistent length ratios as erroneous targets.

9. The method for matching marine target heterogeneous data based on spatial topology and multiple features according to claim 8 is characterized in that: The step 5.2 is as follows: The coordinates of the four corner points of target 1 in the visible light image are (a11, b11), (a12, b12), (a13, b13), and (a14, b14) clockwise from the upper left corner. The coordinates of the four corner points of the nth target are (an1, bn1), (an2, bn2), (an3, bn3), and (an4, bn4) clockwise from the upper left corner. The slopes of the corresponding corner points of the two targets are calculated. The slopes of the four corner points of targets 1 and 2 are: k1=(b21-b11) / (a21-a11), k2=(b22-b12) / (a22-a12), k3=(b23-b13) / (a23-a13), k4=(b24-b14) / (a24-a14). The calculation method of the slopes of the four corner points of any other two targets is the same as the calculation method of the slopes of the four corner points of target 1 and target 2.

Images