A method for counting ocean vessels based on feature association density map generation

Abstract

This invention discloses a marine vessel counting method based on feature association density map generation, relating to the fields of machine learning algorithms and image processing technology. The invention constructs a marine vessel counting model comprising a feature association density map generation network and a marine vessel counting network based on residual feature fusion. The model is trained using original images and real image density maps. The loss function of the feature association density map generation network, constructed by combining feature association regularization constraints, numerical loss function, and distributed loss function, along with the L2 loss function of the marine vessel counting network based on residual feature fusion, are used as the overall loss function for training. By using the feature association density map generation network to learn the texture and neighborhood relationships of image targets, high-quality matching between the density map and the target shape is achieved. Furthermore, the invention employs feature association regularization constraints to focus on image differences and learn common features of marine vessels, thereby improving the accuracy of the marine vessel counting task.

Description

Technical Field

[0001] This invention relates to the fields of machine learning algorithms and image processing technology, specifically to a method and system for counting marine vessels based on feature correlation density maps. Background Technology

[0002] Maritime vessel counting is the process of identifying and counting the number of vessels in a specific sea area using remote sensing technology to capture images from satellites or aircraft. This technology, by analyzing and interpreting high-resolution images from satellites or aircraft, enables researchers and regulatory agencies to accurately monitor and count the number of vessels in a given sea area. This capability has important applications in various fields, including marine resource management, maritime safety monitoring, and port management.

[0003] In common target counting methods, a density map is typically generated by placing a Gaussian kernel at each target location in the image. This process assumes that the targets are spatially uniformly distributed and independent, while the Gaussian kernel is used to simulate the effect of each target on its surrounding pixels. However, a core assumption of this method is that the shape and distribution of the targets are relatively standardized and uniform, which is particularly insufficient when dealing with targets with complex shapes and varying sizes, such as ships.

[0004] The representation of ships in different images varies greatly due to factors such as their type, size, viewing angle, and distance from the camera. For example, small boats and large cargo ships photographed from the air differ drastically in size and shape, while Gaussian kernel convolution methods often use kernels of fixed size and shape, limiting their adaptability to different situations. Furthermore, clustering of ships can lead to overlapping areas on the density map, making it difficult for Gaussian kernel methods to distinguish individual targets in such cases, thus affecting counting accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a marine vessel counting method based on feature association density map generation. This method fuses cross-domain features by extracting texture and neighborhood relationships between vessel targets in the original image. The method utilizes a feature association density map generation network to generate a true density map of the image based on the labeled point map, effectively learning feature information from the original image and compensating for the deficiency of labeled point maps in failing to achieve high-quality matching with target shapes.

[0006] To achieve the above objectives, in a first aspect, the present invention proposes a method for counting marine vessels based on feature correlation density maps, characterized in that the method includes the following steps:

[0007] S100: Collect remote sensing image datasets and preprocess the remote sensing image datasets, dividing the preprocessed image data into training set, validation set and test set;

[0008] S200, construct a marine vessel counting model, which includes a feature association density map generation network and a marine vessel counting network based on residual feature fusion;

[0009] S300: Input the image data in the training set into the ocean vessel counting model, perform joint training on the model using a weighted combined loss function, and update the ocean vessel counting model through the backpropagation algorithm until the ocean vessel counting model with the minimum loss function is obtained;

[0010] S400: Use the validation set constructed in step S100 to validate the trained ocean vessel counting model, and select the model with the best performance as the trained ocean vessel counting model according to the MAE evaluation index.

[0011] S500 inputs the target remote sensing image into the trained marine vessel counting model, and the marine vessel counting model outputs the total number of vessels detected in a given image area.

[0012] Furthermore, step S100 specifically includes:

[0013] Acquire remote sensing image data, resize the images in the remote sensing image dataset to a uniform size, and obtain the image. and their corresponding point labels Based on the point label information, the remote sensing images are preprocessed to obtain the true density map corresponding to each remote sensing image, which is then saved as a density point map. The training set is defined as follows: , where N represents the total number of images in the training set.

[0014] Furthermore, step S300 specifically includes:

[0015] S310: Input the original images and the true image density maps in the training set into the pre-trained feature association density map generation network, and generate the predicted density map based on the feature association density map;

[0016] S320: The predicted density map generated by the feature association density map generation network and the original image true density map are input into the pre-trained marine vessel counting network based on residual feature fusion to obtain the total number of vessels detected in a given image region.

[0017] Furthermore, the process of generating the pre-trained feature association density map network in step S310 specifically includes:

[0018] S311, the original images in the training set are input into the improved VGG-10 feature extraction network for feature extraction to obtain feature maps. The VGG-10 feature extraction network retains the first ten layers of the original VGG-16 as the feature extraction network, and retains the three max pooling layers in the original VGG-16.

[0019] S312, the true density map of the image is input into a lightweight convolutional network to extract the convolutional features of the image density map, resulting in a feature map. ;

[0020] S313, through the cross-domain feature fusion module, the feature map is processed... and Feature fusion is performed to obtain a cross-domain fused feature map. ;

[0021] S314, the obtained cross-domain fusion feature map The input is fed into the encoding / decoding network, and after network regression, a predicted density map corresponding to the original image is obtained.

[0022] S315 uses feature association regularization constraints, numerical loss function, and distributed loss function to construct the loss function of the feature association density map generation network, and obtains the model parameters of the feature association density map generation network corresponding to the minimum loss function.

[0023] Furthermore, in step S313, the feature map is processed by the cross-domain feature fusion module. and Feature fusion is performed to obtain a cross-domain fused feature map. Specifically, it includes:

[0024] feature map Batch Normalization is applied, followed by adaptive max pooling and a one-dimensional convolutional layer to obtain the weight map. ;

[0025] Will and Element-wise multiplication yields the result after passing through the chain. The weight map is obtained by processing with activation functions. Finally, the feature map extracted from the original image is... and The fused feature map can be obtained by multiplying element by element. .

[0026] Furthermore, in step S315, the loss function used to construct the feature association density map generation network using feature association regularization constraints, numerical loss function, and distributed loss function includes:

[0027] The formula for feature association regularization constraint is:

[0028]

[0029] in, Indicates the passageway characteristics of all ships. This represents the average characteristic channel of all ships. This represents the error between the channel characteristic and the average characteristic of the channel. and Satisfying the formula:

[0030] , , This represents a cross-domain feature fusion feature map. Represents the target point map. This represents an index retrieval operation;

[0031] Numerical loss is used to measure the error between the number of ships obtained by integrating the predicted density map and the actual number of ships, given a predicted density map. Image True Density Map ,use Indicates based on the true density map of the image The calculated actual number of ships, The expression is as follows:

[0032]

[0033] The numerical loss function is expressed as follows:

[0034]

[0035] in, This indicates the number of training images in each training batch. This indicates that based on the predicted density map The estimated number of ships obtained from the calculation;

[0036] The expression for the distributed loss function is:

[0037]

[0038] in, Indicates the image height. Indicates the image width. This indicates the coordinate position in the image.

[0039] Based on the aforementioned feature association regularization constraints, numerical loss function, and distributed loss function, the final loss function is... It can be represented as:

[0040]

[0041] in, This represents the weight of the regularization constraint that controls the association between features.

[0042] Furthermore, in step S320, the loss function in the pre-training of the marine vessel counting network based on residual feature fusion is the l2 loss function; the l2 loss function evaluates the difference between the predicted density map and the true density map by calculating the mean square error between the pixels of the predicted density map and the true density map, and the expression is as follows:

[0043]

[0044] In the formula, These are model parameters. This represents the original image of the input ocean vessel. This represents the true density map corresponding to the input image of a marine vessel. This represents the predicted density map corresponding to the marine vessel image input to the model.

[0045] Furthermore, the weighted combined loss function used in step S300 includes two loss functions: the loss function of the feature association density map generation network and the loss function used in the pre-training of the marine vessel counting network based on residual feature fusion. Specifically, it is expressed as follows:

[0046]

[0047] in, The loss function for generating the feature correlation density map network. The loss function in the pre-training of the marine vessel counting network based on residual feature fusion is... This represents the hyperparameters of the loss function controlling the density map prediction network.

[0048] In a second aspect, the present invention discloses a marine vessel counting computer device based on feature association density map generation, which includes a memory, a processor, and program instructions stored in the memory that can be executed by the processor, wherein the processor executes the program instructions to implement the steps of the method described in any of the first aspects.

[0049] Thirdly, the present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method described in any one of the first aspects.

[0050] The beneficial effects of this invention are as follows:

[0051] 1. High adaptability: This invention analyzes the texture and neighborhood relationships between ships and uses a feature association density map generation network to generate a density map that more closely matches the actual ship distribution. This method is not only applicable to ships with different shapes, sizes, and viewing angles, but also adapts to changes in distance and clustering, effectively improving the model's adaptability and flexibility under various conditions.

[0052] 2. Reduced overlap error: In traditional methods, the clustering of ships often leads to overlap on the density map. This invention generates a more detailed density map, which can better resolve overlapping areas, effectively distinguish and count objects that are close to or overlap each other, and improve the accuracy of counting.

[0053] 3. High model generalization ability: This invention uses feature association regularization constraints to enhance the model's comprehensive learning of ship channel features, improving the model's accuracy in identifying ships under different environments and conditions. This invention effectively balances the feature representation of different ship types in images, ensuring the model's fairness and comprehensiveness when processing diverse ship data. Attached Figure Description

[0054] Figure 1 This is an overall flowchart of a marine vessel counting method based on feature correlation density map generation according to the present invention;

[0055] Figure 2 This is a schematic diagram of a model for a counting task in remote sensing images in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the cross-domain feature fusion module mentioned in the embodiments of the present invention. Detailed Implementation

[0057] To specifically illustrate the objectives, technical solutions, advantages, and feasibility of the present invention, the invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

[0058] The first embodiment of the present invention takes a counting task in a remote sensing image as a scenario, and provides a detailed description of a marine vessel counting method based on feature association density map generation provided by the present invention.

[0059] like Figure 1 As shown, the present invention proposes a marine vessel counting method based on feature correlation density map generation, which is applied to counting tasks in remote sensing images. The detailed steps of the method are as follows:

[0060] S100 collects remote sensing image datasets and preprocesses them, dividing the preprocessed image data into training, validation, and test sets.

[0061] In this embodiment, the preprocessing of the remote sensing image dataset in step S100 specifically includes: adjusting the images in the dataset to a uniform size to obtain images. and their corresponding point labels Based on the point label information, the remote sensing images are preprocessed to obtain the true density map corresponding to each remote sensing image, which is then saved as a density point map. It should be noted that in some embodiments, the true density map is also referred to as the target point map. This data is further divided proportionally into training, validation, and test sets for subsequent model training, validation, and testing phases. The training set is defined as follows: , where N represents the total number of images in the training set.

[0062] S200 constructs a marine vessel counting model, which includes a feature-association density map generation network and a marine vessel counting network based on residual feature fusion. This model uses a joint training method to improve the quality of density map generation, such as... Figure 2 As shown, it includes:

[0063] S310: The original images and true image density maps from the training set are input into the pre-trained feature association density map generation network to generate a predicted density map based on the feature association density map. The feature association-based density map generation network is first pre-trained using the original images and true image density maps as input to learn the texture features and ship distribution features of the original images and generate a mapping from the original image features to the density map;

[0064] S320: The predicted density map generated by the feature association density map generation network and the true density map of the original image are input into the pre-trained marine vessel counting network based on residual feature fusion to obtain the total number of vessels detected in a given image region. The marine vessel counting network based on residual feature fusion is first pre-trained, using the network's predicted density map and the true density map of the original image in the training set as network inputs. It employs residual fusion to achieve efficient fusion of regional features and global contextual features.

[0065] In this embodiment, the process of pre-training the feature association density map generation network in step S310 specifically includes:

[0066] S311, the modified original image feature extraction network VGG-10 module, extracts features from remote sensing images using the VGG-10 feature extraction network to obtain feature maps. Given that ships in remote sensing images are small in size and occupy few pixels, excessive pooling operations would result in significant loss of the pixels occupied by ships. Therefore, the first ten layers of the original VGG-16 are retained as the feature extraction network, and the three max pooling layers in the original VGG-16 are also retained. This reduces the shift in the estimated mean caused by sample errors during feature extraction, thereby better preserving the texture features in the image.

[0067] S312, the image true density map feature extraction module, uses a lightweight convolutional network to extract the convolutional features of the image's true density map, obtaining the feature map. Lightweight convolutional networks reduce computational complexity and memory requirements by simplifying convolutional layers, while maintaining the ability to capture key information.

[0068] S313, Cross-domain Feature Fusion Module, such as Figure 3 As shown, this module learns corresponding weights for each feature channel and enhances the weights of the ship feature channel, making it easier for the network to distinguish between ship and background features. The cross-domain feature fusion module fuses the two feature maps obtained in steps S211 and S212. and The cross-domain fusion feature map is obtained. Specifically, the feature map Batch Normalization is applied, followed by adaptive max pooling and a one-dimensional convolutional layer to obtain the weight map. ;Will and Element-wise multiplication yields the result after passing through the chain. The weight map is obtained by processing with activation functions. Finally, the feature map extracted from the original image is... and The fused feature map can be obtained by multiplying element by element. .

[0069] S314 will fuse feature maps The input is fed into an encoder-decoder network, which then regresses to generate a predicted density map. Specifically, the cross-domain fusion feature map obtained in step S213 is used as the encoder input. The encoder gradually reduces the spatial dimension of the data through a series of convolutional and pooling layers, while extracting key features from the image. Subsequently, these features are passed to the decoder, which gradually restores the spatial dimension of the image through deconvolutional layers, while refining and refining the features, ultimately generating a predicted density map corresponding to the original image.

[0070] S315 uses feature association regularization constraints, numerical loss function, and distributed loss function to construct the loss function of the feature association density map generation network, and obtains the model parameters of the feature association density map generation network corresponding to the minimum loss function.

[0071] The feature association density map generation network in this invention uses feature association regularization constraints to learn common features of ships, which can be used to enhance the generalization ability of the model.

[0072] When there are significant differences in ship size within a remotely sensed image, the model naturally focuses on larger ships while neglecting smaller ones. Therefore, this module uses feature association regularization constraints to supervise the training of different feature channels within the same image. This aims to balance the distribution of channel features across different ship types and penalize ship channels that deviate from the average ship channel features. The expression for extracting all ship channel features is as follows:

[0073]

[0074] in, This represents a cross-domain feature fusion feature map. Represents the true density map of the image. This represents an index retrieval operation.

[0075] To reduce computational load, feature errors are measured by pairwise supervision of each ship's feature channel and the average feature channel of all ships. The formula for calculating the error G between the ship channel feature and the average feature channel is:

[0076]

[0077] in, It is the average characteristic channel of all ships.

[0078] The formula for feature association regularization constraint is:

[0079] .

[0080] In step S315, the loss function used to construct the feature association density map generation network using feature association regularization constraints, numerical loss function, and distributed loss function includes:

[0081] The formula for feature association regularization constraint is:

[0082]

[0083] in, Indicates the passageway characteristics of all ships. This represents the average characteristic channel of all ships. This represents the error between the channel characteristic and the average characteristic of the channel. and Satisfying the formula:

[0084] ;

[0085] Numerical loss is used to measure the error between the number of ships obtained by integrating the predicted density map and the actual number of ships, given a predicted density map. Image True Density Map ,use Indicates based on the true density map of the image The calculated actual number of ships, The expression is as follows:

[0086]

[0087] The numerical loss function is expressed as follows:

[0088]

[0089] in, This indicates the number of training images in each training batch. This indicates that based on the predicted density map The estimated number of ships obtained from the calculation;

[0090] The expression for the distributed loss function is:

[0091]

[0092] in, Indicates the image height. Indicates the image width. This indicates the coordinate position in the image.

[0093] Based on the aforementioned feature association regularization constraints, numerical loss function, and distributed loss function, the final loss function is... It can be represented as:

[0094]

[0095] in, This represents the weight of the regularization constraint that controls the association between features.

[0096] The ship counting network based on residual feature fusion uses the L2 loss function. It evaluates the difference between the predicted and actual density maps by calculating the mean squared error between pixels in the predicted density map and the actual density map. Using this loss function, accurate results can be obtained when the model training is complete. The expression is as follows:

[0097]

[0098] In the formula, These are model parameters. This represents the original image of the input ocean vessel. This represents the true density map corresponding to the input image of a marine vessel. This represents the predicted density map corresponding to the marine vessel image input to the model.

[0099] The weighted combined loss function used in the marine vessel counting model generated based on feature association density maps can be expressed as:

[0100]

[0101] in, This represents the hyperparameters of the loss function controlling the density map prediction network.

[0102] S400: Use the validation set constructed in step S100 to validate the trained ocean vessel counting model, and select the model with the best performance as the trained ocean vessel counting model according to the MAE evaluation index.

[0103] S500 inputs the target remote sensing image into the trained marine vessel counting model, and the marine vessel counting model outputs the total number of vessels detected in a given image area.

[0104] In a second embodiment, the present invention also proposes a marine vessel counting computer device based on feature association density map generation, which includes a memory, a processor, and program instructions stored in the memory that can be executed by the processor, wherein the processor executes the program instructions to implement any of the methods in the first embodiment.

[0105] In a third embodiment of the present invention, the present invention further provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method of any one of the first embodiments described above.

[0106] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0107] It should be understood that those skilled in the art can make improvements or modifications based on the principles of this invention and the above description, or apply the method provided by this invention to similar aerial image recognition tasks, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for counting marine vessels based on feature correlation density maps, characterized in that, The method includes the following steps: S100: Collect remote sensing image datasets and preprocess the remote sensing image datasets, dividing the preprocessed image data into training set, validation set and test set; S200, construct a marine vessel counting model, which includes a feature association density map generation network and a marine vessel counting network based on residual feature fusion; S300, input the image data from the training set into the ocean vessel counting model, perform joint training on the model using a weighted combined loss function, and update the ocean vessel counting model through a backpropagation algorithm until the ocean vessel counting model with the minimum loss function is obtained; step S300 specifically includes the following steps: S310: Input the original images and the true image density maps in the training set into the pre-trained feature association density map generation network, and generate the predicted density map based on the feature association density map; S320, the predicted density map generated by the feature association density map generation network and the original image true density map are input into the pre-trained marine vessel counting network based on residual feature fusion to obtain the total number of vessels detected in a given image region; Step S310 specifically includes the following steps: S311, the original images in the training set are input into the improved VGG-10 feature extraction network for feature extraction to obtain feature maps. The VGG-10 feature extraction network retains the first ten layers of the original VGG-16 as the feature extraction network, and retains the three max pooling layers in the original VGG-16. S312, the true density map of the image is input into a lightweight convolutional network to extract the convolutional features of the true density map of the image, resulting in a feature map. ; S313, through the cross-domain feature fusion module, the feature map is processed... and Feature fusion is performed to obtain a cross-domain fused feature map. ; S314, the obtained cross-domain fusion feature map The input is fed into the encoding / decoding network, and after network regression, a predicted density map corresponding to the original image is obtained. S315, use feature association regularization constraints, numerical loss function and distributed loss function to construct the loss function of the feature association density map generation network, and obtain the model parameters of the feature association density map generation network corresponding to the minimum loss function; In step S313, the feature map is processed by the cross-domain feature fusion module. and Feature fusion is performed to obtain a cross-domain fused feature map. Specifically, it includes: feature map Batch Normalization is applied, followed by adaptive max pooling and a one-dimensional convolutional layer to obtain the weight map. ; Will and Element-wise multiplication yields the result after passing through the chain. The weight map is obtained by processing with activation functions. Finally, the feature map extracted from the original image is... and The fused feature map can be obtained by multiplying element by element. ; S400: Use the validation set constructed in step S100 to validate the trained ocean vessel counting model, and select the model with the best performance as the trained ocean vessel counting model according to the MAE evaluation index. S500 inputs the target remote sensing image into the trained marine vessel counting model, and the marine vessel counting model outputs the total number of vessels detected in a given image area.

2. The marine vessel counting method based on feature correlation density map generation according to claim 1, characterized in that, Step S100 specifically includes: Acquire remote sensing image data, resize the images in the remote sensing image dataset to a uniform size, and obtain the image. and their corresponding point labels Based on the point label information, the remote sensing images are preprocessed to obtain the true density map corresponding to each remote sensing image, which is then saved as a density point map. The training set is defined as follows: , where N represents the total number of images in the training set.

3. The marine vessel counting method based on feature correlation density map generation according to claim 1, characterized in that, The loss function used in step S315 to construct the feature association density map generation network using feature association regularization constraints, numerical loss function, and distributed loss function includes: The formula for feature association regularization constraint is: ; in, Indicates the passageway characteristics of all ships. This represents the average characteristic channel of all ships. This represents the error between the channel characteristic and the average characteristic of the channel. and Satisfying the formula: , , This represents a cross-domain feature fusion feature map. Represents the target point map. This represents an index retrieval operation; Numerical loss is used to measure the error between the number of ships obtained by integrating the predicted density map and the actual number of ships, given a predicted density map. Image True Density Map ,use Indicates based on the true density map of the image The calculated actual number of ships, Indicates the coordinate position in the image. The expression is as follows: ; The numerical loss function is expressed as follows: ; in, This indicates the number of training images in each training batch. This indicates that based on the predicted density map The estimated number of ships obtained from the calculation; The expression for the distributed loss function is: ； in, Indicates the image height. Indicates the image width; Based on the aforementioned feature association regularization constraints, numerical loss function, and distributed loss function, the final loss function is... It can be represented as: ; in, This represents the weight of the regularization constraint that controls the association between features.

4. The marine vessel counting method based on feature correlation density map generation according to claim 1, characterized in that, In step S320, the loss function in the pre-training of the marine vessel counting network based on residual feature fusion is the l2 loss function. The l2 loss function evaluates the difference between the predicted density map and the real density map by calculating the mean square error between the pixels of the two maps, and its expression is as follows: ； In the formula, These are model parameters. This represents the original image of the input ocean vessel. This represents the true density map corresponding to the input image of a marine vessel. This represents the predicted density map corresponding to the marine vessel image input to the model.

5. The marine vessel counting method based on feature correlation density map generation according to claim 1, characterized in that, The weighted combined loss function used in step S300 includes two loss functions: the loss function of the feature association density map generation network and the loss function used in the pre-training of the marine vessel counting network based on residual feature fusion. Specifically, it is expressed as follows: ; in, The loss function for generating the feature correlation density map network. The loss function in the pre-training of the marine vessel counting network based on residual feature fusion is... This represents the hyperparameters of the loss function controlling the density map prediction network.

6. A computer device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to perform the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images