A ship identification model training method, device, equipment and medium

Abstract

The present invention discloses a training method, device, equipment and medium for a ship recognition model, and relates to the field of ship classification. The training method for the ship recognition model comprises: performing feature extraction processing on a first ship image and a second ship image through a pre-constructed ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image; and mixing the second image feature with the first image feature to obtain a mixed feature, wherein the mixed feature comprises a ship texture feature of the second ship image and a ship posture feature of the first ship image; using a ship classification model in the ship recognition model, performing ship recognition on the mixed feature to obtain a probability distribution of a predicted ship type; and training the ship recognition model based on the predicted probability distribution of the ship type, the first ship type of the first ship image and the second ship type of the second ship image.

Description

Technical Field

[0001] The present invention relates to the field of ship classification, and in particular to a training method, device, equipment and medium for a ship identification model. Background Art

[0002] As an important marine monitoring task, marine ship detection and classification is of great significance in marine safety, maritime traffic management, etc. Compared with traditional optical, infrared, hyperspectral and other remote sensing systems, SAR (Synthetic Aperture Radar) can operate all day without being restricted by natural light and is not affected by weather conditions. Moreover, SAR can generate high-resolution images with more detailed target information, and has gradually become a key device for ship detection.

[0003] In recent years, deep learning technology has attracted widespread attention in the field of SAR ship classification and has become a research focus. However, using deep learning to classify ships in SAR images faces many challenges. Compared with optical image data, SAR datasets are relatively scarce and data annotation is difficult, which poses a significant obstacle to deep learning methods that rely on a large number of data samples.

[0004] Therefore, how to use the existing SAR data set to train the ship recognition model and improve the accuracy and generalization ability of the ship model has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a training method, device, equipment and medium for a ship identification model. The method solves the problem of the relative scarcity of SAR data sets based on the use of existing SAR data sets, which is conducive to improving the accuracy and generalization ability of the ship model.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for training a ship identification model, comprising:

[0008] Performing feature extraction processing on the first ship image and the second ship image by using a pre-built ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image;

[0009] and, mixing the second image feature with the first image feature to obtain a mixed feature, wherein the mixed feature includes a ship texture feature of the second ship image and a ship posture feature of the first ship image;

[0010] Using the ship classification model in the ship identification model, the mixed features are used to perform ship identification to obtain a probability distribution of predicted ship types;

[0011] The ship recognition model is trained based on the probability distribution of the predicted ship type, the first ship type of the first ship image, and the second ship type of the second ship image.

[0012] In an optional implementation manner of the present application, the performing feature extraction processing on the first ship image and the second ship image to obtain a first image feature of the first ship image and a second image feature of the second ship image includes:

[0013] Performing image detail enhancement on the first ship image and the second ship image to obtain the first ship image after image detail enhancement and the second ship image after image detail enhancement;

[0014] An encoder is used to perform feature extraction processing on the first ship image after image detail enhancement and the second ship image after image detail enhancement to obtain the first image feature and the second image feature.

[0015] In an optional implementation manner of the present application, the step of mixing the second image feature with the first image feature to obtain the mixed feature includes:

[0016] Performing feature alignment on the second image feature and the first image feature to obtain an aligned third image feature;

[0017] The third image feature is mixed with the first image feature to obtain the mixed feature.

[0018] In an optional implementation manner of the present application, aligning the second image feature with the first image feature to obtain an aligned third image feature includes:

[0019] constructing an alignment matrix based on a distance between the first image feature and the second image feature;

[0020] According to the alignment matrix, the second image feature is aligned with the first image feature to obtain an aligned third image feature.

[0021] In an optional implementation manner of the present application, the second image feature is mixed with the first image feature by the following formula to obtain a mixed feature:

[0022] ;

[0023] Wherein, H represents the mixed feature; λ represents the weight of the first image feature, ; A represents the first image feature; represents the third image feature; It means that when the weight of the first image feature is λ and the weight of the third image feature is 1-λ, the first image feature and the third image feature are mixed.

[0024] In an optional implementation manner of the present application, the training of the ship recognition model based on the predicted probability distribution of the ship type, the first ship type of the first ship image, and the second ship type of the second ship image includes:

[0025] constructing a loss function of the ship identification model according to the probability distribution of the predicted ship type, the first ship type, and the second ship type;

[0026] Based on the loss function, the ship identification model is trained.

[0027] In an optional implementation of the present application, the loss function is expressed by the following formula:

[0028] ;

[0029] in, represents the classification loss; λ represents the weight of the first image feature in the mixed feature, ; pred represents the probability distribution of the predicted ship type; y represents the first ship type; represents the second ship type; represents a cross entropy loss between the probability distribution of the predicted ship type and the first ship type; Represents the cross entropy loss between the probability distribution of the predicted ship type and the second ship type.

[0030] In an optional implementation of the present application, it also includes:

[0031] The image of the ship to be identified is input into the ship identification model, so as to classify the ships in the image of the ship to be identified by the ship identification model to obtain a ship classification result of the image of the ship to be identified.

[0032] Compared with the prior art, the training method of the ship recognition model provided by the present invention extracts the first image features of the first ship image and the second ship image features of the second ship image through the ship recognition model, performs feature alignment and mixing, constructs a mixed feature including the ship texture features of the second ship image and the ship posture features of the first ship image, and further performs ship recognition on the mixed feature through the ship recognition model to obtain a probability distribution of predicted ship types, and at the same time trains the ship recognition model in combination with the probability distribution of predicted ship types, the first ship type of the first ship image, and the second ship type of the second ship image. The scheme can construct new sample data (the mixed feature) based on limited sample data (the first ship image and the second ship image) to train the ship recognition model. The method utilizes the existing SAR data set, solves the problem of relative scarcity of SAR data sets, and is conducive to improving the accuracy and generalization ability of the ship model.

[0033] The present invention also provides a training device for a ship identification model, comprising:

[0034] a feature extraction unit, configured to perform feature extraction processing on the first ship image and the second ship image by using a pre-built ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image;

[0035] a feature fusion unit, configured to mix the second image feature with the first image feature to obtain a mixed feature, wherein the mixed feature includes a ship texture feature of the second ship image and a ship posture feature of the first ship image;

[0036] A ship prediction unit, configured to use a ship classification model in the ship recognition model to perform ship recognition on the mixed features to obtain a probability distribution of the predicted ship type;

[0037] A model training unit is used to train the ship recognition model based on the probability distribution of the predicted ship type, the first ship type of the first ship image, and the second ship type of the second ship image.

[0038] Compared with the prior art, the beneficial effects of the ship identification model training device provided by the present invention are the same as the beneficial effects of the ship identification model training method described in the above technical solution, which will not be described in detail here.

[0039] The present invention also provides an electronic device, comprising:

[0040] processor;

[0041] a memory for storing instructions executable by the processor;

[0042] The processor is used to execute the training method of the ship identification model by running the instructions in the memory.

[0043] Compared with the prior art, the beneficial effects of the electronic device provided by the present invention are the same as the beneficial effects of the training method of the ship identification model described in the above technical solution, which will not be described in detail here.

[0044] The present invention also provides a computer storage medium, in which instructions are stored. When the instructions are executed, the training method of the ship identification model is implemented.

[0045] Compared with the prior art, the beneficial effects of the computer storage medium provided by the present invention are the same as the beneficial effects of the training method of the ship identification model described in the above technical solution, and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0047] Figure 1 Training process of the ship identification model provided in the embodiment of the present application Figure 1 ;

[0048] Figure 2 A characteristic mixing effect diagram provided in an embodiment of the present application;

[0049] Figure 3 Training process of the ship identification model provided in the embodiment of the present application Figure 2 ;

[0050] Figure 4 A structural diagram of a training device for a ship identification model provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0053] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0054] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0055] As an important marine monitoring task, marine ship detection and classification is of great significance in marine safety, maritime traffic management, etc. Compared with traditional optical, infrared, hyperspectral and other remote sensing systems, SAR (Synthetic Aperture Radar) can operate all day without being restricted by natural light and is not affected by weather conditions. Moreover, SAR can generate high-resolution images with more detailed target information, and has gradually become a key device for ship detection.

[0056] In recent years, deep learning technology has attracted widespread attention in the field of SAR ship classification and has become a research focus. However, using deep learning to classify ships in SAR images faces many challenges. Compared with optical image data, SAR datasets are relatively scarce and data annotation is difficult, which poses a significant obstacle to deep learning methods that rely on a large number of data samples.

[0057] Therefore, how to use the existing SAR data set to train the ship recognition model and improve the accuracy and generalization ability of the ship model has become a technical problem that technical personnel in this field urgently need to solve.

[0058] In order to solve the above technical problems, the embodiments of the present application provide a method, device, equipment and medium for training a ship identification model, which are described in detail one by one in the following embodiments.

[0059] The present application embodiment first provides a method for training a ship recognition model. Figure 1 , Figure 1 Training process of the ship identification model provided in the embodiment of the present application Figure 1 .

[0060] like Figure 1 As shown, the training method of the ship identification model includes the following S101 to S104:

[0061] S101, performing feature extraction processing on a first ship image and a second ship image using a pre-constructed ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image.

[0062] The ship identification model can be understood as a machine learning model. In an optional embodiment of the present application, the ship identification model can be built based on the ShuffleNetV2 convolutional neural network. The ship identification model mainly includes a feature extraction network, a feature mixing module and a classification model.

[0063] The feature extraction network is used to execute the above-mentioned steps of performing feature extraction processing on the first ship image and the second ship image, and output the first image features and the second image features.

[0064] In addition, considering that the first ship image and the second ship image are SAR images, different parts of the same target in each image may show different scattering characteristics due to structural differences, which leads to incomplete contours and other features of the first ship image and the second ship image, affecting the accuracy of ship classification. Therefore, the ship recognition model also includes a detail enhancement module.

[0065] The detail enhancement module is used to enhance the image details of the image input into the ship recognition model. Specifically, the detail enhancement module includes: center difference convolution, angle difference convolution, horizontal difference convolution, vertical difference convolution and an ordinary convolution.

[0066] Among them, the center difference convolution is used to capture the detail changes around the center pixel; the angle difference convolution is used to capture the detail changes along a specific angle; the horizontal difference convolution is used to capture the detail changes in the horizontal direction; the vertical difference convolution is used to capture the detail changes in the vertical direction; the ordinary convolution is used to extract the intensity level information, that is, the brightness or grayscale information of the image.

[0067] The above convolutional layers use 2D kernels of the same size and operate on the input ship image with the same stride and padding. These kernels are added at the same position to obtain equivalent kernels so that the kernels can produce equivalent outputs. In practical applications, the above content can be achieved through reparameterization technology.

[0068] After performing image detail enhancement on the first ship image and the second ship image and obtaining the first ship image after image detail enhancement and the second ship image after image detail enhancement, feature extraction processing can be performed on the first ship image and the second ship image through the feature extraction network.

[0069] In an optional implementation of the present application, the feature extraction network can be constructed by an encoder to realize feature extraction of the first ship image and the second ship image; corresponding algorithms can also be used to realize feature extraction of the first ship image and the second ship image, such as edge detection algorithms such as Canny edge detection, Sobel operator, Laplacian operator, etc., can be used to extract boundary and contour features in the first ship image and the second ship image, and a method based on Gray-Level Co-occurrence Matrix (GLCM) or Local Binary Pattern (LBP) can be used to extract texture features of the image; corner detection algorithms such as Harris corner detection, Shi-Tomasi corner detection, FAST (Features from Accelerated Segment Test), etc. are used to detect features of key points in the first ship image and the second ship image (e.g., extract bow key points to obtain posture features of the ships in the first ship image and the second ship image), etc.

[0070] S102: Mix the second image feature with the first image feature to obtain a mixed feature, where the mixed feature includes a ship texture feature of the second ship image and a ship posture feature of the first ship image.

[0071] Specifically, the above S102 includes the following S11 and S21:

[0072] S11, aligning the second image feature with the first image feature to obtain an aligned third image feature.

[0073] Specifically, it includes: constructing an alignment matrix based on the distance between the first image feature and the second image feature; and aligning the second image feature with the first image feature according to the alignment matrix to obtain an aligned third image feature.

[0074] The constructing of the alignment matrix based on the distance between the first image feature and the second image feature may be implemented by the following method:

[0075] Assume that the first image feature is , the second image feature is , and The shape is , where c represents and The number of channels of the corresponding feature tensor, w and h represent and The width and height of the feature tensor of . and The spatial dimension of is flattened to , r represents the flattened spatial dimension, and and Represent the flattened features and ,in, , n=1,...,r.

[0076] Furthermore, we define D as The distance matrix of and The distance between all elements in , where and The distance between them is expressed by the following formula (1):

[0077] (1);

[0078] in, , is the element in the i-th row and j-th column of the distance matrix D; Representation characteristics and The distance between Representation characteristics The i-th element of ; Representation characteristics The jth element in .

[0079] Furthermore, define Reason Towards or by Towards The feasible domains that meet all the conditions are expressed by the following formula (2):

[0080] (2);

[0081] Where E is The all-1 vector in the feasible domain All elements in are non-negative, and the sum of rows and columns is 1 / r, indicating the joint probability of having uniform edges in spatial position; at this time, we need to find an optimal transformation matrix , so that and The overall distance between them is minimal.

[0082] Furthermore, entropy regularization is introduced to make the calculation and The feasible region is smoother when the distance between is:

[0083] Specifically, the entropy regularization process can be expressed by the following formulas (3) and (4):

[0084] (3);

[0085] (4);

[0086] in, represents the entropy of P; represents the matrix inner product; represents the regularization coefficient; represents the optimal transformation matrix, which can be quickly solved by the Sinkhorn-Knopp iterative algorithm.

[0087] Furthermore, after obtaining the optimal transformation matrix After that, according to the optimal change matrix , aligning the second image feature with the first image feature to obtain an aligned third image feature.

[0088] Specifically, the alignment process can be expressed by the following formula (5):

[0089] (5);

[0090] in, represents the second image feature; represents the third image feature; R is The alignment matrix, , the column of the first image feature A after alignment Corresponding to the second image feature Column The allocation weight of .

[0091] S21, performing feature mixing of the third image feature and the first image feature to obtain the mixed feature.

[0092] Further, after obtaining the third image feature Then, the third image feature Convert back , the third image feature Mixed with the first image A to obtain the mixed feature. Specifically, the above S21 can be implemented by the following formula (6):

[0093] (6);

[0094] Wherein, H represents the mixed feature; λ represents the weight in the first image feature, , the value is taken from Beta Distribution (Beta Distribution); A represents the first image feature; represents the third image feature; It means that when the weight of the first image feature is λ and the weight of the third image feature is 1-λ, the first image feature and the third image feature are mixed.

[0095] For further information, please refer to Figure 2 , Figure 2 A characteristic mixing effect diagram provided for an embodiment of the present application.

[0096] like Figure 2 As shown, Figure 2 The image x includes a ship image x and a ship image x', wherein the ship image x is specifically an image of a bulk carrier, and the ship image x' is specifically an image of an oil tanker, and the two have different ship texture features and ship posture features.

[0097] Based on the above formula (6), choose the appropriate (like ) Mix the image features of the ship image x and the ship image x' to generate mixed features, wherein each mixed feature corresponds to a different ship feature mixed image.

[0098] In the actual application process, Figure 2 The mixed image of ship features shown can be restored by decoding the mixed features through a decoder.

[0099] S103, using the ship classification model in the ship identification model, perform ship identification on the mixed features to obtain a probability distribution of predicted ship types.

[0100] The ship classification model is a part of the ship recognition model. The ship classification model can predict the mixed features to obtain the probability distribution of the predicted ship type of the mixed features, that is, by mixing the mixed features of the ship texture features of the second ship image and the ship posture features of the first ship image, the probability distribution of the ship type corresponding to the mixed features can be identified.

[0101] S104: Training the ship recognition model based on the predicted probability distribution of the ship type, the first ship type of the first ship image, and the second ship type of the second ship image.

[0102] Specifically, the above S104 includes:

[0103] According to the probability distribution of the predicted ship type, the first ship type and the second ship type, a loss function of the ship identification model is constructed; and based on the loss function, the ship identification model is trained.

[0104] Specifically, the loss function can be expressed by the following formula (7):

[0105] (7);

[0106] in, represents the classification loss; λ represents the weight of the first image feature in the mixed feature, ; pred represents the probability distribution of the predicted ship type; y represents the first ship type; represents the second ship type; represents a cross entropy loss between the probability distribution of the predicted ship type and the first ship type; Represents the cross entropy loss between the probability distribution of the predicted ship type and the second ship type.

[0107] Further, in order to facilitate the understanding of the above training process, the following is combined Figure 3 The process is described in detail.

[0108] Please refer to Figure 3 , Figure 3 Training process of the ship identification model provided in the embodiment of the present application Figure 2 .

[0109] like Figure 3 As shown, the training of the ship identification model includes:

[0110] S1, initialize the parameters of the ship identification model.

[0111] S2, input the training set image x and the classification label y of the training set image x, the training set image x' and the classification label y' of the training set image x', and the validation set image u and the classification label v of the validation set image u.

[0112] Among them, the training set image x and the training set image x' can be understood as the first ship image and the second ship image in the above S101, the classification label y and the classification label y' can be understood as the first ship type of the first ship image and the second ship type of the second ship image mentioned in the above S104; the verification set image u and the classification label v of the verification set image u can be understood as a data set for evaluating the classification accuracy of the ship recognition model.

[0113] S3, performing image detail enhancement processing on the training set images x and the training set images x', and performing feature extraction on the training set images x and the training set images x' after the image detail enhancement processing through the ship recognition model to obtain image features A and image features A'.

[0114] S4, align and mix the image feature A' with the image feature A to obtain a mixed feature H.

[0115] Specifically, the alignment and mixing of the image feature A' with the image feature A can be achieved by the above formulas (5) and (6). In the above formula (6) in the embodiment of the present application, the value of λ is random and obeys the Beta distribution. For relevant details, please refer to the relevant introduction of formulas (5) and (6) in the previous text, which will not be repeated here.

[0116] S5, using the ship classification model in the ship recognition model to identify the mixed features, obtain the probability distribution pred of the predicted ship type, and combine the classification label y, the classification label y' and the probability distribution pred of the predicted ship type to determine the classification loss of the ship recognition model.

[0117] Specifically, the classification loss of the ship recognition model is determined by combining the classification label y, the classification label y' and the predicted ship type pred, which can be achieved by formula (7). For related content, please refer to the above introduction to formula (7), which will not be repeated here.

[0118] S6, based on the weights of each level of the ship recognition model, classify the ships in the verification set image u to obtain the probability distribution pred' of the predicted ship type of the verification set image u, determine the predicted ship type v' according to the probability distribution pred' of the predicted ship type, and determine the classification accuracy of the ship recognition model according to the predicted ship type v' and the classification label v of the verification set image u.

[0119] The above S6 can be understood as the verification stage of a round of training of the ship recognition model. In this stage, the verification set image u and the classification label v of the verification set image u are used alone to evaluate the classification accuracy of the ship recognition model, that is, the ship recognition is performed on the verification set image u through the ship recognition model after a round of training, and then the probability distribution pred' of the predicted ship type is obtained. After that, the predicted ship type v' of the verification set image u is determined according to the probability distribution pred'. Finally, the classification accuracy is determined by judging the probability that the predicted ship type v' is the same as the classification label v of the verification set image u.

[0120] S7, determining whether a training round has been reached.

[0121] If the training round is not reached, execute S8.

[0122] When the training round is reached, S9 is executed.

[0123] S8, based on the classification loss, updates the weights of each level of the ship identification model, and returns to execute S2.

[0124] Specifically, updating the weights of each level of the ship recognition model based on the classification loss can be regarded as back-propagating the classification loss, and calculating the partial derivative of the loss function with respect to the weight of each neuron layer by layer as the gradient of the objective function with respect to the weight, and modifying the weights of each level according to the gradient.

[0125] S9, end the training and save the weights of the round with the best classification accuracy.

[0126] Furthermore, the ship identification model training method provided in the present application further includes the following S105:

[0127] S105, inputting the image of the ship to be identified into the ship classification model, so as to classify the ships in the image to be identified by the ship classification model, and obtain a ship classification result of the image of the ship to be identified.

[0128] That is, the ships displayed in the ship images to be identified can be directly classified using the trained ship classification model.

[0129] In an optional implementation manner of the present application, the ship image to be identified can be obtained by processing raw echo data collected by satellite-borne or airborne SAR using a range-Doppler algorithm.

[0130] In addition, in order to eliminate the influence of the land content involved in the ship image data collected by satellite or airborne SAR on the subsequent ship classification, the ship image to be identified can also be processed by sea and land segmentation, that is, a land mask of the ship image to be identified is made using high-precision coastal data lines, and the sea and land segmentation of the ship image to be identified is achieved through the sea and land segmentation tool in the Gdal library (Geospatial Data Abstraction Library).

[0131] Furthermore, in order to eliminate noise interference and image coherent waves of the image of the ship to be identified collected by satellite or airborne SAR, the Gdal library can be used to crop the image of the ship to be identified into sub-images of a preset size, and each sub-image can be subjected to radiation calibration and atmospheric correction processing, and the azimuth fuzzy noise can be suppressed based on the two-dimensional convolution function 2D-CF, while the navigation contrast can be enhanced; and the Garrote threshold denoising algorithm can be used to suppress the image coherent waves of the ship to be identified.

[0132] Furthermore, in order to further eliminate the influence of other factors other than ships in the image of the ship to be identified after the sea and land segmentation on the ship classification, it is also possible to extract pixels related to the ship in the image of the ship to be identified after the sea and land segmentation based on a target detection model, such as Faster R-CNN (Faster Region-based Convolutional Neural Network), and obtain image slices with only the ship. In an optional implementation of the present application, the size of the slice can be unified as 512*512.

[0133] Finally, the image slice is input into the ship recognition model to determine the category of the ship in the ship image to be recognized.

[0134] In summary, the training method of the ship recognition model provided in the present application extracts the first image features of the first ship image and the second ship image features of the second ship image through the ship recognition model, performs feature alignment and mixing, and constructs a mixed feature including the ship texture features of the second ship image and the ship posture features of the first ship image, and further performs ship recognition on the mixed feature through the ship recognition model to obtain the probability distribution of the predicted ship type, and at the same time, trains the ship recognition model in combination with the probability distribution of the predicted ship type, the first ship type of the first ship image, and the second ship type of the second ship image. This scheme can construct new sample data (the mixed feature) based on limited sample data (the first ship image and the second ship image) to train the ship recognition model. This method utilizes the existing SAR data set, solves the problem of relative scarcity of SAR data sets, and is conducive to improving the accuracy and generalization ability of the ship model.

[0135] Corresponding to the above method embodiment, the present application embodiment also provides a training device for a ship identification model, please refer to Figure 4 , Figure 4 A structural diagram of a training device for a ship identification model provided in an embodiment of the present application.

[0136] like Figure 4 As shown, the training device of the ship identification model comprises:

[0137] The feature extraction unit 401 is used to perform feature extraction processing on the first ship image and the second ship image by using a pre-built ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image;

[0138] A feature fusion unit 402 is used to mix the second image feature with the first image feature to obtain a mixed feature, wherein the mixed feature includes a ship texture feature of the second ship image and a ship posture feature of the first ship image;

[0139] A ship prediction unit 403 is used to perform ship recognition on the mixed features by using the ship classification model in the ship recognition model to obtain a probability distribution of predicted ship types;

[0140] The model training unit 404 is used to train the ship recognition model based on the probability distribution of the predicted ship type, the first ship type of the first ship image, and the second ship type of the second ship image.

[0141] In an optional implementation manner of the present application, the performing feature extraction processing on the first ship image and the second ship image to obtain a first image feature of the first ship image and a second image feature of the second ship image includes:

[0142] Performing image detail enhancement on the first ship image and the second ship image to obtain the first ship image after image detail enhancement and the second ship image after image detail enhancement;

[0143] An encoder is used to perform feature extraction processing on the first ship image after image detail enhancement and the second ship image after image detail enhancement to obtain the first image feature and the second image feature.

[0144] In an optional implementation manner of the present application, it is characterized in that the step of mixing the second image feature with the first image feature to obtain the mixed feature comprises:

[0145] Performing feature alignment on the second image feature and the first image feature to obtain an aligned third image feature;

[0146] The third image feature is mixed with the first image feature to obtain the mixed feature.

[0147] In an optional implementation manner of the present application, it is characterized in that the step of aligning the second image feature with the first image feature to obtain an aligned third image feature includes:

[0148] constructing an alignment matrix based on a distance between the first image feature and the second image feature;

[0149] According to the alignment matrix, the second image feature is aligned with the first image feature to obtain an aligned third image feature.

[0150] In an optional implementation manner of the present application, the second image feature is mixed with the first image feature by the following formula to obtain a mixed feature:

[0151] ;

[0152] Wherein, H represents the mixed feature; λ represents the weight of the first image feature in the mixed feature, ; A represents the first image feature; represents the third image feature; It means that when the weight of the first image feature is λ and the weight of the third image feature is 1-λ, the first image feature and the third image feature are mixed.

[0153] In an optional implementation manner of the present application, the training of the ship recognition model based on the predicted probability distribution of the ship type, the first ship type of the first ship image, and the second ship type of the second ship image includes:

[0154] constructing a loss function of the ship identification model according to the probability distribution of the predicted ship type, the first ship type, and the second ship type;

[0155] Based on the loss function, the ship identification model is trained.

[0156] In an optional implementation of the present application, the loss function is expressed by the following formula:

[0157] ;

[0158] in, represents the classification loss; λ represents the weight of the first image feature, ; pred represents the probability distribution of the predicted ship type; y represents the first ship type; represents the second ship type; represents a cross entropy loss between the probability distribution of the predicted ship type and the first ship type; Represents the cross entropy loss between the probability distribution of the predicted ship type and the second ship type.

[0159] In an optional implementation of the present application, the training device for the ship identification model is further used for:

[0160] The image of the ship to be identified is input into the ship identification model, so as to classify the ships in the image of the ship to be identified by the ship identification model to obtain a ship classification result of the image of the ship to be identified.

[0161] The above-mentioned device embodiment provided in this embodiment belongs to the same application concept as the method embodiment of this application, and can execute the training method of the ship identification model provided in any of the above-mentioned embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the training method of the ship identification model. For the technical details not described in detail in this embodiment, please refer to the specific processing content of the training method of the ship identification model provided in the above-mentioned embodiment of this application, and will not be repeated here.

[0162] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0163] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0164] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0165] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0166] The present application also provides an electronic device, such as Figure 5 As shown, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0167] like Figure 5 As shown, the electronic device comprises:

[0168] Processor 210;

[0169] A memory 200 for storing instructions executable by the processor 210;

[0170] The processor 210 is used to execute the ship identification model training method disclosed in any of the above embodiments by running the instructions in the memory 200.

[0171] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0172] A bus may include a pathway that transfers information between components of a computer system.

[0173] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0174] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0175] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0176] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a touch screen, etc.

[0177] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0178] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0179] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement each step of the training method of any ship identification model provided in the above embodiments of the present application.

[0180] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the training method of the ship identification model of various embodiments of the present application.

[0181] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0182] In addition, the embodiments of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps in the training method of the ship identification model of various embodiments of the present application.

[0183] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0184] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0186] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0188] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0190] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0191] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0192] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0193] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a ship identification model, characterized in that: include: Performing feature extraction processing on the first ship image and the second ship image by using a pre-built ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image; and, mixing the second image feature with the first image feature to obtain a mixed feature, comprising: constructing an alignment matrix based on the distance between the first image feature and the second image feature; performing feature alignment on the second image feature to the first image feature according to the alignment matrix to obtain an aligned third image feature; performing feature mixing on the third image feature and the first image feature to obtain the mixed feature, wherein the mixed feature comprises a ship texture feature of the second ship image and a ship attitude feature of the first ship image, and the ship texture features and ship attitude features of the first ship image and the second ship image are different; Using the ship classification model in the ship identification model, the mixed features are used to perform ship identification to obtain a probability distribution of predicted ship types; training the ship recognition model based on the predicted probability distribution of the ship type, the first ship type of the first ship image, and the second ship type of the second ship image; The step of constructing an alignment matrix based on the distance between the first image feature and the second image feature includes: The distance between the flattened first image feature and the flattened second image feature is calculated by the following formula; ； ； in, represents the i-th element in the first image feature after flattening, represents the j-th element in the second image feature after flattening; is the element in the i-th row and j-th column of the distance matrix, representing the feature and The distance between Reason Towards or by Towards Variations of represents the first image feature after flattening, represents the second image feature after flattening; E is All 1 vectors in the feasible domain All elements in are non-negative, and the sum of the values ​​of the elements in the row or column is 1 / r, which represents the joint probability of having a uniform edge in the spatial position, and the spatial dimensions of the first image feature and the second image feature are flattened to c×r, r represents the flattened spatial dimension, and c represents the number of channels of the feature tensor corresponding to the first image feature and the second image feature; Entropy regularization is used to make the feasible domain when calculating the distance between the flattened first image feature and the flattened second image feature smoother, so as to obtain an optimal transformation matrix; According to the optimal transformation matrix, the alignment matrix is ​​obtained; wherein the alignment matrix is ​​expressed by the following formula: ； Among them, R is The alignment matrix, Denotes the optimal transformation matrix.

2. The ship identification model training method according to claim 1, characterized in that: The performing feature extraction processing on the first ship image and the second ship image to obtain a first image feature of the first ship image and a second image feature of the second ship image includes: Performing image detail enhancement on the first ship image and the second ship image to obtain the first ship image after image detail enhancement and the second ship image after image detail enhancement; An encoder is used to perform feature extraction processing on the first ship image after image detail enhancement and the second ship image after image detail enhancement to obtain the first image feature and the second image feature.

3. The ship identification model training method according to claim 1, characterized in that: The second image feature is mixed with the first image feature by the following formula to obtain a mixed feature: ； Wherein, H represents the mixed feature; λ represents the weight of the first image feature in the mixed feature, ; A represents the first image feature; represents the third image feature; It means that when the weight of the first image feature is λ and the weight of the third image feature is 1-λ, the first image feature and the third image feature are mixed.

4. The ship identification model training method according to claim 1, characterized in that: The training of the ship recognition model based on the predicted probability distribution of the ship type, the first ship type of the first ship image, and the second ship type of the second ship image includes: constructing a loss function of the ship identification model according to the probability distribution of the predicted ship type, the first ship type, and the second ship type; Based on the loss function, the ship identification model is trained.

5. The ship identification model training method according to claim 4, characterized in that: The loss function is expressed by the following formula: ； in, represents the classification loss; λ represents the weight of the first image feature, ; pred represents the probability distribution of the predicted ship type; y represents the first ship type; represents the second ship type; represents a cross entropy loss between the probability distribution of the predicted ship type and the first ship type; Represents the cross entropy loss between the probability distribution of the predicted ship type and the second ship type.

6. The ship identification model training method according to claim 1, characterized in that: Also includes: The image of the ship to be identified is input into the ship identification model, so as to classify the ships in the image of the ship to be identified by the ship identification model to obtain a ship classification result of the image of the ship to be identified.

7. A training device for a ship identification model, characterized in that: A method for training a ship identification model according to any one of claims 1 to 6, comprising: a feature extraction unit, configured to perform feature extraction processing on the first ship image and the second ship image by using a pre-built ship recognition model to obtain a first image feature of the first ship image and a second image feature of the second ship image; a feature fusion unit, configured to mix the second image feature with the first image feature to obtain a mixed feature, wherein the mixed feature includes a ship texture feature of the second ship image and a ship posture feature of the first ship image; A ship prediction unit, used to use the ship classification model in the ship identification model to perform ship identification on the mixed features to obtain a probability distribution of predicted ship types; A model training unit is used to train the ship recognition model based on the probability distribution of the predicted ship type, the first ship type of the first ship image, and the second ship type of the second ship image.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the ship identification model training method described in any one of claims 1 to 6 by running the instructions in the memory.

Images