Unsupervised ship detection method and system based on multi-modal model

Abstract

The invention discloses an unsupervised ship detection method and system based on a multi-modal model. According to the method, an original ship detection model is modified, and a matching module is added to the original ship detection model, so that feature matching supervision is added to ship detection features, and the accuracy of ship detection is improved; a multi-modal model is adopted to perform automatic calibration on the training set, manual calibration does not need to be performed, and the efficiency is greatly improved; and the ship detection model adds multi-modal model supervision in a loss structure, so that the accuracy of ship detection is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship detection, and in particular to an unsupervised ship detection method and system based on a multimodal model. Background Art

[0002] With the rapid development of water transport technology, water traffic safety is becoming more and more important. The location of ships is one of the important factors in water traffic. However, due to the limited number and types of training set data, the detection rate and robustness of the current ship detection model are limited. It is impossible to achieve self-training of ship detection and update model weights in real time, thereby improving the detection rate and robustness of the ship detection model. At present, the ship detection model has the following defects: (1) The current supervised detection model requires manual calibration, which consumes a lot of manpower; (2) Since it is impossible to collect ship datasets in some scenarios, the ship training set has limitations; (3) Due to the lack of manual calibration, the unsupervised model lacks effective constraints, resulting in instability.

[0003] Currently, no effective solution has been proposed to the above problems. Summary of the invention

[0004] In an embodiment of the present invention, an unsupervised ship detection method based on a multimodal model is provided to solve the problem that the supervised detection model in the prior art requires manual calibration, which consumes a lot of manpower, and the unsupervised model cannot collect ship data sets in certain scenarios due to lack of manual calibration, lack of effective constraints, and instability.

[0005] To achieve the above-mentioned purpose, on the one hand, the present invention provides an unsupervised ship detection method based on a multimodal model, the method comprising: S1, constructing a target ship matching model according to an original ship detection model and a matching module; S2, extracting each target in a manually calibrated training set to obtain a first picture set; inputting the first picture set into the target ship matching model to obtain a first feature sequence of each target; inputting the uncalibrated training set into the SAM segmentation model to obtain a first coordinate of each target; and extracting each target to obtain a second picture set; inputting the second picture set into the clip multimodal model to obtain a first category and a first confidence of each target; inputting the second picture set into the target ship matching model to obtain a second feature sequence of each target; according to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second picture set, obtain a second category and a second confidence of each target in the second picture set; S3, inputting the uncalibrated training set into the first ship detection model for prediction, and obtain a third category, a second coordinate and a third confidence of each target in each picture; S4, inputting the uncalibrated training set into the second ship detection model for prediction. Predict the fourth category, third coordinate and fourth confidence of each target in each picture; calculate the first loss value, second loss value and third loss value of the current target in the current picture in the uncalibrated training set according to the fourth category, third category, second category, first category, third coordinate, second coordinate, first coordinate, fourth confidence, third confidence, second confidence and first confidence; S5, extract each target according to the second coordinate of each target in each picture in the uncalibrated training set to obtain a third picture set; input the third picture set into the target ship matching model, and obtain To the fifth category and fifth confidence of each target, the third picture set is input into the clip multimodal model to obtain the sixth category and sixth confidence of each target; the fourth loss value of the current target in the current picture in the uncalibrated training set is calculated according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence, the fourth confidence, the fifth confidence, and the sixth confidence; S6, according to the sum of the first loss value, the second loss value, the third loss value, and the fourth loss value of all targets in the uncalibrated training set, the second ship detection model is reversely updated; S7, repeat S2~S6, if the sum fluctuates within the first preset range, stop training, and use the last updated second ship detection model as the target ship detection model; input the ship picture to be detected into the target ship detection model for detection to obtain the target category and position.

[0006] Optionally, S1 includes: S11, copying the manually calibrated training set and saving only one category of targets for each image in the copied manually calibrated training set to obtain an updated training set; S12, adding a matching module to the backbone network of the original ship detection model to obtain the original ship matching model, and freezing the parameters of the backbone network of the original ship detection model; S13, inputting the updated training set into the original ship detection model for prediction to obtain the category, coordinates, and confidence of each image; S14, screening the updated training set according to the predicted category, coordinates, confidence of each image in the updated training set and the manually calibrated category and coordinates, training the screened updated training set through the original ship matching model to obtain the feature vector of each image; S15, calculating the total matching loss value according to the feature vectors of all images in the screened updated training set, and reversely updating the matching module according to the total matching loss value; S16, repeating S13-S15 until the finally calculated total matching loss value fluctuates within the second preset range, and the training is stopped to obtain the target ship matching model.

[0007] Optionally, obtaining the second category and the second confidence of each target in the second picture set according to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second picture set includes: calculating similarity values ​​according to the second feature sequence of the current target in the second picture set and the first feature sequence of each target in the first picture set, respectively, to obtain the similarity value between the current target in the second picture set and each target in the first picture set; classifying the targets in the first picture set corresponding to all the similarity values ​​by category, to obtain the number of real targets in each category; filtering all the similarity values ​​according to a preset similarity threshold, to obtain all the filtered similarity values, and counting the number as the predicted number; classifying the targets in the first picture set corresponding to all the filtered similarity values ​​by category, to obtain the accurate number of targets in each category, and calculating the predicted number according to the predicted number. The number of accurate targets and the predicted number of each category are used to calculate the accuracy of the current target and each category in the second picture set; the recall rate of the current target and each category in the second picture set is calculated according to the number of accurate targets in each category and the number of real targets in each category; the accuracy rate of the current target and each category in the second picture set is filtered according to the accuracy rate threshold to obtain the filtered accuracy rate; each category corresponding to the filtered accuracy rate is used as the pre-target category; the recall rate of the current target and each pre-target category in the second picture set is sorted, and the pre-target category corresponding to the highest recall rate is selected as the target category of the current target in the second picture set; the target category is the second category; the second confidence of the current target in the second picture set is calculated according to the accuracy rate of the current target and the target category in the second picture set and the recall rate of the current target and the target category in the second picture set.

[0008] Optionally, S4 includes: if the intersection and union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is greater than or equal to a first preset intersection and union ratio threshold, and the first category, the second category, and the third category of the current target in the second picture set are consistent, then the first coordinate loss value is calculated according to the first coordinate of the current target in the second picture set, the second coordinate of the current target in the uncalibrated training set, and the third coordinate of the current target in the uncalibrated training set; the first category loss value is calculated according to the first category or the second category of the current target in the second picture set, or the third category of the current target in the uncalibrated training set, and the fourth category of the current target in the uncalibrated training set; the first weight is calculated according to the first confidence and the second confidence of the current target in the second picture set; the first loss value is calculated according to the first coordinate loss value, the first category loss value, and the first weight; if the intersection and union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is When the intersection-and-union ratio is greater than or equal to the first preset intersection-and-union ratio threshold, and the first category, the second category, and the third category of the current target in the uncalibrated training set of the current target in the second picture set are completely inconsistent, or only two of the categories are consistent, the second category loss value is calculated according to the first category, the first confidence, the second category, and the second confidence of the current target in the second picture set, the third category and the third confidence of the current target in the uncalibrated training set, and the fourth category and the fourth confidence of the current target in the uncalibrated training set; the second loss value is calculated according to the second category loss value and the first coordinate loss value; if the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is less than the first preset intersection-and-union ratio threshold, and the first confidence or the second confidence of the current target in the second picture set is greater than the first preset confidence, the third loss value is calculated according to the first category, the first confidence, the second category, the second confidence of the current target in the second picture set, and the fourth category and the fourth confidence of the current target in the uncalibrated training set.

[0009] Optionally, the fourth loss value of the current target in the current picture in the uncalibrated training set is calculated according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence level, the fourth confidence level, the fifth confidence level, and the sixth confidence level, including: if the intersection and union ratio of the second coordinate of the current target in the third picture set and the first coordinate of the current target in the second picture set is less than a second preset intersection and union ratio threshold, then the fourth loss value is calculated according to the third category, third confidence level, fifth category, fifth confidence level, sixth category, and sixth confidence level of the current target in the third picture set, and the fourth category and fourth confidence level of the current target in the uncalibrated training set.

[0010] Optionally, the first loss value is calculated according to the following formula: ; in, is the first loss value, is the first weight, is the first coordinate loss value, is the first category loss value, is the coordinate loss value of the third coordinate of the current target in the uncalibrated training set and the first coordinate of the current target in the second picture set, the coordinate loss value of the third coordinate of the current target in the uncalibrated training set and the second coordinate of the current target in the uncalibrated training set, min is the minimum coordinate loss value, is the second confidence of the current target in the second picture set, is the first confidence of the current target in the second picture set, is the accuracy of the current target and target category in the second picture set, is the recall rate of the current target and the target category in the second picture set.

[0011] Optionally, the second loss value is calculated according to the following formula: ; in, is the category loss value a of the fourth category of the current target in the uncalibrated training set and the first category of the current target in the second picture set, or the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set, or the category loss value c of the fourth category of the current target in the uncalibrated training set and the third category of the current target in the uncalibrated training set; when When is the category loss value a, is the first confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the first category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value b, is the second confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the second category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value c, is the third confidence of the current target in the uncalibrated training set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the third category of the current target in the uncalibrated training set, y is 1, otherwise, y is 0; is the fourth confidence of the current target in the uncalibrated training set, is the average of the category loss values ​​a, b, and c, which is the second category loss value. is the first coordinate loss value, is the second loss value.

[0012] Optionally, the third loss value is calculated according to the following formula: ; ; in, is the third loss value, is the category loss value a of the fourth category of the current target in the uncalibrated training set and the first category of the current target in the second picture set, or the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set. When is the category loss value a, is the first confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the first category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value b, is the second confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the second category of the current target in the second picture set, y is 1, otherwise, y is 0; is the fourth confidence of the current target in the uncalibrated training set. When the first confidence and the second confidence of the current target in the second picture set are both greater than the first preset confidence, k is 2. When only one confidence is greater than the first preset confidence, k is 1.

[0013] Optionally, the fourth loss value is calculated according to the following formula: ; in, is the fourth loss value, is the category loss value d of the fourth category of the current target in the uncalibrated training set and the third category of the current target in the third picture set, or the category loss value e of the fourth category of the current target in the uncalibrated training set and the fifth category of the current target in the third picture set, or the category loss value f of the fourth category of the current target in the uncalibrated training set and the sixth category of the current target in the third picture set, is the average value of category loss value d, category loss value e, and category loss value f. When is the category loss value d, is the third confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the third category of the current target in the third picture set, y is 1, otherwise, y is 0; When is the category loss value e, is the fifth confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the fifth category of the current target in the third picture set, y is 1, otherwise, y is 0; When is the category loss value f, is the sixth confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the sixth category of the current target in the third picture set, y is 1, otherwise, y is 0.

[0014] On the other hand, the present invention provides an unsupervised ship detection system based on a multimodal model, the system comprising: a construction unit, used to construct a target ship matching model according to an original ship detection model and a matching module; a first prediction unit, used to extract each target in a manually calibrated training set to obtain a first picture set; input the first picture set into the target ship matching model to obtain a first feature sequence of each target; input the uncalibrated training set into a SAM segmentation model to obtain a first coordinate of each target; and extract each target to obtain a second picture set; input the second picture set into a clip multimodal model to obtain a first category and a first confidence of each target; input the second picture set into the target In the ship matching model, a second feature sequence of each target is obtained; according to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second preprocessed picture set, a second category and a second confidence of each target in the second picture set are obtained; the second prediction unit is used to input the uncalibrated training set into the first ship detection model for prediction, and obtain the third category, second coordinates and third confidence of each target in each picture; the first calculation unit is used to input the uncalibrated training set into the second ship detection model for prediction, and obtain the fourth category, third coordinates and fourth confidence of each target in each picture; according to the fourth category, the third category, the second category, the first category, the third coordinates, The first loss value, the second loss value, and the third loss value of the current target in the current picture in the uncalibrated training set are calculated by the second coordinate, the first coordinate, the fourth confidence, the third confidence, the second confidence, and the first confidence; the second calculation unit is used to clip each target according to the second coordinate of each target in each picture in the uncalibrated training set to obtain a third picture set; the third picture set is input into the target ship matching model to obtain the fifth category and the fifth confidence of each target, and the third picture set is input into the clip multimodal model to obtain the sixth category and the sixth confidence of each target; according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third The fourth confidence level, the fourth confidence level, the fifth confidence level and the sixth confidence level are used to calculate the fourth loss value of the current target in the current picture in the uncalibrated training set; a reverse updating unit is used to reversely update the second ship detection model according to the sum of the first loss value, the second loss value, the third loss value and the fourth loss value of all targets in the uncalibrated training set; a detection unit is used to repeat the first prediction unit, the second prediction unit, the first calculation unit, the second calculation unit and the reverse updating unit, and if the sum value fluctuates within the first preset range, the training is stopped, and the last updated second ship detection model is used as the target ship detection model; the image of the ship to be detected is input into the target ship detection model for detection to obtain the target category and position.

[0015] Beneficial effects of the present invention: The present invention provides an unsupervised ship detection method and system based on a multimodal model, wherein the method uses a multimodal model to automatically calibrate a training set; the original ship detection model is modified to add a matching module to the original ship detection model, thereby adding feature matching supervision to the ship detection feature; and the ship detection model adds multimodal model supervision in the loss structure. The method of the present invention can improve the accuracy and efficiency of ship detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of an unsupervised ship detection method based on a multimodal model provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of an unsupervised ship detection system based on a multimodal model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Figure 1 An unsupervised ship detection method based on a multimodal model is provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S1, constructing a target ship matching model based on the original ship detection model and matching structure; Specifically, the S1 includes: S11, copying the manually calibrated training set and storing only one category of targets in each picture in the copied manually calibrated training set to obtain an updated training set; Take a picture (the current picture) in the manual calibration training set as an example: There are two targets in the current picture, and the two targets are of different categories, namely A and B. Then copy the current picture into two pictures, in which only the targets of category A are retained in one picture, and the targets of category B are blackened or the background information is filled on the targets of category B; in the other picture, only the targets of category B are retained, and the targets of category A are blackened or the background information is filled on the targets of category A.

[0019] There are two targets in the current picture, and the categories of the two targets are the same, both A, then the current picture is copied into a picture, and both targets are retained in the picture.

[0020] All images in the manually calibrated training set are processed as above to obtain an updated training set.

[0021] S12, adding a matching module to the backbone network of the original ship detection model to obtain the original ship matching model, and freezing the parameters of the backbone network of the original ship detection model; Specifically, the backbone network of the original ship matching model adopts the backbone network (ResNet50) of the original ship detection model. Compared with the original ship detection model, the original ship matching model adds a matching module. The specific steps of the matching module are: Perform self-attention operation on the feature map output by the backbone network to obtain a self-attention feature map; Add the self-attention feature map to the feature map output by the backbone network to obtain a fused feature map; Perform convolution operation on the fused feature map to obtain features with more effective information; The fused feature map after the convolution operation is subjected to global maximum pooling and full convolution operations to obtain a 1*512-dimensional feature vector.

[0022] During the training process of the original ship matching model, the backbone network was frozen and no backpropagation was performed. Only the matching module performed backpropagation.

[0023] S13, inputting the updated training set into the original ship detection model for prediction, and obtaining the category, coordinates, and confidence of each image; S14, screening the updated training set according to the predicted category, coordinates, confidence level of each image in the updated training set and the manually calibrated category and coordinates, and training the screened updated training set through the original ship matching model to obtain a feature vector of each image; Take updating a picture in the training set (the current picture) as an example: If the category of the current picture in the updated training set is consistent with the manually calibrated category, the confidence of the current picture in the updated training set is greater than the second preset confidence, and the intersection and union ratio of the coordinates of the current picture in the updated training set and the manually calibrated coordinates is greater than the third preset intersection and union ratio threshold, the current picture in the updated training set will be retained.

[0024] Specifically, if there is only one target in the current picture, and the category of the target is consistent with the manually calibrated category, the confidence of the target is greater than the second preset confidence (set to 0.1 in the present invention), and the intersection and union ratio of the target coordinates and the manually calibrated coordinates is greater than the third preset intersection and union ratio threshold (set to 0.25 in the present invention), then the current picture is retained and subsequent matching model training is performed.

[0025] If there are at least two targets in the current image, as long as one target meets the above three conditions, the current image can be retained for subsequent matching model training.

[0026] All retained images, i.e. the updated training set after filtering, are trained through the original ship matching model. Take one image (the current image) in the updated training set after filtering as an example: The current image in the filtered update training set is passed through the backbone network and the matching module in turn to obtain the feature vector of the current image; that is, the current image is subjected to a self-attention operation through the feature map output by the backbone network, and the feature map after the self-attention operation and the feature map output by the backbone network are subjected to a fusion operation, a convolution operation, a global maximum pooling operation, and a full convolution operation in turn to obtain the feature vector of the current image.

[0027] S15, calculating a total matching loss value according to the feature vectors of all images in the screened update training set, and reversely updating the matching module according to the total matching loss value; Specifically, the filtered updated training set is divided into multiple iterations, taking one iteration (the current iteration) as an example: The matching loss value of the current iteration is calculated according to the following formula: ; in, represents the matching loss value of the current iteration, N represents the number of pictures in the current iteration, i represents the i-th picture in the current iteration, j represents the j-th picture among the N pictures in the current iteration except the i-th picture itself, represents the feature vector obtained by the original ship matching model for the i-th image in the current iteration, It represents the feature vector obtained by the original ship matching model of the jth picture in the current iteration. y is 0 or 1. If the category of the i-th picture in the current iteration matches the category of the j-th picture in the current iteration, y is 1, otherwise it is 0.

[0028] The matching loss values ​​of all iterations are summed and averaged to obtain the total matching loss value.

[0029] S16, repeat S13 to S15 until the final calculated total matching loss value fluctuates within the second preset range, and then stop training to obtain a target ship matching model.

[0030] Repeat the above process to update the original ship matching model multiple times until the final calculated total matching loss value is within the second preset range ( The training is stopped when the fluctuation is within 0.1%, and the target ship matching model is obtained. Among them, only the matching module in the original ship matching model is updated.

[0031] S2, extract each target in the manual calibration training set to obtain a first picture set; input the first picture set into the target ship matching model to obtain a first feature sequence of each target; input the uncalibrated training set into the SAM segmentation model to obtain the first coordinate of each target; and extract each target to obtain a second picture set; input the second picture set into the clip multimodal model to obtain a first category and a first confidence of each target; input the second picture set into the target ship matching model to obtain a second feature sequence of each target; obtain a second category and a second confidence of each target in the second picture set according to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second picture set; Specifically, each target in the manually calibrated training set is extracted according to the coordinates of each target manually calibrated to obtain a first picture set; each picture in the first picture set has only one target.

[0032] Inputting the first picture set into the target ship matching model to obtain a first feature sequence of each target; The uncalibrated training set is input into the SAM segmentation model to obtain the feature information of each target in each picture in the uncalibrated training set; then the connected domain method is used to select each target to obtain the target frame and the first coordinate of each target in each picture in the uncalibrated training set; each target is extracted according to the target frame of each target in each picture in the uncalibrated training set to obtain the second picture set; each picture in the second picture set has only one target.

[0033] Inputting the second picture set into the clip multimodal model to obtain a first category and a first confidence level of each target; inputting the second picture set into the target ship matching model to obtain a second feature sequence of each target; According to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second picture set, a second category and a second confidence level of each target in the second picture set are obtained; including: Calculate similarity values ​​according to the second feature sequence of the current target in the second picture set and the first feature sequence of each target in the first picture set, and obtain similarity values ​​between the current target in the second picture set and each target in the first picture set; In a preferred embodiment, if there are 50 objects in the first picture set, similarity values ​​are calculated between the objects and the current object in the second picture set to obtain 50 similarity values.

[0034] Classify the targets in the first picture set corresponding to all similarity values ​​by category to obtain the number of real targets in each category; filter all similarity values ​​according to a preset similarity threshold to obtain all filtered similarity values, and count the number as the predicted number; classify the targets in the first picture set corresponding to all filtered similarity values ​​by category to obtain the number of accurate targets in each category, and calculate the accuracy of the current target in the second picture set and each category based on the number of accurate targets in each category and the predicted number; calculate the recall rate of the current target in the second picture set and each category based on the number of accurate targets in each category and the number of real targets in each category; If there are three categories of targets in the first picture set, namely, category A, category B, and category C; the number of real targets in category A is 20, the number of real targets in category B is 20, and the number of real targets in category C is 10.

[0035] The 50 similarity values ​​are screened according to a preset similarity threshold (set to 0.35 in the present invention), and 40 similarity values ​​are obtained, so the number of predictions is 40. The number of each category in the first picture set corresponding to the 40 similarity values ​​is counted, and the number of accurate targets in category A is 18, the number of accurate targets in category B is 15, and the number of accurate targets in category C is 7.

[0036] The accuracy of category A is 18 / 40, the accuracy of category B is 15 / 40, and the accuracy of category C is 7 / 40; The recall rate of class A is 18 / 20, the recall rate of class B is 15 / 20, and the recall rate of class C is 7 / 10; The accuracy of the current target and each category in the second picture set is screened according to the accuracy threshold to obtain the screened accuracy; each category corresponding to the screened accuracy is used as the pre-target category; the recall rate of the current target and each pre-target category in the second picture set is sorted, and the pre-target category corresponding to the highest recall rate is selected as the target category of the current target in the second picture set; the target category is the second category; In an optional implementation, if the accuracy threshold is 1 / 2, the accuracy greater than the accuracy threshold is screened, and the A and B categories corresponding to the screened accuracy are used as pre-target categories; The recall rate of class A is greater than the recall rate of class B, so class A is taken as the target class; therefore, the class of the current target in the second picture set is class A, and it is taken as the second class.

[0037] A second confidence of the current target in the second picture set is obtained by calculating the accuracy of the current target and the target category in the second picture set and the recall rate of the current target and the target category in the second picture set.

[0038] In an optional implementation, the second confidence of the current target in the second picture set is (18 / 40+18 / 20) / 2=0.675.

[0039] S3, inputting the uncalibrated training set into the first ship detection model for prediction, and obtaining the third category, second coordinates, and third confidence of each target in each picture; S4, inputting the uncalibrated training set into the second ship detection model for prediction, and obtaining the fourth category, third coordinate and fourth confidence of each target in each picture; calculating the first loss value, second loss value and third loss value of the current target in the current picture in the uncalibrated training set according to the fourth category, third category, second category, first category, third coordinate, second coordinate, first coordinate, fourth confidence, third confidence, second confidence and first confidence; Specifically, the first coordinate, the first category, the first confidence, the second category, the second confidence of the current target in the second picture set, and the third category, the second coordinate, and the third confidence of the current target in the uncalibrated training set are used as pseudo-labels and compared with the fourth category, the third coordinate, and the fourth confidence of the current target in the uncalibrated training set to calculate the loss value.

[0040] The S4 includes: ① If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is greater than or equal to the first preset intersection-and-union ratio threshold, and the first category, the second category of the current target in the second picture set, and the third category of the current target in the uncalibrated training set are consistent, then the first coordinate loss value is calculated according to the first coordinate of the current target in the second picture set, the second coordinate of the current target in the uncalibrated training set, and the third coordinate of the current target in the uncalibrated training set; the first category loss value is calculated according to the first category or the second category of the current target in the second picture set, or the third category of the current target in the uncalibrated training set, and the fourth category of the current target in the uncalibrated training set; the first weight is calculated according to the first confidence and the second confidence of the current target in the second picture set; the first loss value is calculated according to the first coordinate loss value, the first category loss value, and the first weight; The first loss value is calculated according to the following formula: ; in, is the first loss value, is the first weight, is the first coordinate loss value, is the first category loss value, is the coordinate loss value of the third coordinate of the current target in the uncalibrated training set and the first coordinate of the current target in the second picture set, the coordinate loss value of the third coordinate of the current target in the uncalibrated training set and the second coordinate of the current target in the uncalibrated training set, min is the minimum coordinate loss value, is the second confidence of the current target in the second picture set, is the first confidence of the current target in the second picture set, is the accuracy of the current target and target category in the second picture set, is the recall rate of the current target and the target category in the second picture set.

[0041] Because the first category of the current target in the second picture set, the second category of the current target in the second picture set, and the third category of the current target in the uncalibrated training set are consistent, only one first category loss value needs to be calculated, that is, the category loss value of the first category of the current target in the second picture set and the fourth category of the current target in the uncalibrated training set is used as the first category loss value, or the category loss value of the second category of the current target in the second picture set and the fourth category of the current target in the uncalibrated training set is used as the first category loss value, or the category loss value of the third category of the current target in the uncalibrated training set and the fourth category of the current target in the uncalibrated training set is used as the first category loss value.

[0042] ② If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is greater than or equal to the first preset intersection-and-union ratio threshold, and the first category, the second category of the current target in the second picture set, and the third category of the current target in the uncalibrated training set are completely inconsistent, or only two of the categories are consistent, then the second category loss value is calculated according to the first category, the first confidence, the second category, the second confidence of the current target in the second picture set, the third category and the third confidence of the current target in the uncalibrated training set, and the fourth category and the fourth confidence of the current target in the uncalibrated training set; the second loss value is calculated according to the second category loss value and the first coordinate loss value; The second loss value is calculated according to the following formula: ; in, is the category loss value a of the fourth category of the current target in the uncalibrated training set and the first category of the current target in the second picture set, or the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set, or the category loss value c of the fourth category of the current target in the uncalibrated training set and the third category of the current target in the uncalibrated training set; when When is the category loss value a, is the first confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the first category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value b, is the second confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the second category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value c, is the third confidence of the current target in the uncalibrated training set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the third category of the current target in the uncalibrated training set, y is 1, otherwise, y is 0; is the fourth confidence of the current target in the uncalibrated training set, is the average of the category loss values ​​a, b, and c, which is the second category loss value. is the first coordinate loss value, is the second loss value.

[0043] ③If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is less than the first preset intersection-and-union ratio threshold, and the first confidence or the second confidence of the current target in the second picture set is greater than the first preset confidence, then the third loss value is calculated according to the first category, first confidence, second category, second confidence of the current target in the second picture set, and the fourth category and fourth confidence of the current target in the uncalibrated training set.

[0044] The third loss value is calculated according to the following formula: ; in, is the third loss value, is the category loss value a of the fourth category of the current target in the uncalibrated training set and the first category of the current target in the second picture set, or the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set. When is the category loss value a, is the first confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the first category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value b, is the second confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the second category of the current target in the second picture set, y is 1, otherwise, y is 0; is the fourth confidence of the current target in the uncalibrated training set. When the first confidence and the second confidence of the current target in the second picture set are both greater than the first preset confidence, k is 2. When only one confidence is greater than the first preset confidence, k is 1.

[0045] Specifically, if the intersection and union of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is less than the first preset intersection and union threshold, and the first confidence of the current target in the second picture set is greater than the first preset confidence, but the second confidence of the current target in the second picture set is less than the first preset confidence, then the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set is 0, and the third loss value is the category loss value a / 2.

[0046] If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is less than the first preset intersection-and-union ratio threshold, and the first confidence of the current target in the second picture set is less than the first preset confidence, but the second confidence of the current target in the second picture set is greater than the first preset confidence, then the category loss value a of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set is 0, and the third loss value is the category loss value b / 2.

[0047] If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is less than the first preset intersection-and-union ratio threshold, and the first confidence of the current target in the second picture set is greater than the first preset confidence, and the second confidence of the current target in the second picture set is also greater than the first preset confidence, then the third loss value is (category loss value a + category loss value b) / 2.

[0048] S5. Clip each target according to the second coordinate of each target in each picture in the uncalibrated training set to obtain a third picture set; input the third picture set into the target ship matching model to obtain the fifth category and the fifth confidence of each target, input the third picture set into the clip multimodal model to obtain the sixth category and the sixth confidence of each target; calculate the fourth loss value of the current target in the current picture in the uncalibrated training set according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence, the fourth confidence, the fifth confidence, and the sixth confidence; Specifically, the third category, the third confidence, the fifth category, the fifth confidence, the sixth category, and the sixth confidence of the current target in the third picture set are used as pseudo labels and compared with the fourth category and the fourth confidence of the current target in the uncalibrated training set to calculate the loss value.

[0049] The S5 includes: ① If the intersection-and-union ratio of the second coordinate of the current target in the third picture set and the first coordinate of the current target in the second picture set is greater than or equal to the second preset intersection-and-union ratio threshold, and the third category, fifth category, and sixth category of the current target in the third picture set are consistent, no processing is performed.

[0050] ② If the intersection-and-union ratio of the second coordinate of the current target in the third picture set and the first coordinate of the current target in the second picture set is greater than or equal to the second preset intersection-and-union ratio threshold, and the third category, fifth category, and sixth category of the current target in the third picture set are completely inconsistent, or only two of the categories are consistent, no processing is performed.

[0051] ③If the intersection-and-union ratio of the second coordinate of the current target in the third picture set and the first coordinate of the current target in the second picture set is less than the second preset intersection-and-union ratio threshold, then the fourth loss value is calculated according to the third category, third confidence, fifth category, fifth confidence, sixth category, and sixth confidence of the current target in the third picture set and the fourth category and fourth confidence of the current target in the uncalibrated training set.

[0052] The fourth loss value is calculated according to the following formula: ; in, is the fourth loss value, is the category loss value d of the fourth category of the current target in the uncalibrated training set and the third category of the current target in the third picture set, or the category loss value e of the fourth category of the current target in the uncalibrated training set and the fifth category of the current target in the third picture set, or the category loss value f of the fourth category of the current target in the uncalibrated training set and the sixth category of the current target in the third picture set, is the average value of category loss value d, category loss value e, and category loss value f. When is the category loss value d, is the third confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the third category of the current target in the third picture set, y is 1, otherwise, y is 0; When is the category loss value e, is the fifth confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the fifth category of the current target in the third picture set, y is 1, otherwise, y is 0; When is the category loss value f, is the sixth confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the sixth category of the current target in the third picture set, y is 1, otherwise, y is 0.

[0053] S6, reversely updating the second ship detection model according to the sum of the first loss value, the second loss value, the third loss value, and the fourth loss value of all targets in the uncalibrated training set; The total loss value of the current target is obtained by summing up the first loss value, the second loss value, the third loss value, and the fourth loss value of the current target in the current image in the uncalibrated training set; The sum of the total loss values ​​of all targets in the uncalibrated training set is used to reversely update the second ship detection model.

[0054] S7, repeat S2-S6, if the sum is within the first preset range ( ) stops training when the fluctuation occurs, and uses the last updated second ship detection model as the target ship detection model; inputs the image of the ship to be detected into the target ship detection model for detection to obtain the target category and position.

[0055] Figure 2 An unsupervised ship detection system based on a multimodal model is provided in an embodiment of the present invention. Figure 2 As shown, the system includes: A construction unit 201 is used to construct a target ship matching model according to the original ship detection model and the matching module; The first prediction unit 202 is used to extract each target in the manual calibration training set to obtain a first picture set; input the first picture set into the target ship matching model to obtain a first feature sequence of each target; input the uncalibrated training set into the SAM segmentation model to obtain a first coordinate of each target; and extract each target to obtain a second picture set; input the second picture set into the clip multimodal model to obtain a first category and a first confidence of each target; input the second picture set into the target ship matching model to obtain a second feature sequence of each target; obtain a second category and a second confidence of each target in the second picture set according to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second preprocessed picture plus clip set; The second prediction unit 203 is used to input the uncalibrated training set into the first ship detection model for prediction, and obtain the third category, second coordinates and third confidence of each target in each picture; The first calculation unit 204 is used to input the uncalibrated training set into the second ship detection model for prediction, and obtain the fourth category, third coordinates and fourth confidence of each target in each picture; calculate the first loss value, second loss value and third loss value of the current target in the current picture in the uncalibrated training set according to the fourth category, third category, second category, first category, third coordinates, second coordinates, first coordinates, fourth confidence, third confidence, second confidence and first confidence; The second calculation unit 205 is used to clip each target according to the second coordinate of each target in each picture in the uncalibrated training set to obtain a third picture set; input the third picture set into the target ship matching model to obtain the fifth category and the fifth confidence of each target, input the third picture set into the clip multimodal model to obtain the sixth category and the sixth confidence of each target; calculate the fourth loss value of the current target in the current picture in the uncalibrated training set according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence, the fourth confidence, the fifth confidence, and the sixth confidence; A reverse updating unit 206, configured to reversely update the second ship detection model according to the sum of the first loss value, the second loss value, the third loss value, and the fourth loss value of all targets in the uncalibrated training set; The detection unit 207 is used to repeat the first prediction unit, the second prediction unit, the first calculation unit, the second calculation unit, and the reverse update unit. If the sum value fluctuates within the first preset range, the training is stopped, and the last updated second ship detection model is used as the target ship detection model; the image of the ship to be detected is input into the target ship detection model for detection to obtain the target category and position.

[0056] The system of the present invention corresponds to the above-mentioned method, and the specific embodiments of the system will not be repeated here.

[0057] Beneficial effects of the present invention: The method of the present invention adopts a multimodal model to automatically calibrate the training set, without the need for manpower-consuming manual calibration, thereby greatly improving efficiency; the original ship detection model is modified to add a matching structure to the original ship detection model, thereby adding feature matching supervision to the ship detection features, thereby improving the accuracy of ship detection; the ship detection model adds multimodal model supervision in the loss structure, thereby further improving the accuracy of ship detection.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised ship detection method based on a multimodal model, characterized in that: include: S1, constructing a target ship matching model based on the original ship detection model and the matching module; S2, extracting each target in the manual calibration training set to obtain a first picture set; inputting the first picture set into the target ship matching model to obtain a first feature sequence of each target; Input the uncalibrated training set into the SAM segmentation model to obtain the first coordinate of each target; And extract each target to obtain a second picture set; Input the second picture set into the clip multimodal model to obtain the first category and the first confidence of each target; Inputting the second picture set into the target ship matching model to obtain a second feature sequence of each target; According to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second picture set, a second category and a second confidence level of each target in the second picture set are obtained; S3, inputting the uncalibrated training set into the first ship detection model for prediction, and obtaining the third category, second coordinates, and third confidence of each target in each picture; S4, inputting the uncalibrated training set into the second ship detection model for prediction, and obtaining the fourth category, third coordinate and fourth confidence of each target in each picture; calculating the first loss value, second loss value and third loss value of the current target in the current picture in the uncalibrated training set according to the fourth category, third category, second category, first category, third coordinate, second coordinate, first coordinate, fourth confidence, third confidence, second confidence and first confidence; S5, clipping each target according to the second coordinate of each target in each picture in the uncalibrated training set to obtain a third picture set; inputting the third picture set into the target ship matching model to obtain a fifth category and a fifth confidence level of each target; inputting the third picture set into the clip multimodal model to obtain a sixth category and a sixth confidence level of each target; A fourth loss value of the current target in the current picture in the uncalibrated training set is calculated according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence level, the fourth confidence level, the fifth confidence level, and the sixth confidence level; S6, reversely updating the second ship detection model according to the sum of the first loss value, the second loss value, the third loss value, and the fourth loss value of all targets in the uncalibrated training set; S7, repeat S2-S6, if the sum value fluctuates within the first preset range, stop training, use the last updated second ship detection model as the target ship detection model; input the image of the ship to be detected into the target ship detection model for detection, and obtain the target category and position.

2. The method according to claim 1, characterized in that The S1 includes: S11, copying the manually calibrated training set and storing only one category of targets in each picture in the copied manually calibrated training set to obtain an updated training set; S12, adding a matching module to the backbone network of the original ship detection model to obtain the original ship matching model, and freezing the parameters of the backbone network of the original ship detection model; S13, inputting the updated training set into the original ship detection model for prediction, and obtaining the category, coordinates, and confidence of each image; S14, screening the updated training set according to the predicted category, coordinates, confidence level of each image in the updated training set and the manually calibrated category and coordinates, and training the screened updated training set through the original ship matching model to obtain a feature vector of each image; S15, calculating a total matching loss value according to the feature vectors of all images in the screened update training set, and reversely updating the matching module according to the total matching loss value; S16, repeat S13 to S15 until the final calculated total matching loss value fluctuates within the second preset range and the training is stopped to obtain a target ship matching model.

3. The method according to claim 1, characterized in that The step of obtaining a second category and a second confidence level of each target in the second picture set according to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second picture set comprises: Calculate similarity values ​​according to the second feature sequence of the current target in the second picture set and the first feature sequence of each target in the first picture set, and obtain similarity values ​​between the current target in the second picture set and each target in the first picture set; Classify the targets in the first picture set corresponding to all similarity values ​​by category to obtain the number of real targets in each category; filter all similarity values ​​according to a preset similarity threshold to obtain all filtered similarity values, and count the number as the predicted number; classify the targets in the first picture set corresponding to all filtered similarity values ​​by category to obtain the number of accurate targets in each category, and calculate the accuracy of the current target in the second picture set and each category based on the number of accurate targets in each category and the predicted number; calculate the recall rate of the current target in the second picture set and each category based on the number of accurate targets in each category and the number of real targets in each category; The accuracy of the current target and each category in the second picture set is screened according to the accuracy threshold to obtain the screened accuracy; each category corresponding to the screened accuracy is used as the pre-target category; the recall rate of the current target and each pre-target category in the second picture set is sorted, and the pre-target category corresponding to the highest recall rate is selected as the target category of the current target in the second picture set; the target category is the second category; A second confidence of the current target in the second picture set is obtained by calculating the accuracy of the current target and the target category in the second picture set and the recall rate of the current target and the target category in the second picture set.

4. The method according to claim 1, characterized in that: The S4 includes: If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is greater than or equal to the first preset intersection-and-union ratio threshold, and the first category, the second category of the current target in the second picture set, and the third category of the current target in the uncalibrated training set are consistent, then the first coordinate loss value is calculated according to the first coordinate of the current target in the second picture set, the second coordinate of the current target in the uncalibrated training set, and the third coordinate of the current target in the uncalibrated training set; the first category loss value is calculated according to the first category or the second category of the current target in the second picture set, or the third category of the current target in the uncalibrated training set, and the fourth category of the current target in the uncalibrated training set; the first weight is calculated according to the first confidence and the second confidence of the current target in the second picture set; the first loss value is calculated according to the first coordinate loss value, the first category loss value, and the first weight; If the intersection-and-union ratio of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is greater than or equal to the first preset intersection-and-union ratio threshold, and the first category, the second category of the current target in the second picture set, and the third category of the current target in the uncalibrated training set are completely inconsistent, or only two of the categories are consistent, then the second category loss value is calculated according to the first category, the first confidence, the second category, the second confidence of the current target in the second picture set, the third category and the third confidence of the current target in the uncalibrated training set, and the fourth category and the fourth confidence of the current target in the uncalibrated training set; the second loss value is calculated according to the second category loss value and the first coordinate loss value; If the intersection and union of the first coordinate of the current target in the second picture set and the second coordinate of the current target in the uncalibrated training set is less than the first preset intersection and union threshold, and the first confidence or the second confidence of the current target in the second picture set is greater than the first preset confidence, then the third loss value is calculated based on the first category, first confidence, second category, and second confidence of the current target in the second picture set, and the fourth category and fourth confidence of the current target in the uncalibrated training set.

5. The method according to claim 1, characterized in that The fourth loss value of the current target in the current picture in the uncalibrated training set is calculated according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence level, the fourth confidence level, the fifth confidence level, and the sixth confidence level, including: If the intersection-and-union ratio of the second coordinate of the current target in the third picture set and the first coordinate of the current target in the second picture set is less than the second preset intersection-and-union ratio threshold, the fourth loss value is calculated according to the third category, third confidence level, fifth category, fifth confidence level, sixth category, and sixth confidence level of the current target in the third picture set and the fourth category and fourth confidence level of the current target in the uncalibrated training set.

6. The method according to claim 4, characterized in that The first loss value is calculated according to the following formula: ； in, is the first loss value, is the first weight, is the first coordinate loss value, is the first category loss value, is the coordinate loss value of the third coordinate of the current target in the uncalibrated training set and the first coordinate of the current target in the second picture set, the coordinate loss value of the third coordinate of the current target in the uncalibrated training set and the second coordinate of the current target in the uncalibrated training set, min is the minimum coordinate loss value, is the second confidence of the current target in the second picture set, is the first confidence of the current target in the second picture set, is the accuracy of the current target and target category in the second picture set, is the recall rate of the current target and the target category in the second picture set.

7. The method according to claim 4, characterized in that The second loss value is calculated according to the following formula: ； in, is the category loss value a of the fourth category of the current target in the uncalibrated training set and the first category of the current target in the second picture set, or the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set, or the category loss value c of the fourth category of the current target in the uncalibrated training set and the third category of the current target in the uncalibrated training set; when When is the category loss value a, is the first confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the first category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value b, is the second confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the second category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value c, is the third confidence of the current target in the uncalibrated training set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the third category of the current target in the uncalibrated training set, y is 1, otherwise, y is 0; is the fourth confidence of the current target in the uncalibrated training set, is the average of the category loss values ​​a, b, and c, which is the second category loss value. is the first coordinate loss value, is the second loss value.

8. The method according to claim 4, characterized in that The third loss value is calculated according to the following formula: ； in, is the third loss value, is the category loss value a of the fourth category of the current target in the uncalibrated training set and the first category of the current target in the second picture set, or the category loss value b of the fourth category of the current target in the uncalibrated training set and the second category of the current target in the second picture set. When is the category loss value a, is the first confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the first category of the current target in the second picture set, y is 1, otherwise, y is 0; When is the category loss value b, is the second confidence of the current target in the second picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the second category of the current target in the second picture set, y is 1, otherwise, y is 0; is the fourth confidence of the current target in the uncalibrated training set. When the first confidence and the second confidence of the current target in the second picture set are both greater than the first preset confidence, k is 2. When only one confidence is greater than the first preset confidence, k is 1.

9. The method according to claim 5, characterized in that The fourth loss value is calculated according to the following formula: ； in, is the fourth loss value, is the category loss value d of the fourth category of the current target in the uncalibrated training set and the third category of the current target in the third picture set, or the category loss value e of the fourth category of the current target in the uncalibrated training set and the fifth category of the current target in the third picture set, or the category loss value f of the fourth category of the current target in the uncalibrated training set and the sixth category of the current target in the third picture set, is the average value of category loss value d, category loss value e, and category loss value f. When is the category loss value d, is the third confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the third category of the current target in the third picture set, y is 1, otherwise, y is 0; When is the category loss value e, is the fifth confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the fifth category of the current target in the third picture set, y is 1, otherwise, y is 0; When is the category loss value f, is the sixth confidence of the current target in the third picture set, y is 0 or 1. If the fourth category of the current target in the uncalibrated training set matches the sixth category of the current target in the third picture set, y is 1, otherwise, y is 0.

10. An unsupervised ship detection system based on a multimodal model, characterized in that: include: A construction unit, used for constructing a target ship matching model according to the original ship detection model and the matching module; The first prediction unit is used to extract each target in the manual calibration training set to obtain a first picture set; the first picture set is input into the target ship matching model to obtain a first feature sequence of each target; Input the uncalibrated training set into the SAM segmentation model to obtain the first coordinate of each target; And extract each target to obtain a second picture set; Input the second picture set into the clip multimodal model to obtain the first category and the first confidence of each target; Inputting the second picture set into the target ship matching model to obtain a second feature sequence of each target; According to the first feature sequence of each target in the first picture set and the second feature sequence of each target in the second preprocessed picture set, a second category and a second confidence of each target in the second picture set are obtained; a second prediction unit, used for inputting the uncalibrated training set into the first ship detection model for prediction, and obtaining a third category, a second coordinate, and a third confidence level of each target in each picture; The first calculation unit is used to input the uncalibrated training set into the second ship detection model for prediction, and obtain the fourth category, third coordinate and fourth confidence of each target in each picture; and calculate the first loss value, second loss value and third loss value of the current target in the current picture in the uncalibrated training set according to the fourth category, third category, second category, first category, third coordinate, second coordinate, first coordinate, fourth confidence, third confidence, second confidence and first confidence; The second calculation unit is used to clip each target according to the second coordinate of each target in each picture in the uncalibrated training set to obtain a third picture set; input the third picture set into the target ship matching model to obtain a fifth category and a fifth confidence of each target; input the third picture set into the clip multimodal model to obtain a sixth category and a sixth confidence of each target; A fourth loss value of the current target in the current picture in the uncalibrated training set is calculated according to the second coordinate, the first coordinate, the third category, the fourth category, the fifth category, the sixth category, the third confidence level, the fourth confidence level, the fifth confidence level, and the sixth confidence level; A reverse updating unit, configured to reversely update the second ship detection model according to the sum of the first loss value, the second loss value, the third loss value, and the fourth loss value of all targets in the uncalibrated training set; The detection unit is used to repeat the first prediction unit, the second prediction unit, the first calculation unit, the second calculation unit, and the reverse update unit. If the sum value fluctuates within a first preset range, the training is stopped, and the last updated second ship detection model is used as the target ship detection model; the image of the ship to be detected is input into the target ship detection model for detection to obtain the target category and position.

Images