A method and system for detecting open-set ship targets based on text information

Abstract

The present invention discloses a method and system for detecting open-set targets of ships based on text information. The method comprises: fine-tuning the initial text module of the first open-set detection model of ships to obtain the final text module; assigning the weight of the final text module to the initial text module of the second open-set detection model of ships, fine-tuning the initial visual module of the second open-set detection model of ships to obtain the final visual module; modifying the features of the final visual module to obtain a modified visual module and the corresponding final second open-set detection model of ships; inputting the image of the ship to be detected into the final second open-set detection model of ships for detection to obtain the coordinates, category and description of the ship. The method strengthens the feature expression of the ship target by combining visual information and text description information; by adding effective text extraction modules and effective visual extraction modules, the professional ability of ship detection is maintained on the basis of maintaining the generalization ability of the original model.

Description

Technical Field

[0001] The present invention relates to the technical field of ship detection, and in particular to a method and system for ship open set target detection based on text information. Background Art

[0002] With the rapid development of water transport technology, ships are an important tool for water transportation. In order to facilitate various businesses, there are more and more types of ships. The closed set detection algorithm cannot detect new types of ships, resulting in a large number of missed detections, which will pose a safety hazard to water transportation. The current ship detection algorithm has the following problems:

[0003] (1) Closed-set target detection cannot effectively detect new types of ships, resulting in a large number of missed detections and difficulty in collecting materials for new or uncommon types of ships, making it difficult to collect large data sets;

[0004] (2) The existing open-collection inspection lacks professionalism and can only inspect conventional ships, but will miss a large number of new or unconventional ships;

[0005] (3) Retraining the open-set target ship detection algorithm is time-consuming and requires high hardware equipment.

[0006] Currently, no effective solution has been proposed to the above problems. Summary of the invention

[0007] The embodiment of the present invention provides a method and system for open set target detection of ships based on text information to solve the problems in the prior art that closed set detection and open set detection can only detect conventional ships, and a large number of new or unconventional ships will be missed; and retraining the open set target ship detection algorithm is time-consuming.

[0008] To achieve the above-mentioned purpose, on the one hand, the present invention provides a method for detecting open-set targets of ships based on text information, wherein the method comprises: S1, fine-tuning the initial text module of the first ship open-set detection model, comprising: adding a text extraction effective module to the initial text module to obtain an updated text module; and freezing the parameters of the initial text module; constructing a text training set, comprising: text containing ship description information and text not containing ship description information, and setting a label for each text; using the initial text module to extract and encode features of the labels to obtain a label feature sequence; and using the frozen updated text module to extract and encode the text training set. Feature extraction and encoding obtain the training set feature sequence; according to the label feature sequence and the training set feature sequence, calculate the cosine similarity loss, and optimize the text extraction effective module according to the calculated loss value; repeat the above process until the final calculated loss value fluctuates within the first preset range, stop training, and obtain the final text module; S2, fine-tune the initial visual module of the second ship open set detection model, including: obtaining a picture training set, labeling the target as a ship as a ship, manually calibrating the coordinates of each target, and inputting them into the original ship open set detection model for detection to obtain a description of each target; assigning the weight of the final text module to the first The first step is to modify the features in the initial text module of the ship open set detection model and obtain the modified text module; add the visual extraction effective module to the initial visual module to obtain the updated visual module; freeze the parameters of the initial visual module and the modified text module; use the modified text module to extract and encode the features of each target description to obtain a text feature sequence; use the frozen updated visual module to extract and encode the features of the image training set to obtain a visual feature sequence; fuse the text feature sequence and the visual feature sequence and input them into the detection head for detection to obtain the predicted target coordinates, categories, description feature sequences and probabilities; according to the manually calibrated target coordinates, the target description feature sequence and probability are obtained. The method comprises the following steps: step S1, step S2, step S3, step S4, step S5, step S6, step S7, step S8, step S9, step S10, step S11, step S12, step S13, step S14, step S15, step S16, step S17, step S18, step S19, step S20, step S21, step S22, step S23, step S24, step S25, step S26, step S27, step S28, step S29, step S30, step S31, step S32, step S33, step S34, step S35, step S36, step S37, step S38, step S39, step S40, step S41, step S42, step S43, step S44, step S45, step S46, step S47, step S48, step S49, step S50, step S51, step S52, step S53, step S54, step S55, step S56, step S57, step S58, step S59, step S60, step S61, step S62

[0009] Optionally, before using the frozen updated text module to perform feature extraction and encoding on the text training set to obtain the training set feature sequence, the method includes: dividing the text training set into multiple iterations, randomly extracting a fixed number of texts from the text training set in each iteration, and each extracted text includes both text containing ship description information and text not containing ship description information; all texts in each iteration are input as the current iteration text training set into the frozen updated text module for feature extraction and encoding to obtain each text feature sequence.

[0010] Optionally, the final text module is obtained by the following method: according to all label feature sequences of the current iteration text training set and the current text feature sequence of the current iteration text training set, the cosine similarity value between the current text and each label is calculated; according to the cosine similarity value between the current text and each label, and the label of the current text, the cosine similarity loss value of the current text is calculated; the cosine similarity loss value of each text in the current iteration text training set is summed and averaged to obtain the current iteration cosine similarity loss value; according to the current iteration cosine similarity loss value, the text extraction effective module is optimized; the above iterative process is repeated until the text training set is trained and multiple rounds of training are performed, and the training is stopped when the finally calculated current iteration cosine similarity loss value fluctuates within a first preset range, thereby obtaining the final text module of the first ship open set detection model.

[0011] Optionally, the current iteration cosine similarity loss value is calculated according to the following formula:

[0012]

[0013] in, is the cosine similarity value between the current j-th text and the current i-th label, is the current i-th label feature sequence, is the current j-th text feature sequence, is the cosine similarity loss value of the current iteration, M is the number of labels in the current iteration text training set, i is the i-th label in the current iteration text training set, P is the number of texts in the current iteration text training set, j is the j-th text in the current iteration text training set, and y indicates whether the label of the current text corresponds to the current label. If so, y is 1, otherwise, y is 0.

[0014] Optionally, the visual loss is calculated according to the manually calibrated target coordinates, predicted target coordinates, text feature sequence, predicted target description feature sequence, predicted target category, predicted target probability, and target label, and the visual extraction effective module is optimized according to the calculated loss value, including: performing intersection and union comparison on the manually calibrated target coordinates and the predicted target coordinates to obtain an intersection and union ratio. If the intersection and union ratio is greater than a preset intersection and union ratio: generating a coordinate loss value according to the manually calibrated target coordinates and the predicted target coordinates; generating a first feature sequence loss value according to the text feature sequence and the predicted target description feature sequence; and optimizing the visual extraction effective module according to the predicted target category, the target label, and the predicted target location. confidence, generate a first category loss value; generate a first visual loss value by summing up the coordinate loss value, the first feature sequence loss value, and the first category loss value; if the intersection-and-union ratio is less than or equal to the preset intersection-and-union ratio: generate a second feature sequence loss value according to the text feature sequence and the predicted target description feature sequence; generate a second category loss value according to the predicted target category, the target label and the predicted target confidence; generate a second visual loss value by summing up the second feature sequence loss value and the second category loss value; calculate the total visual loss value according to the first visual loss value and the second visual loss value, and optimize the visual extraction effective module according to the calculated total visual loss value.

[0015] Optionally, the first feature sequence loss value is calculated according to the following formula:

[0016]

[0017] in, is the similarity between the description of the current target and the description of the current predicted target, is the current text feature sequence, Describe the feature sequence for the current prediction target, is the first feature sequence loss value, K is the number of pictures in the picture training set, k is the kth picture in the picture training set, N is the number of targets in the current picture, n is the nth target in the current picture, D is the type of description of all targets corresponding to the current picture, d is the type of description of the dth target, y indicates whether the type of the current target in the current picture corresponds to the type of the description of the current target, if yes, y is 1, if no, y is 0.

[0018] Optionally, when the predicted target category is a ship, the category loss value of the current predicted target is calculated according to the following formula:

[0019]

[0020] in, is the category loss value of the current prediction target, Whether the current predicted target category corresponds to the current target label. If so, is 1, if there is no corresponding is 0, The probability of predicting the current prediction target as a ship;

[0021] When the predicted target category is not a ship, but the label is a ship, the category loss value of the current predicted target is calculated according to the following formula:

[0022]

[0023] in, is the category loss value of the current prediction target, The probability of predicting other categories for the current prediction target, The probability of predicting the current prediction target as a ship;

[0024] The loss value of the first category is obtained by summing up and averaging the category loss values ​​of all predicted target categories as ships in the image training set and the category loss values ​​of all predicted target categories as ships in the image training set but whose labels are not ships.

[0025] Optionally, the second feature sequence loss value is calculated according to the following formula:

[0026]

[0027] in, is the similarity between the description of the current target and the description of the current predicted target, is the current text feature sequence, Describe the feature sequence for the current prediction target, is the second feature sequence loss value, K is the number of pictures in the picture training set, k is the kth picture in the picture training set, N is the number of targets in the current picture, n is the nth target in the current picture, D is the type of description of all targets corresponding to the current picture, and d is the type of description of the dth target.

[0028] Optionally, the second category loss calculation is performed only when the predicted target category is a ship, and the second category loss value is calculated according to the following formula:

[0029]

[0030] in, is the second category loss value, Whether the current predicted target category corresponds to the current target label. If so, is 1, if there is no corresponding is 0, is the probability that the current prediction target is predicted to be a ship; F is the number of all prediction targets whose categories are ships in the image training set, and f is the fth prediction target whose category is ships in the image training set.

[0031] On the other hand, the present invention provides a ship open set target detection system based on text information, the system comprising: a first fine-tuning unit, used to fine-tune the initial text module of the first ship open set detection model, including: a first adding subunit, used to add a text extraction valid module to the initial text module to obtain an updated text module; and freeze the parameters of the initial text module; a construction subunit, used to construct a text training set, including: text containing ship description information, and text not containing ship description information, and setting a label for each text; a first encoding subunit, used to use the initial text module to perform feature extraction and encoding on the label to obtain a label feature sequence; and use the frozen updated text module to perform feature extraction and encoding on the text training set The encoding obtains the training set feature sequence; the first calculation subunit is used to calculate the cosine similarity loss according to the label feature sequence and the training set feature sequence, and optimize the text extraction effective module according to the calculated loss value; the first repetition subunit is used to repeat the above process until the loss value calculated at last fluctuates within the first preset range, and the training is stopped to obtain the final text module; the second fine-tuning unit is used to fine-tune the initial visual module of the second ship open set detection model, including: a data preprocessing unit is used to obtain the image training set, label the target as a ship as a ship, manually calibrate the coordinates of each target, and input them into the original ship open set detection model for detection to obtain the description of each target; the second increase subunit is used to add the most The weight of the final text module is assigned to the initial text module of the second ship open set detection model and the features are modified to obtain a modified text module; a visual extraction effective module is added to the initial visual module to obtain an updated visual module; the parameters of the initial visual module and the modified text module are frozen; the second encoding subunit is used to use the modified text module to extract and encode the description of each target to obtain a text feature sequence; the frozen updated visual module is used to extract and encode the features of the image training set to obtain a visual feature sequence; the fusion subunit is used to fuse the text feature sequence and the visual feature sequence and input them into the detection head for detection to obtain the predicted target coordinates, categories, description feature sequences and probabilities; the second calculation subunit, It is used to calculate the visual loss according to the manually calibrated target coordinates, predicted target coordinates, text feature sequence, predicted target description feature sequence, predicted target category, predicted target probability, and target label, and optimize the visual extraction effective module according to the calculated loss value; the second repeating subunit is used to repeat the above process until the last calculated loss value fluctuates within the second preset range and the training is stopped to obtain the final visual module; the modification unit is used to modify the features of the final visual module to obtain the modified visual module and the corresponding final second ship open set detection model; the detection unit is used to input the ship image to be detected into the final second ship open set detection model for detection to obtain the ship coordinates, ship category and ship description.

[0032] Beneficial effects of the present invention:

[0033] (1) Enhanced feature extraction capability for ship target detection: By combining visual information and text description information, the feature expression of ship targets is enhanced;

[0034] (2) Improving the expert capabilities of the ship target detection model: By adding an effective text extraction module and increasing the cosine similarity loss, the present invention enables the model to not only have expert knowledge of ship detection, but also effectively extract other text feature information in addition to the ship description;

[0035] (3) Optimize the visual model and reduce the demand for training data: Add a visual extraction effective module to the visual module and only optimize the visual extraction effective module, which reduces the demand for training data, shortens the training time, and reduces the requirements for hardware equipment;

[0036] (4) Enhance the collaboration between the visual and text modules: The visual module training adds category loss and feature sequence loss, so that the model can extract more common features for ship features in the visual and text modules;

[0037] (5) Improving model training efficiency and performance: Through a phased training strategy, the present invention further improves the feature extraction capabilities of text and visual models. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a method for detecting open-set targets of ships based on text information provided by an embodiment of the present invention;

[0039] Figure 2 It is a structural schematic diagram of a text information based ship open set target detection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] With the continuous increase in the types of ships, the existing closed-set target detection and open-set target detection methods are difficult to meet the detection needs of new types of ships. This embodiment proposes a new open-set target detection method by combining text information and visual information, which can effectively solve the problem of missed detection of new types of ships.

[0042] Figure 1 is a flow chart of a method for detecting open-set targets of ships based on text information provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0043] S1. Fine-tune the initial text module of the first ship open set detection model. In a preferred embodiment of the present invention, the first ship open set detection model is GroundingDINO, and its initial text module, i.e., the initial text feature extraction module, is fine-tuned;

[0044] The specific fine-tuning process includes:

[0045] S11, adding a text extraction effective module to the initial text module to obtain an updated text module; and freezing the parameters of the initial text module;

[0046] An additional text extraction effective module (such as LoRA module) is added to the initial text module. The addition of the LoRA module enables the model to better adapt to new ship target detection tasks. The LoRA module enhances the generalization ability of the model when dealing with new ship types (especially unseen ship categories) by reducing the redundancy of model parameters.

[0047] The parameters of the initial text module are frozen, and only the parameters of the newly added text extraction module are optimized to ensure that the feature extraction capability of the ship text description is enhanced while maintaining the original generalization capability.

[0048] S12, constructing a text training set, including: text containing ship description information and text not containing ship description information, and setting a label for each text;

[0049] The text training set is divided into two categories: one is the text describing the ship: for example, "a white cargo ship", the label is set to "ship", "this is a blue cruise ship", the label is set to "ship"; the other is the text not describing the ship: generated by the existing text model (this patent uses Baidu Wenxin Yiyan to generate), and obtained through manual correction. For example, "a blue car", the label is set to "blue car", "red apple", the label is set to "red apple".

[0050] In a specific embodiment, the present invention ① collects a large amount of public data text, namely, words, phrases and sentences describing ships, and ② introduces data that does not contain ship description information in 0.01% of the original text data set; by introducing text that does not contain ship description information, the model's distinguishing ability can be improved and the model can be prevented from overfitting.

[0051] S13, using the initial text module to extract and encode the label to obtain a label feature sequence; using the frozen updated text module to extract and encode the text training set to obtain a training set feature sequence;

[0052] Specifically, before using the frozen updated text module to extract and encode the features of the text training set to obtain the training set feature sequence, the following steps are included:

[0053] The text training set is divided into multiple iterations. A fixed number of texts are randomly extracted from the text training set in each iteration. Each extracted text includes both texts containing ship description information and texts without ship description information. All texts in each iteration are input as the current iteration text training set into the frozen update text module for feature extraction and encoding to obtain each text feature sequence.

[0054] For example, there are 100 text training sets, which are divided into 10 iterations. There are 10 texts in one iteration. In the first iteration, there are 5 texts describing ship information and 5 different texts describing other information (for example, cars, people, cats, dogs, and stones); in the second iteration, there are 5 texts describing ship information and 5 texts describing other information (for example, two describing cars, two describing people, and one describing cats). Then, there are 10 text feature sequences and 6 label sequences in the first iteration, and there are 10 text feature sequences and 4 label feature sequences in the second iteration.

[0055] Each iteration contains text with and without ship description information, allowing the model to learn from diverse data instead of just remembering fixed features of the ship target, thus avoiding overfitting.

[0056] S14, calculating the cosine similarity loss according to the label feature sequence and the training set feature sequence, and optimizing the text extraction effective module according to the calculated loss value;

[0057] Specifically, according to all label feature sequences of the current iteration text training set and the current text feature sequence of the current iteration text training set, the cosine similarity value between the current text and each label is calculated;

[0058] The cosine similarity loss value of the current text is calculated based on the cosine similarity value between the current text and each label, as well as the label of the current text;

[0059] The cosine similarity loss values ​​of each text in the current iteration text training set are summed and averaged to obtain the cosine similarity loss value of the current iteration; the effective module for text extraction is optimized according to the cosine similarity loss value of the current iteration;

[0060] The current iteration cosine similarity loss value is calculated according to the following formula:

[0061]

[0062] in, is the cosine similarity value between the current j-th text and the current i-th label, is the current i-th label feature sequence, is the current j-th text feature sequence, is the cosine similarity loss value of the current iteration, M is the number of labels in the current iteration text training set, i is the i-th label in the current iteration text training set, P is the number of texts in the current iteration text training set, j is the j-th text in the current iteration text training set, and y indicates whether the label of the current text corresponds to the current label. If so, y is 1, otherwise, y is 0.

[0063] Take the first iteration as an example:

[0064] In the first iteration, there are 5 texts describing ship information and 5 different texts describing other information (for example, car, person, cat, dog, stone). In the first iteration, there are 10 texts and 6 labels. Correspondingly, 10 text feature sequences and 6 label feature sequences are generated.

[0065] Calculate the cosine similarity value between the first text (for example, a white ship) and the first label (ship), that is, multiply the first text feature sequence by the first label feature sequence to obtain the first cosine similarity value, and then calculate the cosine similarity value between the first text and the second label (car) to obtain the second cosine similarity value; then calculate the cosine similarity value between the first text and the third label (person) to obtain the third cosine similarity value; then calculate the cosine similarity value between the first text and the fourth label (cat) to obtain the fourth cosine similarity value; then calculate the cosine similarity value between the first text and the fifth label (dog) to obtain the fifth cosine similarity value; then calculate the cosine similarity value between the first text and the sixth label (stone) to obtain the sixth cosine similarity value;

[0066] Since the label of the first text corresponds to the ship, y=1, and the cosine similarity loss value between the first text and the first label is ,in is the first cosine similarity value; since the label of the first text does not correspond to the car, y=0, then the cosine similarity loss value between the first text and the second label is ,in is the second cosine similarity value; sum and average the six cosine similarity loss values ​​to get the cosine similarity loss value of the first text;

[0067] The cosine similarity loss values ​​of the 10 texts in the first iteration are summed and averaged to obtain the cosine similarity loss value of the first iteration. The text extraction effective module is optimized (updated) according to the cosine similarity loss value of the first iteration to improve the effect of the text feature extraction module.

[0068] S15, repeat the above S11-S14 process until the final calculated loss value fluctuates within the first preset range and the training is stopped to obtain the final text module;

[0069] Repeat the above iterative process until the text training set is trained (that is, after multiple iterations) and multiple rounds of training are performed, and the final calculated cosine similarity loss value of the current iteration is within the first preset range (for example: The training is stopped when the fluctuation is within 0.1%, that is, the model tends to be stable, and the final text module of the first ship open set detection model is obtained.

[0070] Fine-tune the initial visual module of the second ship open set detection model.

[0071] In a preferred embodiment of the present invention, the second ship open set detection model is GroundingDINO, and its initial visual module, i.e., initial visual feature extraction module, is fine-tuned;

[0072] include:

[0073] S21, obtaining a training set of images, labeling targets that are ships as ships, manually calibrating the coordinates of each target, and inputting them into the original ship open set detection model for detection to obtain a description of each target;

[0074] Obtain a training set of images containing ship targets, manually calibrate the coordinates of each target, label the targets as ships, and input them into the original ship open set detection model (for example: chatgpt) for detection to obtain the description information of each target. The description information of each target obtained is then manually corrected to filter out inaccurate descriptions.

[0075] S22, assigning the weight of the final text module to the initial text module of the second ship open set detection model and performing feature modification to obtain a modified text module; adding a visual extraction effective module to the initial visual module to obtain an updated visual module; freezing the parameters of the initial visual module and the modified text module;

[0076] Specifically, the weight of the final text module is assigned to the initial text module of the second ship open set detection model and the features are modified, specifically: the features of the initial text module and the features of the optimized text extraction effective module are added to obtain a modified text module; in this way, the model maintains the professional ability of ship detection while maintaining the generalization ability of the original model.

[0077] An additional visual extraction effective module (such as the LoRA module) is added to the initial visual module. The addition of the LoRA module enables the model to better adapt to new ship target detection tasks. The LoRA module enhances the generalization ability of the model when dealing with new ship types (especially unseen ship categories) by reducing the redundancy of model parameters.

[0078] The parameters of the initial vision module and the modified text module are frozen, and only the parameters of the newly added vision extraction module are optimized to ensure that the feature extraction capability of ship vision is enhanced while maintaining the original generalization capability; the amount of calculation in the training process is reduced, while ensuring the high efficiency of training and avoiding the high computational cost of large-scale model fine-tuning.

[0079] S23, using the modified text module to extract and encode the features of the description of each target to obtain a text feature sequence; using the frozen updated visual module to extract and encode the features of the image training set to obtain a visual feature sequence;

[0080] Suppose there are 100 pictures, each picture contains one or more targets, and there are 300 targets in total. Feature extraction and encoding are performed on the description of each target to obtain the text feature sequence of each target. Feature extraction and encoding are performed on each picture to obtain the visual feature sequence of each target.

[0081] S24, fusing the text feature sequence and the visual feature sequence and inputting them into the detection head for detection to obtain the predicted target coordinates, category, description feature sequence and probability;

[0082] Specifically, the text feature sequence and visual feature sequence of each target are subjected to a cross-attention mechanism to obtain a fused feature; the fused feature is then subjected to a self-attention mechanism to obtain a visual feature with text description information; the visual feature with text description information is then input into the detection head for detection to obtain each predicted target coordinate, each predicted target category, each predicted target description feature sequence, and each predicted target probability.

[0083] S25, calculating the visual loss according to the manually calibrated target coordinates, the predicted target coordinates, the text feature sequence, the predicted target description feature sequence, the predicted target category, the predicted target probability, and the target label, and optimizing the visual extraction effective module according to the calculated loss value;

[0084] The S25 includes:

[0085] Perform an intersection-and-combination comparison between the manually calibrated target coordinates and the predicted target coordinates to obtain an intersection-and-combination ratio. If the intersection-and-combination ratio is greater than a preset intersection-and-combination ratio (in a preferred embodiment of the present invention, the preset intersection-and-combination ratio is 0.3):

[0086] Generate a coordinate loss value based on the manually calibrated target coordinates and the predicted target coordinates;

[0087] The existing visual module loss calculation only calculates the coordinate loss value. In order to improve the detection accuracy and prevent missed detection and false detection, the present invention adds a feature sequence loss value and a category loss value.

[0088] Generate a first feature sequence loss value according to the text feature sequence and the prediction target description feature sequence;

[0089] The first feature sequence loss value is calculated according to the following formula:

[0090]

[0091] in, is the similarity between the description of the current target and the description of the current predicted target, is the current text feature sequence, Describe the feature sequence for the current prediction target, is the first feature sequence loss value, K is the number of pictures in the picture training set, k is the kth picture in the picture training set, N is the number of targets in the current picture, n is the nth target in the current picture, D is the type of description of all targets corresponding to the current picture, d is the type of description of the dth target, y indicates whether the type of the current target in the current picture corresponds to the type of the description of the current target, if yes, y is 1, if no, y is 0.

[0092] Take a picture as an example: there are two objects in the picture (a ship and a car), and their corresponding object descriptions are: a white ship and a blue car;

[0093] A white ship (the first target description) generates the first text feature sequence by modifying the text module, and a blue car (the second target description) generates the second text feature sequence by modifying the text module;

[0094] The image is predicted by the detection head to obtain the first predicted target description feature sequence (ship) and the second predicted target description feature sequence (car);

[0095] Multiplying the first prediction target description feature sequence with the first text feature sequence to obtain a first similarity value, and multiplying the first prediction target description feature sequence with the second text feature sequence to obtain a second similarity value;

[0096] Since the category of the predicted target corresponding to the first target of the image corresponds to the first target description, y=1, and the similarity loss value between the first predicted target and the first target description is , the category of the predicted target corresponding to the first target of the image does not correspond to the second target description, then y=0, then the similarity loss value between the first predicted target and the second target description is ; Sum and average the two similarity loss values ​​to obtain the similarity loss value of the first target of the image, sum and average the similarity loss values ​​of the two targets of the image to obtain the similarity loss value of the image, sum and average the similarity loss values ​​of all images to obtain the similarity loss value of the image training set, that is, the first feature sequence loss value.

[0097] Generate the first category loss value based on the predicted target category, target label and predicted target confidence;

[0098] The present invention is to retain the characteristics of open set target detection and have the expert ability of ships. Therefore, the present invention only performs category loss calculation for ships, and other targets are set as others.

[0099] Specifically, when the predicted target category is a ship, the category loss value of the current predicted target is calculated according to the following formula:

[0100]

[0101] in, is the category loss value of the current prediction target, Whether the current predicted target category corresponds to the current target label. If so, is 1, if there is no corresponding is 0, The probability of predicting the current prediction target as a ship;

[0102] When the predicted target category is not a ship, but the label is a ship, the category loss value of the current predicted target is calculated according to the following formula:

[0103]

[0104] in, is the category loss value of the current prediction target, The probability of predicting other categories for the current prediction target, The probability of predicting the current prediction target as a ship;

[0105] The loss value of the first category is obtained by summing up and averaging the category loss values ​​of all predicted target categories as ships in the image training set and the category loss values ​​of all predicted target categories as ships in the image training set but whose labels are not ships.

[0106] Take an object in a picture as an example: Assuming that the actual label of the object is a ship, the object is predicted by the detection head and the predicted target category is a ship, which means that the current predicted target category corresponds to the current target label. is 1; assuming that the actual label of the target is a car, and the target is predicted by the detection head, the predicted target category is a ship, which means that the current predicted target category does not correspond to the current target label. is 0. This method can be used to obtain the category loss value of the predicted target;

[0107] Take an object in a picture as an example: Assuming that the actual label of the object is a ship, the object is predicted by the detection head to obtain a predicted object, and the probability of the predicted object being a car is 0.7, and the probability of the predicted object being a ship is 0.3, then the predicted target category is a car. This method can be used to obtain the category loss value of the object;

[0108] The loss value of the first category is obtained by summing up and averaging the category loss values ​​of all predicted target categories as ships in the image training set and the category loss values ​​of all predicted target categories as ships in the image training set but whose labels are not ships.

[0109] The first visual loss value is generated by summing the coordinate loss value, the first feature sequence loss value, and the first category loss value;

[0110] Specifically, the formula is:

[0111]

[0112] in, is the first visual loss value, is the coordinate loss value, is the loss value of the first feature sequence, is the first category loss value.

[0113] If the intersection-and-combination ratio is less than or equal to the preset intersection-and-combination ratio:

[0114] Generate a second feature sequence loss value according to the text feature sequence and the prediction target description feature sequence;

[0115] The second feature sequence loss value is calculated according to the following formula:

[0116]

[0117] in, is the similarity between the description of the current target and the description of the current predicted target, is the current text feature sequence, Describe the feature sequence for the current prediction target, is the second feature sequence loss value, K is the number of pictures in the picture training set, k is the kth picture in the picture training set, N is the number of targets in the current picture, n is the nth target in the current picture, D is the type of description of all targets corresponding to the current picture, and d is the type of description of the dth target.

[0118] Generate the second category loss value based on the predicted target category, the target label and the predicted target confidence;

[0119] The second category loss calculation is performed only when the predicted target category is a ship. The second category loss value is calculated according to the following formula:

[0120]

[0121] in, is the second category loss value, Whether the current predicted target category corresponds to the current target label. If so, is 1, if there is no corresponding is 0, is the probability that the current prediction target is predicted to be a ship; F is the number of all prediction targets whose categories are ships in the image training set, and f is the fth prediction target whose category is ships in the image training set.

[0122] The second feature sequence loss value and the second category loss value are summed to generate a second visual loss value;

[0123] Specifically, the formula is:

[0124]

[0125] in, is the first visual loss value, is the loss value of the second feature sequence, is the second category loss value.

[0126] A total visual loss value is calculated according to the first visual loss value and the second visual loss value, and the visual extraction effective module is optimized according to the calculated total visual loss value.

[0127] Specifically, the formula is:

[0128]

[0129] in, is the total visual loss value, is a hyperparameter (in a preferred embodiment of the present invention, it is set to 0.3), is a hyperparameter (in a preferred embodiment of the present invention, it is set to 0.7);

[0130] S26, repeat the above S21-S25 process until the final calculated loss value fluctuates within the second preset range, and then stop training to obtain the final visual module;

[0131] Repeat the above iterative process until the image training set is trained for multiple rounds and the total visual loss value finally calculated is within the second preset range (for example: The training is stopped when the fluctuation is within 0.1%, that is, the model tends to be stable, and the final visual module of the second ship open set detection model is obtained.

[0132] S3, modifying the features of the final visual module to obtain a modified visual module and a corresponding final second ship open set detection model;

[0133] The final visual module is feature modified, specifically: the features of the initial visual module and the features of the optimized visual extraction effective module are added to obtain a modified visual module; this allows the model to maintain the professional ability of ship detection while maintaining the generalization ability of the original model.

[0134] S4. Input the image of the ship to be detected into the final second ship open set detection model for detection, and obtain the ship coordinates, ship category and ship description.

[0135] By inputting the ship images to be detected into the optimized final second ship open set detection model, the model can accurately detect and output the coordinates, category and description information of the ship. This process utilizes the multimodal fusion of visual and text information to improve the accuracy and robustness of ship target detection, especially when facing new or unseen ship types, it shows excellent adaptability.

[0136] Figure 2 is a structural diagram of a text information based ship open set target detection system provided by an embodiment of the present invention, such as Figure 2 As shown, the system includes:

[0137] The first fine-tuning unit 201 is used to fine-tune the initial text module of the first ship open set detection model, including:

[0138] The first adding subunit 2011 is used to add a text extraction effective module to the initial text module to obtain an updated text module; and freeze the parameters of the initial text module; the constructing subunit 2012 is used to construct a text training set, including: text containing ship description information and labeled as ship; and text not containing ship description information and labeled as non-ship; the first encoding subunit 2013 is used to use the initial text module to extract and encode the label to obtain a label feature sequence; use the frozen updated text module to extract and encode the text training set to obtain a training set feature sequence; the first calculating subunit 2014 is used to calculate the cosine similarity loss according to the label feature sequence and the training set feature sequence, and optimize the text extraction effective module according to the calculated loss value; the first repeating subunit 2015 is used to repeat the above process until the last calculated loss value fluctuates within the first preset range, and the training is stopped to obtain the final text module;

[0139] The second fine-tuning unit 202 is used to fine-tune the initial visual module of the second ship open set detection model, including:

[0140] The data preprocessing unit 2021 is used to obtain a picture training set, label the target as a ship as a ship, manually calibrate the coordinates of each target, and input them into the original ship open set detection model for detection to obtain a description of each target; the second adding subunit 2022 is used to assign the weight of the final text module to the initial text module of the second ship open set detection model and perform feature modification to obtain a modified text module; add a visual extraction valid module to the initial visual module to obtain an updated visual module; freeze the parameters of the initial visual module and the modified text module; the second encoding subunit 2023 is used to use the modified text module to extract and encode the description of each target to obtain a text feature sequence; use the frozen updated visual module The module extracts and encodes features of the image training set to obtain a visual feature sequence; a fusion subunit 2024 is used to fuse the text feature sequence and the visual feature sequence and input them into the detection head for detection to obtain predicted target coordinates, categories, description feature sequences and probabilities; a second calculation subunit 2025 is used to calculate the visual loss based on the manually calibrated target coordinates, predicted target coordinates, text feature sequences, predicted target description feature sequences, predicted target categories, predicted target probabilities, and target labels, and optimize the visual extraction effective module based on the calculated loss value; a second repetition subunit 2026 is used to repeat the above process until the last calculated loss value fluctuates within the second preset range, and the training is stopped to obtain the final visual module;

[0141] A modification unit 203 is used to modify the features of the final visual module to obtain a modified visual module and a corresponding final second ship open set detection model;

[0142] The detection unit 204 is used to input the image of the ship to be detected into the final second ship open set detection model for detection, and obtain the ship coordinates, ship category and ship description.

[0143] The text information based ship open set target detection system provided by the present invention corresponds to the above method, and aims to achieve efficient ship target detection through multimodal data fusion. The system includes multiple functional units, each unit corresponds to a different step in the method, and works together to achieve the detection and classification of ship targets. In order to avoid redundant description, the present invention will no longer describe the implementation details of the system one by one.

[0144] Beneficial effects of the present invention:

[0145] 1. Improve the accuracy and professionalism of ship target detection:

[0146] The present invention enhances the professionalism of the ship target detection model by introducing text description information. The text feature extraction module can extract specific description information of the ship, such as ship type, function, size, etc., so that the model can not only recognize the visual features of the ship, but also understand the semantic information of the ship, thereby improving the accuracy of ship detection, especially when facing complex backgrounds or new types of ships, it can maintain a high detection accuracy.

[0147] 2. Enhance the generalization ability of the model:

[0148] By freezing the parameters of the initial text module, only the newly added text extraction effective module is optimized; and by freezing the parameters of the initial visual module and modifying the text module, only the newly added visual extraction effective module is optimized; the present invention maintains the original generalization ability of the model, and the optimized module enables the model to efficiently learn the characteristics of ship targets while maintaining a low computing cost, thereby improving the performance of the model in the detection of new types of ships and having better adaptability.

[0149] 3. Efficient multimodal information fusion:

[0150] By fusing visual features and text description information, the present invention not only enhances the positioning capability of ship targets, but also can accurately combine the description of the ship with the visual features. This multimodal fusion method can better cope with open set target detection tasks, improves the model's ability to recognize unknown ship types, and ensures high accuracy and robustness of ship target detection.

[0151] 4. Optimize training efficiency and reduce hardware requirements:

[0152] By freezing some parameters of the initial text model and only optimizing the newly added effective text extraction modules, the number of parameters that need to be updated during the model training process is greatly reduced, reducing the consumption of computing resources. This not only improves training efficiency, but also reduces the demand for hardware resources, especially for environments with limited hardware resources.

[0153] 5. Flexible data utilization and training optimization:

[0154] Calculation methods such as cosine similarity loss, feature sequence loss, and category loss are used to optimize the matching of text features and visual features during the training process. Through this joint training strategy, the model can effectively handle the heterogeneity of text and visual information and improve the overall performance of ship target detection.

[0155] 6. Solve the problem of missed detection in ship category detection:

[0156] Traditional closed-set target detection methods often cannot identify ship categories that do not appear in the training set. The present invention is based on open-set target detection. When processing new and unseen ship types, it can effectively reduce the incidence of missed detection by combining text information, thereby improving the robustness of the ship target detection model.

[0157] 7. Wide applicability:

[0158] The method of the present invention can not only be applied to ship target detection, but also be extended to open set target detection tasks in other fields, and has strong versatility and applicability. Whether facing fixed target categories or constantly changing target categories, the model can cope well and provide effective detection results.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting open-set ship targets based on text information, characterized in that: include: S1. Fine-tune the initial text module of the first ship open set detection model, including: adding a text extraction effective module to the initial text module to obtain an updated text module; and freezing the parameters of the initial text module; constructing a text training set, including: text containing ship description information and text without ship description information, and setting a label for each text; using the initial text module to extract and encode the label to obtain a label feature sequence; using the frozen updated text module to extract and encode the text training set to obtain a training set feature sequence; calculating the cosine similarity loss according to the label feature sequence and the training set feature sequence, and optimizing the text extraction effective module according to the calculated loss value; repeating the above process until the last calculated loss value fluctuates within the first preset range, stopping the training, and obtaining the final text module; S2. Fine-tune the initial visual module of the second ship open set detection model, including: obtaining a picture training set, labeling the target as a ship as a ship, manually calibrating the coordinates of each target, and inputting them into the original ship open set detection model for detection to obtain a description of each target; assigning the weight of the final text module to the initial text module of the second ship open set detection model and performing feature modification to obtain a modified text module; adding a visual extraction effective module to the initial visual module to obtain an updated visual module; freezing the parameters of the initial visual module and the modified text module; using the modified text module to extract and encode the description of each target to obtain text features. feature sequence; use the frozen updated visual module to extract and encode the features of the image training set to obtain a visual feature sequence; fuse the text feature sequence and the visual feature sequence and input them into the detection head for detection to obtain the predicted target coordinates, category, description feature sequence and probability; calculate the visual loss based on the manually calibrated target coordinates, predicted target coordinates, text feature sequence, predicted target description feature sequence, predicted target category, predicted target probability, and target label, and optimize the visual extraction effective module based on the calculated loss value; repeat the above process until the last calculated loss value fluctuates within the second preset range and stops training to obtain the final visual module; S3, modifying the features of the final visual module to obtain a modified visual module and a corresponding final second ship open set detection model; S4. Input the image of the ship to be detected into the final second ship open set detection model for detection, and obtain the ship coordinates, ship category and ship description.

2. The method according to claim 1, characterized in that: Before using the frozen updated text module to extract and encode the text training set to obtain the training set feature sequence, the following steps are included: The text training set is divided into multiple iterations. A fixed number of texts are randomly extracted from the text training set in each iteration. Each extracted text includes both texts containing ship description information and texts without ship description information. All texts in each iteration are input as the current iteration text training set into the frozen update text module for feature extraction and encoding to obtain each text feature sequence.

3. The method according to claim 2, characterized in that The final text module is obtained by the following method: According to all label feature sequences of the current iteration text training set and the current text feature sequence of the current iteration text training set, the cosine similarity value between the current text and each label is calculated; The cosine similarity loss value of the current text is calculated based on the cosine similarity value between the current text and each label, as well as the label of the current text; The cosine similarity loss values ​​of each text in the current iteration text training set are summed and averaged to obtain the cosine similarity loss value of the current iteration; the effective module for text extraction is optimized according to the cosine similarity loss value of the current iteration; The above iterative process is repeated until the text training set is trained and multiple rounds of training are performed. The training is stopped when the cosine similarity loss value of the current iteration calculated at last fluctuates within the first preset range, and the final text module of the first ship open set detection model is obtained.

4. The method according to claim 3, characterized in that The current iteration cosine similarity loss value is calculated according to the following formula: ； ； in, is the cosine similarity value between the current j-th text and the current i-th label, is the current i-th label feature sequence, is the current j-th text feature sequence, is the cosine similarity loss value of the current iteration, M is the number of labels in the current iteration text training set, i is the i-th label in the current iteration text training set, P is the number of texts in the current iteration text training set, j is the j-th text in the current iteration text training set, and y indicates whether the label of the current text corresponds to the current label. If so, y is 1, otherwise, y is 0.

5. The method according to claim 1, characterized in that The method of calculating the visual loss according to the manually calibrated target coordinates, the predicted target coordinates, the text feature sequence, the predicted target description feature sequence, the predicted target category, the predicted target probability, and the target label, and optimizing the visual extraction effective module according to the calculated loss value includes: Perform an intersection-and-combination comparison between the manually calibrated target coordinates and the predicted target coordinates to obtain an intersection-and-combination ratio. If the intersection-and-combination ratio is greater than the preset intersection-and-combination ratio: Generate a coordinate loss value based on the manually calibrated target coordinates and the predicted target coordinates; Generate a first feature sequence loss value according to the text feature sequence and the prediction target description feature sequence; Generate the first category loss value based on the predicted target category, target label and predicted target confidence; The first visual loss value is generated by summing the coordinate loss value, the first feature sequence loss value, and the first category loss value; If the intersection-and-combination ratio is less than or equal to the preset intersection-and-combination ratio: Generate a second feature sequence loss value according to the text feature sequence and the prediction target description feature sequence; Generate the second category loss value based on the predicted target category, the target label and the predicted target confidence; The second feature sequence loss value and the second category loss value are summed to generate a second visual loss value; A total visual loss value is calculated according to the first visual loss value and the second visual loss value, and the visual extraction effective module is optimized according to the calculated total visual loss value.

6. The method according to claim 5, characterized in that: The first feature sequence loss value is calculated according to the following formula: ； in, is the similarity between the description of the current target and the description of the current predicted target, is the current text feature sequence, Describe the feature sequence for the current prediction target, is the first feature sequence loss value, K is the number of pictures in the picture training set, k is the kth picture in the picture training set, N is the number of targets in the current picture, n is the nth target in the current picture, D is the type of description of all targets corresponding to the current picture, d is the type of description of the dth target, y indicates whether the type of the current target in the current picture corresponds to the type of the description of the current target, if yes, y is 1, if no, y is 0.

7. The method according to claim 5, characterized in that: When the predicted target category is a ship, the category loss value of the current predicted target is calculated according to the following formula: ； in, is the category loss value of the current prediction target, Whether the current predicted target category corresponds to the current target label. If so, is 1, if there is no corresponding is 0, The probability of predicting the current prediction target as a ship; When the predicted target category is not a ship, but the label is a ship, the category loss value of the current predicted target is calculated according to the following formula: ； in, is the category loss value of the current prediction target, The probability of predicting other categories for the current prediction target, The probability of predicting the current prediction target as a ship; The loss value of the first category is obtained by summing up and averaging the category loss values ​​of all predicted target categories as ships in the image training set and the category loss values ​​of all predicted target categories as ships in the image training set but whose labels are not ships.

8. The method according to claim 5, characterized in that: The second feature sequence loss value is calculated according to the following formula: ； in, is the similarity between the description of the current target and the description of the current predicted target, is the current text feature sequence, Describe the feature sequence for the current prediction target, is the second feature sequence loss value, K is the number of pictures in the picture training set, k is the kth picture in the picture training set, N is the number of targets in the current picture, n is the nth target in the current picture, D is the type of description of all targets corresponding to the current picture, and d is the type of description of the dth target.

9. The method according to claim 5, characterized in that: The second category loss calculation is performed only when the predicted target category is a ship. The second category loss value is calculated according to the following formula: ； in, is the second category loss value, Whether the current predicted target category corresponds to the current target label. If so, is 1, if there is no corresponding is 0, is the probability that the current prediction target is predicted to be a ship; F is the number of all prediction targets whose categories are ships in the image training set, and f is the fth prediction target whose category is ships in the image training set.

10. A ship open set target detection system based on text information, characterized in that: include: The first fine-tuning unit is used to fine-tune the initial text module of the first ship open set detection model, including: The first adding subunit is used to add a text extraction effective module to the initial text module to obtain an updated text module; and freeze the parameters of the initial text module; the constructing subunit is used to construct a text training set, including: text containing ship description information and text without ship description information, and set a label for each text; the first encoding subunit is used to use the initial text module to extract and encode the label to obtain a label feature sequence; use the frozen updated text module to extract and encode the text training set to obtain a training set feature sequence; the first calculating subunit is used to calculate the cosine similarity loss according to the label feature sequence and the training set feature sequence, and optimize the text extraction effective module according to the calculated loss value; the first repeating subunit is used to repeat the above process until the last calculated loss value fluctuates within the first preset range, and the training is stopped to obtain a final text module; The second fine-tuning unit is used to fine-tune the initial visual module of the second ship open set detection model, including: The data preprocessing unit is used to obtain a picture training set, label the target as a ship as a ship, manually calibrate the coordinates of each target, and input them into the original ship open set detection model for detection to obtain a description of each target; the second adding subunit is used to assign the weight of the final text module to the initial text module of the second ship open set detection model and perform feature modification to obtain a modified text module; add a visual extraction effective module to the initial visual module to obtain an updated visual module; freeze the parameters of the initial visual module and the modified text module; the second encoding subunit is used to use the modified text module to extract and encode the description of each target to obtain a text feature sequence; use the frozen updated visual module to extract and encode the description of each target The module extracts and encodes features of the image training set to obtain a visual feature sequence; a fusion subunit is used to fuse the text feature sequence and the visual feature sequence and input them into the detection head for detection to obtain predicted target coordinates, categories, description feature sequences and probabilities; a second calculation subunit is used to calculate the visual loss based on the manually calibrated target coordinates, predicted target coordinates, text feature sequences, predicted target description feature sequences, predicted target categories, predicted target probabilities, and target labels, and optimize the visual extraction effective module based on the calculated loss value; a second repetition subunit is used to repeat the above process until the last calculated loss value fluctuates within the second preset range, and the training is stopped to obtain the final visual module; A modification unit, used for performing feature modification on the final visual module to obtain a modified visual module and a corresponding final second ship open set detection model; The detection unit is used to input the image of the ship to be detected into the final second ship open set detection model for detection, and obtain the ship coordinates, ship category and ship description.

Images