Marine work equipment identification method based on vision

Abstract

The invention provides a vision-based maritime work equipment monitoring method, which comprises the following steps of: acquiring and preprocessing a maritime work site image, and performing image segmentation by using an SAM model to automatically generate a high-quality target mask; and in combination with a text prompt input by a user, the Grounding DINO model automatically detects the maritime work equipment in the image and generates a detection frame, and the detection frame is combined with a segmentation mask of the SAM model to refine a detection result. A Grouping DINO model is finely adjusted on an SAM automatic labeling data set, the ability of the Grouping DINO model to adapt to the marine environment is improved, and the detection performance is continuously improved through the loop optimization process. According to the method, excellent performance is shown on a ShipDataset data set, the average precision (AP) of the method exceeds that of a traditional model, particularly on maritime work equipment detection, high robustness and precision are shown, and powerful technical support is provided for safety and efficiency of offshore operation.

Description

Technical Field

[0001] The invention relates to computer vision and marine working equipment recognition technology, and in particular to a vision-based efficient marine working equipment recognition method. Background Art

[0002] When operating at sea, if there are ships passing nearby, the risk of collision increases, which may seriously affect the success rate of the operation. In order to reduce this risk, it is necessary to be able to more comprehensively evaluate the operating status of equipment under complex marine conditions and identify potential risk factors in advance.

[0003] In recent years, the automatic detection technology of ships and surface objects has made significant progress and can be divided into three categories: (1) background modeling and subtraction methods, (2) methods based on human visual attention mechanism, and (3) technologies based on edge and texture features. Background modeling and subtraction techniques model the sea surface background through median modeling or Gaussian model and Gaussian mixture model, and then identify ships through background subtraction. Although such methods are effective, their performance is limited on non-fixed camera platforms. The second type of method is based on the human visual attention model and uses local contrast or spectral residuals to construct a saliency map to highlight the target area.

[73] . Although these methods can reduce the ocean background noise to a certain extent, they are not very effective when dealing with large-scale ocean waves. These methods reduce the ocean background noise to a certain extent, but the effect is still insufficient when dealing with large-scale ocean wave interference. The third type of method focuses on using edge and texture features, such as gradient information, Hough transform, local binary pattern and gradient histogram to detect ships. These technologies improve detection accuracy by analyzing the texture information of the sea background, but their computational complexity poses a great challenge to real-time video processing.

[0004] Equipment inspection on offshore platforms faces significant challenges (see Figure 1 ), mainly because the reflected light (glare) from the water surface interferes with the image and reduces the accuracy of the algorithm. In addition, the dynamic environment of the ocean, such as waves, water quality and weather changes, further exacerbates the difficulty of detection. These factors affect the image quality, making it difficult to distinguish between the target and the background. For example, the similar visual features of waves and small devices can easily lead to misjudgment, and changes in water transparency also affect image clarity.

[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0006] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a method for identifying marine working equipment based on vision.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for identifying marine working equipment based on vision comprises the following steps:

[0009] S1. Collect image data of the offshore operation site and preprocess it to prepare a dataset for subsequent model training and testing;

[0010] S2. Use the SAM model to perform image segmentation and automatically generate high-quality target object masks, where the SAM model performs pixel-level analysis on the preprocessed image to identify and segment the offshore equipment in the image, generate the corresponding segmentation mask, and construct an automatic annotation dataset containing rich offshore equipment images and corresponding masks;

[0011] S3. Automatic detection using the Grounding DINO model, wherein the Grounding DINO model is used to automatically detect the marine equipment in the image and generate a detection frame based on the text prompts input by the user, which is combined with the segmentation mask generated by the SAM model to refine the detection results;

[0012] S4. On the automatically annotated dataset constructed by the SAM model, the Grounding DINO model is fine-tuned through cross-modal feature fusion and self-supervised learning strategy to make it better adapt to the offshore operating platform environment and improve the generalization ability and detection accuracy of the model;

[0013] S5. Loop through the optimization of the annotation and object detection system, in which the fine-tuned Grounding DINO model is replaced back into the automatic annotation system. The automatic annotation system uses the fine-tuned model to automatically generate more accurate detection boxes and segmentation masks to improve the annotation accuracy, and uses the newly annotated data to continue fine-tuning the Grounding DINO model, forming a loop optimization process to continuously improve the detection performance.

[0014] Furthermore, the SAM model includes:

[0015] Input processing module, which is used to standardize the input image, including resizing, color space conversion, and enhancement techniques to improve the robustness of the model;

[0016] The feature extraction network uses a CNN-based backbone network, such as ResNet or EfficientNet, and gradually extracts high-level features of the image through multi-layer convolution operations. At the same time, a self-attention mechanism is introduced to improve feature expression capabilities;

[0017] The segmentation mask generation module uses a combination of multi-layer perceptron MLP and convolutional layers to generate the final segmentation mask, while considering contextual information to improve segmentation accuracy, and applying post-processing algorithms such as conditional random fields (CRF) to optimize the segmentation results;

[0018] The post-processing module, including edge smoothing technology and pixel recalibration scheme, uses graphics algorithms to optimize the segmentation contours and enhance the quality of the final mask.

[0019] Furthermore, the SAM model is trained using a diverse dataset containing labeled and unlabeled images to enhance the performance of the model under self-supervised learning; a combination of cross-entropy loss and contrast loss is adopted to ensure the accuracy of the model in pixel-level segmentation; dynamic learning rate adjustment based on training progress is implemented to adjust the learning rate in real time according to the training progress and actual performance of the model to ensure that the model can converge quickly in the early stage of training while remaining stable in the later fine-tuning stage.

[0020] Furthermore, the Grounding DINO model is based on the DINO architecture and integrates self-supervised learning and dynamic feature extraction. It processes the input image through multi-scale feature representation to enhance the model's ability to capture details. It uses self-supervised learning to train the model on unlabeled data. The model dynamically adjusts the feature extraction process to adapt to different marine environments and equipment characteristics.

[0021] Furthermore, the Grounding DINO model has the following features:

[0022] Dynamically adjust normalization parameters based on the distribution of input features to improve feature expression and alleviate the performance degradation of batch normalization;

[0023] During the training process, the model adopts a self-supervised learning strategy, training with unlabeled data to discover the potential structure in the data, reduce dependence on labeled data and improve generalization ability.

[0024] The model uses a multi-layer convolutional neural network (CNN) structure to extract features at different levels, capture object diversity, and combine object appearance features with contextual information to accurately locate the target object;

[0025] In the SAM-Grounding DINO framework, users interactively select image parts as expert prompts, and the model automatically matches entities and generates accurate boxes and masks to achieve automatic annotation of the entire image.

[0026] Furthermore, the operation process of the automatic annotation system includes:

[0027] Using the automatic segmentation capability of the SAM model, we assist annotators in preliminary mask annotation of target objects in the image, providing basic data for subsequent automatic annotation and model training.

[0028] The manually annotated data is used to fine-tune the Grounding DINO model so that it can learn the characteristics of offshore equipment and improve its performance in detection tasks;

[0029] The fine-tuned Grounding DINO model is applied to unlabeled images to automatically detect offshore equipment and generate detection frames. The detection results are input into the SAM model together with the unlabeled images to generate accurate segmentation masks and complete fully automatic labeling.

[0030] Through cyclic adaptive learning, the model learns the characteristics of offshore engineering equipment in specific scenarios in the manual labeling stage, and applies these characteristics in the automatic labeling stage to expand the labeling scope and efficiency.

[0031] Furthermore, the Grounding DINO model includes:

[0032] The text encoder receives the input text and converts it into text features to provide semantic information for the model;

[0033] An image encoder processes the input image, extracts image features, and provides a numerical description of the image content;

[0034] The text-image fusion module combines text features and image features, enhances feature expression capabilities through cross-modal learning strategies, and understands and associates text and image content;

[0035] The language-guided index extraction module uses the fused features to extract the image area related to the input text in a language-guided manner to achieve target positioning;

[0036] The cross-modal decoder combines text features (Key) and image features (Value) to generate accurate detection boxes through the cross-modal decoder, realizing the mapping of text to image content.

[0037] Furthermore, the fine-tuning of the Grounding DINO model includes:

[0038] Extract visual information from images using Swin transformer, extract text descriptions using BERT, and perform cross-modal feature fusion using the feature enhancer module;

[0039] Enhance feature fusion by adding image-to-text cross-attention and text-to-image cross-attention, as well as a language-guided query selection module;

[0040] Create a cross-modal decoder to fuse text and image features to improve modality alignment and feature representation capabilities;

[0041] During fine-tuning, we use L1 loss and GIOU loss for bounding box regression following GLIP, and use contrastive loss between predicted objects and language tags for classification.

[0042] A computer-readable storage medium stores a computer program, which implements the method when executed by a processor.

[0043] A computer program product comprises a computer program, which implements the method when executed by a processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The main technical advantages of the vision-based marine equipment identification method of the present invention include:

[0046] Automated processing and real-time detection: Utilize the visual basic models SAM and Grounding DINO to realize the automated processing and real-time detection of marine platform data, automatically process image data, and detect offshore equipment in real time. It can quickly and accurately perceive and track changes in the physical world, and reflect these changes in real time in the virtual scene, ensuring the consistency between the virtual environment and the real offshore operating environment, thus providing an efficient and automated solution for the monitoring of offshore equipment.

[0047] High-precision image segmentation and annotation: The SAM model automatically identifies and segments offshore equipment in images through pixel-level analysis, generates high-quality segmentation masks, provides accurate image segmentation information for the automatic annotation system, and constructs an automatic annotation dataset containing a wealth of offshore equipment images and corresponding masks.

[0048] Enhanced detection accuracy: Combined with the text prompts entered by the user, the Grounding DINO model automatically detects the offshore equipment in the image and generates a detection box. The detection box is combined with the segmentation mask generated by the SAM model to refine the detection results and improve the detection accuracy.

[0049] Adaptability and generalization ability: By fine-tuning the GroundingDINO model on the automatically annotated dataset built by the SAM model, it can be better adapted to the offshore operating platform environment and improve the generalization ability and detection accuracy of the model.

[0050] Loop optimization: Replace the fine-tuned Grounding DINO model back into the automatic annotation system, and use the model to automatically generate more accurate detection boxes and segmentation masks to improve annotation accuracy. Then, use the newly annotated data to continue to fine-tune the Grounding DINO model, forming a loop optimization process to continuously improve detection performance.

[0051] Robustness and efficiency: Experiments on the ShipDataset ship image dataset show that compared with other methods, this method exhibits excellent performance and the highest average precision (AP), showing outstanding performance in ship detection. In addition, this method outperforms the YOLOv7 model on the offshore equipment dataset, demonstrating its separation accuracy and robustness in the open domain.

[0052] Ability to cope with complex offshore environments: The method of the present invention can accurately detect target content based on user input text in offshore scenes, and can accurately mark even in the case of multiple targets and complex backgrounds, demonstrating the ability to identify wind turbines in wind farm environments, as well as the detection accuracy of large offshore engineering vessels and their crane equipment in complex offshore operating environments.

[0053] In general, the marine working equipment identification method of the present invention significantly improves the automation level and detection accuracy of marine working equipment monitoring through automated processing, precise segmentation, text prompt detection, model fine-tuning, loop optimization and other steps, providing new solutions and technical support for virtual-reality integration.

[0054] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Demonstrates the problems faced in detecting objects at sea

[0056] Figure 2 This is a flowchart of labeling and fine-tuning based on a visual basic model according to an embodiment of the present invention.

[0057] Figure 3 Schematic diagram of the SAM model perception capability of an embodiment of the present invention.

[0058] Figure 4 4 is an architecture diagram of the SAM model of an embodiment of the present invention.

[0059] Figure 5 4 is an architecture diagram of the Grounding DINO model according to an embodiment of the present invention.

[0060] Figure 6 It is the output result of SAM in the automatic marking system for offshore equipment in an embodiment of the present invention.

[0061] Figure 7 This is a flow chart of the automatic labeling system for marine engineering equipment according to an embodiment of the present invention.

[0062] Figure 8 FIG. 4 is a network flow chart of Grounding DINO according to an embodiment of the present invention.

[0063] Fig. 9 The results of the embodiments of the present invention on the constructed marine engineering data set are demonstrated. DETAILED DESCRIPTION

[0064] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0065] In order to comprehensively and accurately evaluate the operating status of offshore equipment under complex marine conditions and identify potential risk factors in advance, the present invention proposes a vision-based offshore equipment monitoring method, which mainly includes: by collecting and preprocessing images of the offshore operation site, using the SAM model to perform image segmentation to automatically generate high-quality target masks. Combined with the text prompts input by the user, the Grounding DINO model automatically detects the offshore equipment in the image and generates a detection frame, which is combined with the segmentation mask of the SAM model to refine the detection results. The Grounding DINO model is fine-tuned on the SAM automatic annotation dataset to improve its ability to adapt to the offshore environment, and the detection performance is continuously improved through a cyclic optimization process. This method shows excellent performance on the ShipDataset dataset, with AP surpassing traditional models, especially in the detection of offshore equipment, showing high robustness and accuracy, providing strong technical support for the safety and efficiency of offshore operations.

[0066] See also Figures 2 to 9 , a vision-based marine equipment recognition method, comprising the following steps:

[0067] Step S1. Data acquisition and preprocessing, in which image data of the offshore operation site is acquired and preprocessed to prepare a data set for subsequent model training and testing;

[0068] Step S2. Use the SAM (SegmentAnything Model) model to perform image segmentation and automatically generate high-quality target object masks, wherein the SAM model performs pixel-level analysis on the preprocessed image to identify and segment the offshore equipment in the image, and generates corresponding segmentation masks, and constructs an automatic annotation dataset containing rich offshore equipment images and corresponding masks;

[0069] Step S3. Automatic detection using the Grounding DINO model, wherein the Grounding DINO model is used to automatically detect the marine equipment in the image and generate a detection frame through the text prompt input by the user, and is combined with the segmentation mask generated by the SAM model to refine the detection results and improve the accuracy of the detection;

[0070] Step S4. Grounding DINO model fine-tuning, in which the Grounding DINO model is fine-tuned on the automatically labeled dataset constructed by the SAM model to make it better adapt to the offshore operating platform environment, and the generalization ability and detection accuracy of the model are improved through cross-modal feature fusion and self-supervised learning strategies;

[0071] Step S5. Circularly optimize the annotation and target detection system, wherein the fine-tuned Grounding DINO model is replaced back into the automatic annotation system. The automatic annotation system uses the fine-tuned model to automatically generate more accurate detection boxes and segmentation masks to improve the annotation accuracy, and uses the newly annotated data to continue to fine-tune the Grounding DINO model, forming a cyclic optimization process to continuously improve the detection performance.

[0072] In this method, step S1 provides basic data for subsequent model training and testing; step S2 and step S3 jointly realize automatic detection and labeling of equipment, step S4 optimizes the model to adapt to a specific environment; step S5 further enhances the labeling and detection capabilities of the model through cyclic optimization.

[0073] See also Figure 4 In a preferred embodiment, the SAM model includes: an input processing module, which is used to standardize the input image, including resizing, color space conversion and enhancement technology to improve the robustness of the model; a feature extraction network, which adopts a CNN-based backbone network, such as ResNet or EfficientNet, and gradually extracts high-level features of the image through multi-layer convolution operations, while introducing a self-attention mechanism to improve feature expression capabilities; a segmentation mask generation module, which uses a combination of a multi-layer perceptron (MLP) and a convolutional layer to generate the final segmentation mask, while considering contextual information to improve segmentation accuracy, and applying post-processing algorithms such as conditional random fields (CRF) to optimize the segmentation results; a post-processing module, including edge smoothing technology and pixel recalibration schemes, uses graphics algorithms to optimize the segmentation contour and enhance the quality of the final mask.

[0074] In some embodiments, the SAM model is trained using a diverse dataset containing labeled and unlabeled images to enhance the performance of the SAM model under self-supervised learning; a combination of cross-entropy loss and contrast loss is used to ensure the accuracy of the model in pixel-level segmentation; dynamic learning rate adjustment based on training progress is implemented, and the learning rate is adjusted in real time according to the training progress and actual performance of the model to ensure that the model can converge quickly in the early stages of training while remaining stable in the later fine-tuning stages.

[0075] See also Figure 5 The Grounding DINO model is based on the DINO architecture, integrating self-supervised learning and dynamic feature extraction; processing input images through multi-scale feature representation to enhance the model's ability to capture details; using self-supervised learning, the model is trained on unlabeled data. The model dynamically adjusts the feature extraction process to adapt to different marine environments and equipment characteristics.

[0076] In a preferred embodiment, the Grounding DINO model has the following features: dynamically adjust the normalization parameters according to the distribution of input features to improve feature expression capabilities and alleviate the performance degradation of batch normalization; during the training process, the model adopts a self-supervised learning strategy, trains on unlabeled data to discover potential structures in the data, reduces dependence on labeled data and improves generalization capabilities. The model extracts features at different levels through a multi-layer convolutional neural network (CNN) structure, captures object diversity, combines object appearance features with contextual information, and accurately locates the target object.

[0077] In the SAM-Grounding DINO framework, users interactively select image parts as expert prompts, and the model automatically matches entities and generates accurate boxes and masks to achieve automatic annotation of the entire image.

[0078] See also Figure 7In a preferred embodiment, the operation process of the automatic labeling system includes: manual labeling stage: using the automatic segmentation capability of the SAM model, assisting the labeling personnel to perform preliminary mask labeling on the target objects in the image, and providing basic data for subsequent automatic labeling and model training; fine-tuning the Grounding DINO model: using the manually labeled data to fine-tune the Grounding DINO model, so that it learns the characteristics of marine equipment and improves its performance in the detection task; automatic labeling stage: applying the fine-tuned Grounding DINO model to the unlabeled image, automatically detecting the marine equipment and generating a detection frame, and inputting the detection results together with the unlabeled image into the SAM model to generate an accurate segmentation mask to complete the fully automatic labeling; loop optimization: through loop adaptive learning, the model learns the characteristics of marine equipment in specific scenarios in the manual labeling stage, and efficiently applies these characteristics in the automatic labeling stage to expand the labeling scope and efficiency.

[0079] like Figure 8 As shown, in a preferred embodiment, the Grounding DINO model includes: a text encoder: receiving input text, such as "Ship", and converting it into text features to provide semantic information for the model; an image encoder: processing input images, extracting image features, and providing numerical descriptions for image content; a text image fusion module: combining text features and image features, and enhancing feature expression capabilities through a cross-modal learning strategy to better understand and associate text and image content; a language-guided index extraction module: using the fused features, extracting image areas related to the input text in a language-guided manner to achieve accurate target positioning; a cross-modal decoder: combining text features (Key) and image features (Value), and generating accurate detection boxes through a cross-modal decoder to achieve mapping from text to image content. Through the collaborative work of the above modules, automatic recognition and labeling of offshore equipment can be achieved, and the level of automation of monitoring can be improved.

[0080] The fine-tuning of the Grounding DINO model includes: Feature extraction and fusion: extract visual information from images using Swin transformer, extract text descriptions through BERT, and perform cross-modal feature fusion through the feature enhancer module; Cross-attention mechanism: enhance feature fusion by adding cross-attention from image to text and cross-attention from text to image, as well as a language-guided query selection module; Cross-modal decoder design: create a cross-modal decoder to fuse text and image features to improve modality alignment and feature expression capabilities. During fine-tuning, L1 loss and GIOU loss are used for bounding box regression, following GLIP, and contrast loss between predicted objects and language tags is used for classification.

[0081] The specific embodiments and experimental verification of the present invention are further described below.

[0082] The present invention provides a vision-based method for identifying marine equipment, which can improve the safety and success rate of marine operations and ensure that the equipment can still operate safely and stably in the face of complex environments and ship traffic. The method develops a cascade pipeline that combines a visual base model SAM and a Grounding DINO model. The method uses the SAM-Grounding DINO framework to automatically label relevant data of offshore operating platforms in batches, and fine-tunes the Grounding DINO model to improve the accuracy of marine equipment detection. On the ship image dataset of ShipDataset, the fine-tuned Grounding DINO model achieved the best results compared with other methods. Through these experimental results, the present invention demonstrates the great potential of the visual base model in marine equipment detection tasks, and through its powerful image processing and recognition capabilities, it provides the possibility for automatically generating and updating content in virtual environments.

[0083] The proposed automatic image annotation method based on the visual basic model can generate a large amount of high-quality training data, further optimize the target detection model, and effectively deal with the impact of adverse factors such as waves and glare on the detection results. This provides strong technical support for the safety and success rate of offshore operations. This method uses the automatic image annotation function of the visual basic model to create a large amount of high-quality training data, thereby further optimizing the target detection model based on text prompts. This process is like Figure 2 In this way, the robustness of the basic model can be fully utilized to effectively reduce the adverse effects of factors such as waves and glare in the marine environment on the detection effect. The specific steps are as follows:

[0084] 1. Automatic marking system for offshore equipment

[0085] The powerful capability of the SAM (SegmentAnything Model) model lies in its pre-training on large-scale data sets, which enables it to effectively identify and annotate different objects in images. The present invention cascades the SAM model and the Grounding DINO model to automatically annotate offshore equipment. Through this step, the method can quickly generate high-quality offshore equipment target image annotations, providing additional data for subsequent fine-tuning of the model.

[0086] 2. Grounding DINO model fine-tuning

[0087] The Grounding DINO model was originally designed for natural environments. By fine-tuning the data set obtained in the previous step, the model can better adapt to the particularities of the offshore platform environment, thereby improving the accuracy of offshore equipment detection. In addition, a significant advantage of the Grounding DINO model is that it supports learning based on text prompts. This prompt-based learning method can more effectively utilize the potential of large models, save a lot of computing resources, and improve the flexibility and efficiency of the model in adapting to new tasks.

[0088] 3. Loop optimization of labeling and object detection system

[0089] After fine-tuning the Grounding DINO model, it is used to replace the corresponding model in the marine equipment automatic annotation system to improve the accuracy of the given annotation and the automatic annotation capability of the SAM model. After that, the newly annotated data is used to continue fine-tuning the Grounding DINO model, forming a fully automatic incremental optimization annotation and detection system.

[0090] Automatic marking system for offshore equipment

[0091] The automatic labeling system for offshore equipment of the present invention is a tool based on artificial intelligence and big data technology, which is used to automatically label images, videos or sensor data of offshore engineering equipment. Through deep learning algorithms and computer vision technology, the system can identify and classify offshore equipment and its components, and provide high-precision labeling information for subsequent monitoring, analysis and maintenance work.

[0092] Among them, SAM (SegmentAnything Model) aims to solve the image segmentation problem. Especially in the context of the increasing demand for accurate object segmentation in different application scenarios, the SAM model has demonstrated its wide applicability and flexibility.

[0093] The SAM model introduces a self-attention mechanism into the architecture, allowing the model to focus on the Pay attention and focus on Through this mechanism, the model can capture the differences between different regions in the image. In image segmentation tasks, the features of a certain area may be affected by other areas, so the introduction of this mechanism enhances the model's contextual understanding ability. More importantly, the SAM model uses a large amount of diverse data for pre-training during the training process, covering different scenes, objects, and backgrounds. This not only improves the robustness of the model in various situations, but also enables it to quickly adapt to complex image segmentation tasks without a lot of fine-tuning, see Figure 3 This data-driven approach enables the SAM model to show higher flexibility in practical applications.

[0094] In the process of training the SAM model, a training set containing a diverse image dataset is first constructed. This dataset includes not only labeled images but also unlabeled images so that the model can improve its performance under the framework of self-supervised learning. The model uses a novel loss function design, usually using cross entropy loss combined with contrast loss, to ensure that the model can accurately perform pixel-level segmentation.

[0095] In order to train more efficiently, the SAM model also adopts a dynamic learning rate strategy. This strategy can adjust the learning rate in real time according to the model's training progress and actual performance, ensuring that the model can converge quickly in the early stages of training while remaining stable in the later fine-tuning stages. This flexible learning rate adjustment method enables the SAM model to further improve its convergence speed and final performance.

[0096] The architecture of the SAM model mainly includes the following key components: input processing module, feature extraction network, segmentation mask generation module and post-processing module, such as Figure 4 As shown in Figure 2, the seamless connection between each module enables the model to efficiently perform segmentation tasks and handle complex visual scenes, including:

[0097] 1. Input processing module

[0098] In order to adapt to diverse input data, the SAM model first standardizes the input image, including resizing, color space conversion, and enhancement techniques. These steps are designed to improve the robustness of the model and ensure that the model can maintain stable performance under different conditions.

[0099] 2. Feature extraction network

[0100] Feature extraction is the core of the model. SAM uses a CNN-based backbone network, such as ResNet or EfficientNet. The role of the backbone network is to gradually extract high-level features of the image through multi-layer convolution operations. On this basis, SAM also introduces a self-attention mechanism. The self-attention mechanism can effectively capture the relationship between different areas in the image, thereby improving the expressiveness of the features, especially when dealing with complex backgrounds and fine-grained features, significantly improving the performance of the model.

[0101] 3. Segmentation mask generation module

[0102] After feature extraction, the model sends the feature map to the segmentation mask generation module. The final segmentation mask is generated by a combination of multi-layer perceptron (MLP) and convolutional layers. For each object, contextual information is also considered to improve segmentation accuracy. In addition, this module also applies post-processing algorithms such as conditional random fields (CRF) to further optimize the segmentation results and eliminate potential noise and false segmentations.

[0103] 4. Post-processing module

[0104] To ensure high-quality output, SAM has also designed a powerful post-processing module, which includes both edge smoothing technology and pixel recalibration scheme. The role of the post-processing module is to optimize the segmentation contour based on the preliminary segmentation results using graphics algorithms to enhance the quality of the final mask.

[0105] The automatic labeling system for marine engineering equipment of the present invention also utilizes the Grounding DINO (Dynamic Instance Normalization and Optimization) model. With its advanced characteristics, the Grounding DINO model has gradually become an important tool in tasks such as object detection, segmentation and positioning. Its design concept mainly focuses on improving the robustness and accuracy of the model in complex scenes through dynamic instance normalization and optimization strategies. Traditional target detection models usually face many challenges such as object occlusion, scale changes and background clutter. The Grounding DINO model was proposed to solve these problems.

[0106] The Grounding DINO model is based on the DINO (Self-Distillation with No Labels) architecture, which combines self-supervised learning and dynamic feature extraction methods. The model architecture is shown in Figure 5 By performing multi-scale feature representation on the input image, Grounding DINO can identify and locate different objects in the feature space. The core innovations of Grounding DINO are as follows:

[0107] 1. Dynamic instance normalization

[0108] Unlike traditional normalization methods, DIN can dynamically adjust normalization parameters according to the distribution of current input features. This method not only improves the expressiveness of features, but also effectively alleviates the performance degradation problem caused by batch normalization. When processing salient objects, DIN can specifically enhance certain feature dimensions by considering the feature differences of various objects, thereby improving the model's ability to capture objects.

[0109] 2. Self-supervised learning strategy

[0110] Grounding DINO uses a self-supervised learning strategy during the training process, training with unlabeled data to discover the potential structure in the data. This approach not only reduces the reliance on labeled data, but also improves the generalization ability of the model. In the self-supervised learning stage, the model generates multiple views by performing different enhancements on the image, and uses the consistency between these views to optimize the model's representation ability.

[0111] 3. Multi-scale feature extraction

[0112] Multi-scale feature extraction is an important part of Grounding DINO. The model uses a multi-layer convolutional neural network (CNN) structure to extract features at different levels in order to capture the diversity of objects. In the process of feature extraction, the model not only focuses on the appearance characteristics of the object, but also combines the contextual information of the object. This fusion mechanism enables Grounding DINO to accurately locate the target object in complex scenes.

[0113] In the SAM-Grounding DINO framework of the present invention, users can flexibly click and select the corresponding parts of the image (expert hints), and then these hints are automatically matched with entities in the image. On this basis, the pre-trained object detection model and expert hints can be used as input, and its output results (labels) can be used as input to the grounded SAM, and accurate boxes and masks (such as Figure 6 shown).

[0114] In this way, the entire image can be automatically annotated to realize an automatic annotation system. The automatic annotation system for marine equipment proposed by the present invention can automatically perform category prediction (such as Figure 7 As shown in Figure 2, it provides dense annotations for input images in various scenarios. This greatly reduces the annotation cost and greatly improves the flexibility of image annotation.

[0115] Specifically, SAM is able to generate high-quality target object masks from flexible cues (including points, boxes), and shows strong zero-shot performance on a range of segmentation tasks, further expanding its applicability. Nevertheless, SAM's zero-shot generalization ability in automatic cues may not be sufficient to compete with baseline models, especially in special environments such as ocean scenes.

[0116] Figure 7 It shows that a "data-driven mechanism" has been developed based on the SAM model, which is synchronized with the data set annotation in the model's internal loop, so that the model of the present invention has a strong ability to adapt and generalize to new data sets in specific scenarios. The data engine of the present invention has two stages: manual annotation and automatic annotation.

[0117] 1. Manual annotation stage

[0118] In the manual annotation stage, the annotator uses the SAM model to assist in mask annotation of the target object. This stage is similar to the classic interactive image segmentation task. The annotator uses the tool to perform pixel-level refinement operations on the target object in the image, and the SAM model can assist in generating accurate segmentation masks. The advantage of this stage is that it does not impose too many restrictions on the semantics of the annotated object. Even when faced with non-standardized objects or complex backgrounds, such as blurred or extreme scenes, the system can help automatically abandon these extreme cases and continue to process the next picture, thereby reducing unnecessary time waste in the annotation process.

[0119] However, although SAM provides automatic segmentation capabilities, the granularity of the input cue points can be There may be some ambiguity, so the annotation is further refined under manual supervision. The main task of manual annotation is to ensure the improvement of image quality, especially in the annotation of offshore equipment. The refined annotation lays a solid foundation for the training of subsequent models. Subsequently, the manually annotated data will be used to fine-tune the target detection model Grounding DINO, so that it can gradually learn the characteristics of offshore equipment and show better performance in subsequent detection tasks.

[0120] 2. Automatic annotation stage

[0121] In the automatic annotation stage, the annotation process gradually shifts from manual reliance to full automation. In this stage, the Grounding DINO model that has been fine-tuned in the manual annotation stage is applied to the unlabeled images. By inputting the name of the offshore equipment, the Grounding DINO model can automatically detect the offshore equipment in the image and generate the corresponding detection box.

[0122] Next, the detection results are input into the SAM model together with the unlabeled images. The SAM model further generates accurate segmentation masks for each target based on the detection boxes output by Grounding DINO, thereby completing the fully automatic labeling of the image. This process can efficiently label a large number of images, greatly reducing the cost of manual labeling and improving the labeling consistency of the dataset.

[0123] The core advantage of this data-driven mechanism lies in the cyclic adaptive learning of the model. Through fine-tuning of Grounding DINO in the manual annotation stage, the model gradually learned the characteristics of offshore equipment in specific scenarios and improved its recognition ability in specific fields. In the automatic annotation stage, the model can efficiently apply these features in new data sets, further expanding the scope and efficiency of annotation. At the same time, the introduction of the SAM model greatly improves the accuracy of image segmentation and ensures the accuracy of fully automatic annotation.

[0124] This phased labeling mechanism not only enables the model to show stronger generalization ability in specific scenarios, but also reduces the reliance on human resources through automated processes, providing an effective solution for the rapid labeling of large-scale image datasets. This method has important reference significance for future large-scale data labeling tasks, and also provides technical support for the training of automated data labeling and detection models in specific fields.

[0125] Grounding DINO model fine-tuning

[0126] The results of manual annotation are sent to the Grounding DINO model for fine-tuning to achieve subsequent automated annotation and more accurate detection of marine equipment. To this end, the present invention integrates Grounding DINO as a cross-modal detector into the back of the SAM module. The model structure of Grounding DINO is as follows: Figure 8 It accepts (image, text) pairs as input and outputs multiple object boxes. For example, if there is a pair of scissors on the table in the input image, it will locate the scissors and the table and extract the single The words "scissors" and "table" are used as labels. It is built on the DINO model similar to DETR. When text descriptions appear in images, the model can identify and locate objects in the images.

[0127] The model can understand the linguistic and visual content of an image and can associate visual elements with text or information. The model consists of a Swintransformer to extract visual information from the image and BERT to extract text descriptions. After extracting the fictitious image and fictitious text features, they are fed into the feature enhancer module for cross-modal feature fusion, which uses deformable self-attention to enhance the features.

[0128] To perform feature fusion, the present invention also adds image-to-text cross-attention and text-to-image cross-attention, and designs a language-guided query selection module for selecting features related to the input text.

[0129] In order to fuse the features of text and image modes, the present invention creates a cross-modal decoder. Each cross-modal decoder layer inputs the results of each cross-modal query into a self-attention layer, an image cross-attention layer to merge image features, a text cross-attention layer to merge text features, and a feed-forward network layer. Since text information is injected into the query to improve modality alignment, each decoding layer contains an additional text cross-attention layer compared to the DINO decoding layer.

[0130] During fine-tuning, L1 loss and GIOU loss are used for bounding box regression. In addition, the model follows GLIP and uses contrast loss between predicted objects and language tags for classification. The present invention is tested on the ship image dataset of ShipDataset. The experimental results show that the fine-tuned GroundingDINO model can achieve the best detection results. The detailed results are shown below.

[0131] Ship detection experiment

[0132] The present invention uses the ShipDataset dataset for experimental verification. The ShipDataset ship image dataset is collected by controlling a drone (DJI Phantom 4Pro) over the waters of Shanghai, China. The entire collection process covers scenes under five different shooting angles and lighting conditions.

[0133] Three videos were obtained through drone shooting. The detailed parameters of each video are as follows: resolution 3840x2160, frame rate 23.98 frames / second, flight altitude 500 meters, focal length 24 mm, and all were shot in traffic congestion. These videos contain 3261 frames, 7553 frames, and 7841 frames respectively. Based on these videos, a dataset was constructed, which contains five scenes at different shooting angles to support ship detection and classification. In order to balance the sample distribution in the dataset, the dataset was first manually split into three sets: training, validation, and test sets, with a ratio of 8.5:1:0.5. A total of 18,655 images in the dataset were obtained from a public website.

[0134] Experimental Results

[0135] In order to verify the excellent performance of the proposed method, the present invention is carried out on the ShipDataset dataset. Experiment. The test results are shown in Table 5-1. Compared with other methods, the average accuracy of the method of the present invention is (AP) is the highest, that is, the ship detection performance is the most outstanding. It is worth noting that this method uses the automatic annotation tool to cyclically fine-tune the Grounding DINO model. This operation highlights the efficiency and excellent robustness of the automatic annotation tool and the Grounding DINO model.

[0136] Table 5-1 Comparison of object detection methods on ship dataset

[0137]

[0138] Offshore equipment testing experiment

[0139] The present invention uses a self-developed marine engineering automatic annotation system to automatically annotate data sets from two different sources: one is an actual data set from a ship, and the other is a boom crane data set generated in a simulation environment. The automatic annotation process generates high-quality finely annotated data, which is then used to fine-tune the Grounding DINO model to optimize its performance in marine equipment detection experiments. In order to further improve the generalization ability of the fine-tuned Grounding DINO model, the present invention uses Google image search technology to collect about 1,000 original images related to marine equipment. After strict manual screening, a marine equipment data set for model testing was constructed. This test data set will help evaluate the performance and reliability of the model in practical applications.

[0140] Experimental Results

[0141] The present invention compares the detection accuracy of YOLOv7 and this method on a marine equipment dataset, wherein YOLOv7 uses a pre-trained model and is fine-tuned on an automatically annotated dataset.

[0142] Table 5-2 Comparison of monitoring progress between this method and YOLOv7

[0143]

[0144] Table 5-2 shows the detection accuracy of the two methods. The experimental results show that the method proposed in this invention The ability to achieve better separation accuracy in the open domain demonstrates the robustness of the method.

[0145] In addition, if Fig. 9 As shown in the figure, the method of the present invention can accurately detect the target content according to the user input text in the marine engineering scene. In these images, the text prompts entered by the user are "platform, ship, crane", and the method of the present invention successfully detects and labels the offshore oil and gas platform, ship and crane, indicating that the method can accurately identify and distinguish different types of facilities and equipment in the marine engineering scene.

[0146] The text prompt is "wind turbine". In this scenario, the method effectively detects and marks multiple offshore wind turbines. This shows the method's ability to identify wind turbines in a wind farm environment, and can accurately mark them even in the case of multiple targets and complex backgrounds. The text prompt is "ship, crane". The method successfully identifies and marks large offshore vessels and their crane equipment, demonstrating the detection accuracy of target objects in complex offshore operating environments (including large hulls, cranes, and other devices).

[0147] In summary, the present invention proposes a method for detecting marine equipment targets based on a visual basic model, which facilitates the VP system to quickly and accurately identify and track changes in the real world, and timely reflect these changes in the virtual environment, thereby improving the level of automation and detection accuracy. Through in-depth analysis of the needs of the marine surveillance system and the limitations of the prior art, the present invention proposes a method for automatically labeling and detecting various equipment on the offshore operating system using large-scale visual models, especially SAM and Grounding DINO models. The method first automatically generates high-quality labeled data through the powerful image segmentation capability of the SAM model. Subsequently, the Grounding DINO model is fine-tuned using these data to improve the model's detection accuracy for marine equipment. In addition, by cyclically optimizing the labeling and target detection systems, the method achieves the goal of continuously improving detection performance. Experimental results show that the AP and AR performance of the method on the ShipDataset dataset surpass those of traditional models, especially on the marine equipment dataset, outperforming the YOLOv7 model.

[0148] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0149] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0150] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0151] The storage medium may be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), Erasable Programmable Read-Only Memory (EPROM) Programmable Read-OnlyMemory), Electrically Erasable Programmable Read-Only Memory (EEPROM), Ferromagnetic Random Access Memory (FRAM), Flash Memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memory.

[0152] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0154] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0155] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0156] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product is stored in a A storage medium includes several instructions for causing a computer device (which can It is a personal computer , server, or network device, etc.) to perform all or part of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: a mobile storage device, ROM, RAM, a magnetic disk, or an optical disk, etc., which can store program codes.

[0157] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0158] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0159] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0160] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A vision-based marine equipment identification method, characterized in that: The following steps are involved: S1. Collect image data of the offshore operation site and preprocess it to prepare a dataset for subsequent model training and testing; S2. Use the SAM model to perform image segmentation and automatically generate high-quality target object masks, where the SAM model performs pixel-level analysis on the preprocessed image to identify and segment the offshore equipment in the image and generate the corresponding segmentation mask. In this process, an automatic annotation dataset containing rich offshore equipment images and corresponding masks is constructed; S3. Automatic detection using a Grounding DINO model, which includes automatically detecting the offshore equipment in the image and generating a detection frame using the model based on the text prompt information input by the user, and combining the detection frame result with the segmentation mask generated by the SAM model to refine the detection result; S4. On the automatically annotated dataset constructed by the SAM model, the Grounding DINO model is fine-tuned through cross-modal feature fusion and self-supervised learning strategy to make it better adapt to the offshore operating platform environment, thereby improving the generalization ability and detection accuracy of the model; S5. Loop through the optimization of the annotation and object detection system, in which the fine-tuned Grounding DINO model is replaced back into the automatic annotation system. The automatic annotation system uses the fine-tuned model to automatically generate more accurate detection boxes and segmentation masks to improve the annotation accuracy, and uses the newly annotated data to continue fine-tuning the Grounding DINO model, forming a loop optimization process to continuously improve the detection performance.

2. The vision-based marine equipment identification method according to claim 1, characterized in that: The SAM model includes: The input processing module is used to standardize the input image, involving resizing, color space conversion, and specific enhancement techniques to systematically improve the robustness of the model; Feature extraction network, which uses an advanced CNN-based backbone network (such as ResNet or EfficientNet) to gradually extract high-level features of the image through multi-layer convolution operations, while introducing a self-attention mechanism to improve feature expression capabilities; The segmentation mask generation module combines the multi-layer perceptron (MLP) and convolutional layers to generate the final segmentation mask. This process also considers contextual information to improve segmentation accuracy, and combines post-processing strategies such as conditional random fields (CRF) to optimize the segmentation results. The post-processing module, including edge smoothing technology and pixel recalibration strategy, optimizes the segmentation contour through advanced graphics algorithms to further enhance the quality of the final mask.

3. The method for identifying marine working equipment based on vision according to claim 1 or 2, characterized in that: The SAM model is trained with a diverse dataset of labeled and unlabeled images to enhance the performance of the model under self-supervised learning. A combination of cross entropy loss and contrast loss is used to ensure the accuracy of the model in pixel-level segmentation. A dynamic learning rate adjustment scheme based on training progress is adopted to adjust the learning rate in real time according to the training progress and actual performance of the model, ensuring that the model can converge quickly in the early stage of training while remaining stable in the later fine-tuning stage.

4. The method for identifying marine working equipment based on vision according to any one of claims 1 to 3, characterized in that: The Grounding DINO model is based on the DINO architecture and integrates self-supervised learning and dynamic feature extraction capabilities. It processes input images through multi-scale feature representation to enhance the model's ability to capture details, and uses self-supervised learning to enable the model to be trained on unlabeled data. The model has the ability to dynamically adjust the feature extraction process to adapt to different marine environments and equipment characteristics.

5. The method for identifying marine equipment based on vision according to claim 4, characterized in that: The Grounding DINO model has the following characteristics: Dynamically adjust normalization parameters based on the distribution of input features to improve feature expression and alleviate the performance degradation of batch normalization; During the training process, the model adopts a self-supervised learning strategy, training with unlabeled data to discover the potential structure in the data, thereby reducing dependence on labeled data and improving generalization ability; The model uses a multi-layer convolutional neural network (CNN) architecture to extract features at different levels, capture object diversity, and combine object appearance features with contextual information to achieve accurate positioning of target objects. In the SAM-Grounding DINO framework, users interactively select specific parts of the image as expert prompts, and the model automatically matches entities and generates accurate boxes and masks to achieve fully automatic annotation of the entire image.

6. The method for identifying marine working equipment based on vision according to any one of claims 1 to 5, characterized in that: The operation process of the automatic annotation system includes: Using the automatic segmentation capability of the SAM model, we assist labelers in preliminary mask labeling of target objects, providing basic data for subsequent fully automatic labeling and model training; Use manually annotated data to fine-tune the Grounding DINO model, so that it can better learn the characteristics of offshore equipment and achieve performance improvement in detection tasks; The fine-tuned Grounding DINO model is applied to unlabeled images, which automatically detects offshore equipment and generates detection frames. The detection results and unlabeled images are simultaneously input into the SAM model to generate accurate segmentation masks and complete fully automatic labeling. Through cyclic adaptive learning, the model actively learns the characteristics of offshore engineering equipment in specific scenarios during the manual labeling stage, and applies these characteristics during the automatic labeling stage to expand the scope of labeling and improve the efficiency of labeling.

7. The method for identifying marine working equipment based on vision according to any one of claims 1 to 6, characterized in that: The Grounding DINO model consists of the following components: The text encoder receives the input text and converts it into text features to provide accurate semantic information for the model; Image encoder, which specializes in processing input images, extracting image features, and providing numerical descriptions of image content; The text-image fusion module combines text features with image features and improves feature expression capabilities through cross-modal learning strategies, enabling the model to better understand and associate text and image content; The language-guided index extraction module uses the fused features to accurately extract the image area related to the input text in a language-guided manner to achieve target positioning; The cross-modal decoder combines text features (Key) and image features (Value) to generate accurate detection boxes through the cross-modal decoder, achieving accurate mapping of text to image content.

8. The method for identifying marine working equipment based on vision according to any one of claims 1 to 7, characterized in that: The fine-tuning of the Grounding DINO model includes: Use Swin transformer to efficiently extract visual information from images, extract text descriptions through BERT, and perform cross-modal feature fusion through the feature enhancer module; Add image-to-text cross-attention and text-to-image cross-attention, as well as a language-guided query selection module to enhance feature fusion; Create a cross-modal decoder to optimize the fusion of text and image features to improve modality alignment and feature expression capabilities; During fine-tuning, we use L1 loss with GIOU loss for bounding box regression following GLIP, and classification via contrastive loss between predicted objects and language tags.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 can be implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images