Ship name identification method and system based on cooperation of large and small models
By combining YOLOv8 and the improved DBNet++ model, along with a multimodal large model and database association mechanism, the problem of low accuracy in ship name recognition under complex backgrounds was solved, achieving efficient and accurate ship name recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-13
AI Technical Summary
Existing ship name recognition methods have low accuracy in complex backgrounds, occlusions, overlapping text, or special fonts, making them difficult to adapt to the ever-changing port environment.
The YOLOv8 model is used for ship detection and tracking, and the improved DBNet++ model is used for text localization. Multimodal large model is used for image-text fusion inference, and the ship name recognition results are judged by actual detection boxes, gimbal parameters and image sharpness. A database association mechanism is constructed.
It improves the accuracy and efficiency of ship name recognition, enhances robustness to complex scenarios, reduces invalid calculations, and lowers the rates of false and false recognition.
Smart Images
Figure CN121661626A_ABST
Abstract
Description
Technical Field
[0001] This disclosure belongs to the field of ship name recognition technology, and in particular relates to a ship name recognition method and system based on big-small model collaboration. Background Technology
[0002] With the widespread deployment of intelligent surveillance equipment and the proliferation of high-definition cameras, acquiring high-resolution ship image data in scenarios such as ports and waterways has become much easier. This provides a rich visual information foundation for ship name recognition and promotes the intelligentization of ship identity management. In recent years, computer vision technology has developed rapidly, especially in areas such as object detection, text detection, and image segmentation, achieving significant breakthroughs. These algorithms provide crucial support for ship hull recognition and ship name region localization. For example, object detection algorithms such as YOLO and Faster R-CNN have been widely used in ship hull localization, while text detection algorithms such as EAST and DBNet can be applied to the detection of ship name text regions.
[0003] Existing technologies break down ship name recognition tasks into multiple modules (such as hull detection, text detection, text recognition, and post-processing). While this approach offers a clear structure and flexible implementation, it suffers from strong overall dependency and is prone to error propagation. False detections, missed detections, or positioning errors in any module can lead to final recognition failure. Furthermore, when faced with complex backgrounds, occlusions, overlapping text, multi-line interference, or special fonts (such as handwriting, artistic fonts, or rare characters), the small models lack sufficient recognition and semantic understanding capabilities, resulting in decreased accuracy, poor generalization ability, and difficulty in handling the diverse environments of real ports.
[0004] Therefore, there is an urgent need for a more efficient and accurate ship name identification method to adapt to the complex and ever-changing port environment. Summary of the Invention
[0005] To address the aforementioned issues, this disclosure provides a ship name recognition method and system based on large and small model collaboration. It employs a YOLOv8 model for ship detection and tracking, an improved DBNet++ model for accurate text localization, and a multimodal large model for image-text fusion inference. Combined with an effective frame filtering mechanism that integrates actual detection box information, gimbal parameters, and image clarity, it can efficiently eliminate invalid calculations, accurately identify ship names in complex scenarios, and improve the accuracy of ship name recognition.
[0006] Firstly, this disclosure provides a ship name recognition method based on a size-model collaboration, the method comprising, Key image frames are extracted from surveillance videos based on a preset frequency. The YOLOv8 model is used to detect and track target ships in key image frames, and the actual detection boxes and actual tracking IDs are obtained. Determine whether the target vessel's name has been identified. Based on the determination result, decide not to identify the vessel's name again. Alternatively, determine the vessel name identification result based on the actual information of the actual detection frame, the pan-tilt information of the monitoring camera, and the Laplace variance of the current hull area image. Based on the ship name recognition result, determine whether to stop processing the ship in the current frame, or perform text detection; A target text detection model is constructed. Based on the actual detection box, a small image of the ship hull is obtained. Text detection is performed on the small image of the ship hull based on the target text detection model to obtain the actual text line position coordinates and actual confidence information. The actual text line position coordinates and actual confidence information are embedded into a preset prompt word template to construct a structured question template and form an image-text pair with the ship hull image. The image and text pairs are input into a multimodal large model to perform ship name recognition inference, and the target recognition result is output. The target identification results are matched with the database to update the vessel identification status.
[0007] Furthermore, Determining whether the target vessel's name can be identified specifically includes: The matching result is obtained by matching the actual tracking ID with the ship name identification status table; If a valid ship name record corresponding to the actual tracking ID exists in the ship name recognition status table, then it is determined that the ship name will no longer be recognized. If there is no valid ship name record corresponding to the actual tracking ID in the ship name recognition status table, the ship name recognition result is determined based on the actual information of the actual detection box, the pan-tilt information of the monitoring camera, and the Laplace variance of the current ship hull area image.
[0008] Furthermore, The ship name recognition result is determined based on the actual information of the detection box, the pan-tilt information of the surveillance camera, and the Laplacian variance of the current hull area image. Specifically, it includes: Extract the center point coordinates, width, and height of the actual detection box as the actual information of the actual detection box; The actual horizontal angle, actual vertical angle, and actual zoom level of the surveillance camera are obtained as pan-tilt information. Calculate the Laplacian variance of the current hull region image; The actual information of the actual detection box, the gimbal information, and the Laplace variance are normalized to obtain a normalized feature vector; The normalized feature vector is input into a preset binary classification neural network model to obtain the ship name recognition probability value; The ship name recognition probability value is compared with a preset probability value threshold, and the ship name recognition result is determined based on the comparison result.
[0009] Furthermore, Based on the ship name recognition result, determine whether to stop processing the ship in the current frame, or perform text detection, specifically including: If the ship name recognition probability is greater than the preset probability threshold, then ship name recognition is determined, and the current frame is stopped from processing the ship. If the ship name recognition probability is less than or equal to the preset probability threshold, the ship name is determined to be unrecognizable, and text detection is performed.
[0010] Furthermore, Constructing a target text detection model specifically includes: The classic DBNet++ architecture is structurally extended and trained and optimized to obtain an improved DBNet++ model as a text detector, namely the target text detection model. The improved DBNet++ model includes introducing an auxiliary supervision branch based on 1 / 8 scale feature maps during the basic model training phase, and adding a character center heatmap branch. The auxiliary supervision branch and the character center heatmap branch are only enabled during the training phase and are pruned during the inference phase.
[0011] Furthermore, Text detection is performed on the ship hull thumbnail based on the target text detection model, specifically including: Based on the actual boundary coordinates of the actual detection box, an image region containing the complete hull is cropped from the key image frame as a small hull image. The ship hull image is input into the target text detection model. The model encodes the features of the image through the backbone feature extraction network and outputs the actual probability map, the actual threshold map, and the actual approximate binary map. The actual text region is determined based on the actual approximate binary image; The actual confidence level of each initial text line in the actual text region is calculated based on the actual probability map. The actual confidence level is compared with the preset confidence level, and the actual text line is determined based on the comparison result. Output the actual text line position coordinates and actual confidence information corresponding to the actual text line.
[0012] Furthermore, Construct a structured question template and combine it with a ship hull image to form a text-image pair, specifically including: Generate natural language prompts based on the detected actual text lines; The actual location coordinates are embedded into the prompt text in a preset format; The actual confidence level information is embedded in the prompt words in numerical form; The prompt text is obtained by using a preset prompt template structure; The prompt text is combined with the cropped image of the ship's hull to form an image-text pair input sample.
[0013] Furthermore, The process of performing ship name recognition and reasoning on the input multimodal large model of the image and text pairs specifically includes: The ship hull image is input into a visual encoder to extract the visual features of the entire image and convert them into a Vision Token sequence; Input the image of the text line into a pre-trained CRNN feature extractor to extract character-level features; AlignedCRNN Tokens are generated by aligning the character features extracted by CRNN with visual features through a cross-attention mechanism. Input the structured prompt text into the text encoder to generate a Prompt Token; The Vision Token, Aligned CRNN Token, and Prompt Token are concatenated to form a multimodal input sequence; The multimodal input sequence is fed into the language decoder for joint inference, and the ship name recognition result is output. The output results are post-processed to remove unreasonable character combinations, and the final ship name recognition result is output.
[0014] Furthermore, The target identification results are associated and matched with the database, specifically including: The identified target ship names are matched with ship registration information in the ship information database to obtain matching results; If a match is successful, the vessel name, tracking ID, identification time, and confidence level are written into the structured database, and the tracking ID is marked as identified. If a match fails or some characters are uncertain, the identification record will be retained and marked as pending review.
[0015] Secondly, based on the same inventive concept, this disclosure provides a ship name recognition system based on size model collaboration, the system comprising, The extraction module is used to extract key image frames from surveillance videos based on a preset frequency. The information acquisition module is used to detect and track target ships in key image frames based on the YOLOv8 model, and to obtain the actual detection box and the actual tracking ID. The judgment module is used to determine whether the target vessel's name has been identified. Based on the judgment result, it determines that the vessel name will no longer be identified, or it determines the vessel name recognition result based on the actual information of the actual detection frame, the pan-tilt information of the monitoring camera, and the Laplace variance of the current hull area image. The processing module is used to determine whether to stop processing the ship in the current frame based on the ship name recognition result, or to perform text detection. The text detection module is used to construct a target text detection model, obtain a small image of the ship hull based on the actual detection box, perform text detection on the small image of the ship hull based on the target text detection model, obtain the actual text line position coordinates and actual confidence information, embed the actual text line position coordinates and actual confidence information into a preset prompt word template, construct a structured question template, and form an image-text pair with the ship hull image; The recognition module is used to perform ship name recognition reasoning on the input multimodal large model of the image and text pairs and output the target recognition result; The association module is used to associate and match the target identification results with the database and update the vessel identification status.
[0016] Compared with the prior art, this disclosure has the following advantages: 1. By employing a collaborative mechanism between large and small models, the problem of error propagation between modules in traditional methods is effectively solved. The small model is responsible for rapid screening and initial localization, while the large model focuses on accurate recognition in complex scenarios, forming a complementary collaborative system. The improved DBNet++ model introduced in the text detection stage significantly improves the accuracy of small-sized text detection, especially for artistic fonts and handwritten characters, through 1 / 8 scale feature map-assisted supervision and character center heatmap branching. The cross-attention mechanism adopted in the multimodal inference stage achieves deep fusion of visual features and character features, enabling the system to maintain accurate recognition even in complex scenarios such as occlusion and overlap. The dual-state marking mechanism (identified / pending review) designed in the database association module ensures both recognition efficiency and abnormal situation handling.
[0017] 2. A binary classification judgment mechanism is constructed by using actual detection box information, gimbal parameters, and image sharpness (Laplace variance). Only frames with a ship name recognition probability that meets the standard are processed. At the same time, the identified ships are skipped by tracking ID matching, which greatly reduces invalid calculations and improves the system's operating efficiency.
[0018] 3. By embedding text line positions and confidence scores into prompt word templates and forming image-text pairs with ship images, and combining cross-attention alignment of CRNN character features and visual features, the large model can more accurately understand the text context; at the same time, it links with the database to realize the association matching and state update of recognition results, forming a closed loop and reducing false recognition and missed recognition.
[0019] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart illustrating a ship name recognition method based on size model collaboration according to an embodiment of this disclosure is shown. Figure 2 A schematic diagram of the core algorithm of a ship name recognition module according to an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of an improved DBnet++ text detection structure according to an embodiment of the present disclosure is shown; Figure 4 A schematic diagram of a Qwen2.5-VL-2b structure optimized for ship name recognition according to an embodiment of the present disclosure is shown. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0023] Figure 1 A flowchart illustrating a ship name recognition method based on size model collaboration according to an embodiment of the present disclosure is shown, as follows: Figure 1 As shown, the ship name recognition method based on size model collaboration in this embodiment includes, S1, extracting key image frames from the surveillance video based on a preset frequency; In this embodiment of the disclosure, step S1 specifically includes: Set the time interval to 1 to 5 seconds, and extract key image frames from the monitoring video stream at this time interval through the video frame reading interface; If a motion detection trigger signal exists in the video, when the movement of the ship target is detected, the current frame is extracted in real time as the key image frame to increase the image acquisition density during the period when the ship appears.
[0024] S2, based on the YOLOv8 model, detect and track target ships in key image frames to obtain actual detection boxes and actual tracking IDs; In this embodiment of the disclosure, step S2 specifically includes: YOLOv8s was used as the object detection model, detecting only one category: "ship". The input image size was 1280×1280, demonstrating a good balance between detection speed and accuracy. The training dataset for the model came from a dataset of 10,000 ship object detection images collected and labeled from the China Tower scene. The multi-object tracking algorithm employed BoT-SORT (RobustAssociations Multi-Pedestrian Tracking), assigning stable IDs to achieve consistent target tracking across multiple frames. This module provides spatial target perception capabilities and temporal information assurance for ship name recognition.
[0025] S3, determine whether the target vessel's name has been identified, and determine that the vessel name will no longer be identified based on the determination result, or determine the vessel name identification result based on the actual information of the actual detection frame, the pan-tilt information of the monitoring camera, and the Laplace variance of the current hull area image. In this embodiment of the disclosure, step S3 specifically includes: S31, Match the actual tracking ID with the ship name identification status table to obtain the matching result; S32, if there is a valid ship name record corresponding to the actual tracking ID in the ship name identification status table, then it is determined that the ship name will no longer be identified; S33, if there is no valid ship name record corresponding to the actual tracking ID in the ship name recognition status table, the ship name recognition result is determined based on the actual information of the actual detection box, the pan-tilt information of the monitoring camera, and the Laplace variance of the current ship hull area image.
[0026] In this embodiment of the disclosure, the vessel name identification status table is a structured data table with the actual tracking ID as the unique index, containing fields such as vessel actual tracking ID, vessel name, identification time, identification confidence level, and identification status (e.g., identified, pending confirmation).
[0027] Specifically, each record in the table corresponds to a tracked vessel. When a vessel's name is successfully identified using the multimodal large model, information such as the vessel's actual tracking ID, the identified name, the time of the identification operation, the confidence level of the identification result, and the identification status marked as "identified" is written into the table. Subsequent name identification checks on the vessel only require querying the table using the actual tracking ID. If a valid name record exists (i.e., the identification status is "identified" and the name information is valid), it can be determined that name identification will not be performed on the vessel again, thus avoiding duplicate identification and improving overall process efficiency.
[0028] In this embodiment of the disclosure, step S3 further includes: S34, extract the center point coordinates, width, and height of the actual detection box as the actual information of the actual detection box; S35, obtains the actual horizontal angle, actual vertical angle and actual zoom of the surveillance camera as pan-tilt information; S36, Calculate the Laplacian variance of the current hull region image; S37, normalize the actual information of the actual detection box, the gimbal information and the Laplace variance to obtain a normalized feature vector; S38, Input the normalized feature vector into a preset binary classification neural network model to obtain the ship name recognition probability value; S39, compare the ship name recognition probability value with a preset probability value threshold, and determine the ship name recognition result based on the comparison result.
[0029] Figure 2 A schematic diagram of the core algorithm of a ship name recognition module according to an embodiment of this disclosure is shown. See also: Figure 2 As shown, it is mainly used to determine whether the currently detected ship image has identifiable visual quality and decide whether to enter the ship name recognition process. The core is a multilayer perceptron (MLP) model based on multidimensional feature input. Its task is to predict whether the image of the ship in the current frame meets the recognition requirements. The specific method is as follows: For each detected ship target, 8 manually selected normalized features are extracted as input to the MLP model. These 8 features include: (1) the center point coordinates (cx, cy) of the target detection box; (2) the width and height (w, h) of the detection box; (3) the pan-tilt-zoom (PTZ) information of the monitoring camera: including horizontal angle, vertical angle, zoom factor, a total of 3 items; (4) the Laplacian variance of the current ship body area image (measuring image sharpness). All the above features are normalized to adapt to model training. The normalization formula is as follows:
[0030]
[0031]
[0032]
[0033] In the formula, This represents the coordinates of the center point of the actual detection box, and w represents the width and height of the actual detection box. and The width and height of the image are represented by pan, tilt, and zoom, which represent the PTZ information of the gimbal (horizontal rotation angle, tilt angle, and zoom magnification). and This indicates the maximum and minimum zoom levels, which can be found by looking up the camera model. This means that the Laplacian variance of the image is normalized after taking the natural logarithm, which measures the sharpness of image details and avoids maxima affecting training. Laplacian variance.
[0034] The 8-dimensional input features are fed into the MLP binary classification model, whose structure is: Input layer (8-dimensional) → First hidden layer (64-dimensional) → Second hidden layer (32-dimensional) → Output layer (1-dimensional). The final output is a probability value between 0 and 1, indicating whether the ship's name image possesses identifiable visual quality. When the output value is greater than 0.5, it indicates that the image is clear, the angle is suitable, and the text may be recognizable; the system will then proceed to text detection and recognition. If the value is less than 0.5, the image quality is considered insufficient, possibly due to factors such as excessive distance, blurriness, or poor angle, preventing accurate recognition; the current frame is skipped. It is worth noting that the training data for this MLP model is constructed using a multimodal large model (Qwen2.5-VL-32B). During training, the system uses the multimodal large model to ask historical frame images, "Is the ship's name text in the image clearly visible?" as a criterion for judgment. If the model answers affirmatively (e.g., "clearly visible"), the image is labeled as a positive sample; if the model answers negatively or ambiguously (e.g., "not clearly visible" or "cannot be determined"), it is labeled as a negative sample. This constructs a high-quality training set for image sharpness quality assessment, enabling the final trained MLP model to jointly evaluate image recognizability based on image attributes and device status. This module effectively avoids unnecessary large model inferences on low-quality images, improves the overall system's computational efficiency, and enhances robustness to scenarios with varying lighting, distances, and angles. It is a key element in achieving efficient resource allocation and accuracy assurance in this solution.
[0035] S4, based on the ship name recognition result, determine whether to stop processing the ship in the current frame, or perform text detection; In this embodiment of the disclosure, step S4 specifically includes: S41, when the ship name recognition probability is greater than the preset probability threshold, ship name recognition is determined, and the current frame is stopped from processing the ship. S42, when the ship name recognition probability is less than or equal to the preset probability threshold, it is determined that the ship name cannot be recognized and text detection is performed.
[0036] In this embodiment of the disclosure, the binary classification neural network model outputs a probability value indicating that the ship name is recognizable; if the probability value is greater than a preset threshold (e.g., 0.5), the ship name is determined to be recognizable; if the probability value is less than or equal to the preset threshold, the ship name is determined to be unrecognizable.
[0037] S5, construct a target text detection model, obtain a small image of the ship hull based on the actual detection box, perform text detection on the small image of the ship hull based on the target text detection model, obtain the actual text line position coordinates and actual confidence information, embed the actual text line position coordinates and actual confidence information into a preset prompt word template to construct a structured question template, and form a text-image pair with the ship hull image; In this embodiment of the disclosure, step S5 specifically includes: The classic DBNet++ architecture is structurally extended and trained and optimized to obtain an improved DBNet++ model as a text detector, namely the target text detection model. The improved DBNet++ model includes introducing an auxiliary supervision branch based on 1 / 8 scale feature maps during the basic model training stage, and adding a character center heatmap branch to enhance the ability to perceive the structure of the main axis of the text. The auxiliary supervision branch and the character center heatmap branch are only enabled during the training phase and are pruned during the inference phase.
[0038] In this embodiment, the target text detection model is constructed based on the classic DBNet++ architecture (DBNet++ is an improved version of DBNet (Differentiable Binarization Network, a segmentation-based text detection algorithm that achieves differentiable binarization (DB) by embedding binarization operations into network training; its core architecture retains the original differentiable binarization (DB) module while introducing an adaptive scale fusion (ASF) module). The backbone feature extraction network adopts the multi-scale feature fusion strategy and Adaptive Scale Fusion (ASF) module of DBNet++ to output standard probability maps and threshold maps, and generates approximate binary maps through differentiable binarization (DB) operations, providing a foundation for text region extraction and bounding box formation. Figure 3 A schematic diagram of an improved DBnet++ text detection structure according to an embodiment of this disclosure is shown. See also: Figure 3As shown, to address issues such as uneven character spacing, curve arrangement, and font deformation in ship name recognition scenarios, the model is extended with a dual-branch structure: the introduction of an auxiliary scale supervision branch and character center heatmap modeling based on structural priors, while maintaining a lightweight and non-redundant path in the inference stage. In the backbone feature extraction network, the multi-scale feature fusion strategy and Adaptive Scale Fusion (ASF) module of DBNet++ are adopted, outputting standard probability maps and threshold maps. Approximate binary maps are generated through differentiable binarization (DB) operations for final text region extraction and bounding box formation.
[0039] The first branch introduces an ASF module based on 1 / 8 feature maps in the middle layer of the backbone network, outputting auxiliary probability and threshold maps. Feature maps at this scale contain richer contextual semantic information, making them particularly suitable for compensating for the problem of models easily misinterpreting text as fragmented lines or multiple lines when character spacing is large. During training, an independent supervisory loss is added to the prediction results at the 1 / 8 scale, guiding the model to learn the semantic structure of text regions earlier from the middle layers, thereby improving overall detection robustness and model training convergence speed. It is worth emphasizing that this branch is only enabled during the training phase and is completely pruned during inference, without increasing model computational overhead or latency.
[0040] The second branch adds an independent character structure modeling branch to the 1 / 4 scale main feature map output by ASF. This branch is constructed using a three-layer convolutional module: the first two layers are 3×3 convolutions + BN + ReLU activation, and the last layer is a 1×1 convolution + Sigmoid activation. The output size is B×1×H×W, with a numerical range of [0, 1], used to represent the thermal response of the main axis of the text or the central axis of the character. This thermal map supervision signal is derived from real annotations or rule generation and is used for MSE Loss regression during the training phase. The goal is to guide the model to understand the main structure of the text sequence, especially showing good differentiation and aggregation effects for curved characters, rotated characters, and dense characters. As a lightweight modeling form of structural language prior, this branch does not directly participate in the final detection box generation, but it has a significant effect on improving the overall model's structural perception ability.
[0041] To fully leverage the synergistic training effect of the aforementioned multi-branch structure, this invention designs a composite loss function comprising five sub-items to optimize the collaborative learning objective of the main path and the two auxiliary branches:
[0042] in, The binary classification loss (such as BCE loss) of the probability map of the main branch is used to detect text regions; The loss is the approximate binary map generated from the probability map and threshold map, reflecting the final text region prediction error; This represents the pixel regression loss (such as L1 or MSE) of the threshold map. and The loss represents the probability map and threshold map output of the 1 / 8 scale branch, with the same structure as the main branch, and serves as auxiliary supervision; The regression loss representing the heatmap branch of the character center is defined as:
[0043] When this invention is used for actual training ship name detection: .
[0044] Continue reading Figure 3 As shown, it includes: Image input and backbone feature extraction: The input on the left contains an image of a ship (such as a ship hull image with the ship's name). After passing through multiple convolutions (a series of "square" structures on the left side of the figure, representing convolutional layers), feature extraction is performed. The image size is gradually compressed and semantic features are extracted to provide basic features for subsequent text region detection.
[0045] The Adaptive Scale Fusion (ASF) module is one of the core innovations of the model, used for multi-scale feature fusion. By adaptively fusing feature maps at different levels (such as 1 / 4, 1 / 8 scales, etc.), it can preserve the details of low-level features (such as text edges and small characters) and combine the semantic information of high-level features (such as the overall outline of text regions), thus improving the detection capability of texts such as ship names that may have changes in size and shape.
[0046] Multi-branch output: Probability map: Outputs the probability that each pixel belongs to a text region, used to initially distinguish text from background.
[0047] Threshold map: Provides a threshold for the Differentiable Binarization (DB) operation, helping to generate more accurate binary maps of text regions.
[0048] Approximate binary map: Calculated using a probability map and a threshold map, it is a binary representation of the text region, clearly outlining the approximate range of the text, facilitating subsequent extraction of text outlines and bounding boxes.
[0049] In addition, there are auxiliary branches for smaller scales (such as 1 / 8 scale) to enhance the detection of small characters and sparse text, further improving the completeness of ship name detection.
[0050] Right sub-image: Feature processing details: The processing flow of a certain layer of feature map (such as an input feature map with size 8×256×H×W) is shown. Through operations such as "convolution (Conv) + batch normalization (BatchNorm) + activation function (ReLU)", the features are refined and transformed, and finally output a heat map for text region judgment (the value is between [0,1], representing the confidence that the pixel is a text).
[0051] In this embodiment of the disclosure, step S5 further includes: Based on the actual boundary coordinates of the actual detection box, an image region containing the complete hull is cropped from the key image frame as a small hull image. The ship hull image is input into the target text detection model. The model encodes the features of the image through the backbone feature extraction network and outputs the actual probability map, the actual threshold map, and the actual approximate binary map. The actual text region is determined based on the actual approximate binary image; The actual confidence level of each initial text line in the actual text region is calculated based on the actual probability map. The actual confidence level is compared with the preset confidence level, and the actual text line is determined based on the comparison result. Output the actual text line position coordinates and actual confidence information corresponding to the actual text line.
[0052] In this embodiment of the disclosure, when cropping the small image of the hull, the detection box is extended by 5 to 10 pixels outside the boundary to avoid missing the ship name text on the edge of the hull.
[0053] In this embodiment of the disclosure, a small image of the ship hull is input into the trained target text detection model. The model encodes the features of the small image through the backbone feature extraction network. After the ASF module fuses the multi-scale features, it outputs a probability map, a threshold map, and an approximate binary map.
[0054] In this embodiment of the disclosure, text region extraction is performed based on approximate binary map: connected components are extracted by contour detection algorithm (such as the findContours function in OpenCV), and then the coordinates of the four polygon positions of each text line are determined by minimum bounding polygon fitting algorithm (precisely describing the tilt or curvature of the text line).
[0055] In this embodiment of the disclosure, the confidence level of each text line is calculated based on the probability map: the confidence level is the average value of all probability map pixel values within the text line area, and text lines with a confidence level greater than a preset confidence threshold (such as 0.5) are selected as actual text lines.
[0056] In this embodiment of the disclosure, step S5 further includes: Generate natural language prompts based on the detected actual text lines; The actual location coordinates are embedded into the prompt text in a preset format; The actual confidence level information is embedded in the prompt words in numerical form; The prompt text is obtained by using a preset prompt template structure; The prompt text is combined with the cropped image of the ship's hull to form an image-text pair input sample.
[0057] In this embodiment of the disclosure, the preset prompt word template adopts a fixed format: "The image is a ship with a name, and the image contains multiple text areas. Among them: Text 1: [POS=x1, y1, x2, y2, x3, y3, x4, y4], confidence level=conf1; Text 2: [POS=a1, b1, a2, b2, a3, b3, a4, b4], confidence level=conf2; ... Please identify the ship name in the image", where [POS=...] is used to fill in the four coordinates of the actual text line, and conf is used to fill in the actual confidence level.
[0058] In this embodiment of the disclosure, all actual text lines are traversed, and the four coordinates (x1, y1, x2, y2, x3, y3, x4, y4) of each text line and the corresponding confidence level (retaining two decimal places) are sequentially embedded into the template in descending order of confidence level. If only one text line is detected, only the "Text 1" entry is retained. If multiple text lines are detected, the "Text 1 - Text N" entries are generated.
[0059] S6, perform ship name recognition reasoning on the input multimodal large model of the image and text pair, and output the target recognition result; In this embodiment of the disclosure, step S6 specifically includes: The ship hull image is input into a visual encoder to extract the visual features of the entire image and convert them into a Vision Token sequence; Input the image of the text line into a pre-trained CRNN feature extractor to extract character-level features; AlignedCRNN Tokens are generated by aligning the character features extracted by CRNN with visual features through a cross-attention mechanism. Input the structured prompt text into the text encoder to generate a Prompt Token; The Vision Token, Aligned CRNN Token, and Prompt Token are concatenated to form a multimodal input sequence; The multimodal input sequence is fed into the language decoder for joint inference, and the ship name recognition result is output. The output results are post-processed to remove unreasonable character combinations, and the final ship name recognition result is output.
[0060] In this embodiment of the disclosure, the Vision token is defined as a high-dimensional vector (visual token) that maps an image into patches (such as 224×224 pixel blocks) in a visual model to capture local features and compress information.
[0061] Aligned CRNN tokens are defined in CRNN (Convolutional Neural Network + Recurrent Neural Network) models as follows: image sequences (such as OCR tasks) are converted into sequence tokens, features are extracted through convolution, and then time series are processed by RNN.
[0062] In a language model, a prompt token is defined as a tokenized representation of user input or instructions, used to guide the model in generating specific outputs.
[0063] In this embodiment, during the multimodal ship name recognition stage, a large model architecture based on Qwen2.5-VL-2B is adopted. Furthermore, considering the requirements for character-level visual perception, the input structure and feature fusion method are optimized. The overall process is as follows: Figure 4According to a schematic diagram of a Qwen2.5-VL-2b structure optimized for ship name recognition according to an embodiment of this disclosure, the input of the large model consists of three parts: CRNN-Aligned Token, Vision Token, and Prompt Token. These three are concatenated to form a multimodal fusion input sequence, which is then uniformly input into the language decoder of Qwen2.5-VL for inference. First, the image part is processed by the Vision Encoder, which processes the entire image of the ship hull containing the ship name, extracts the visual features of the entire image to generate a Vision Token, which is used to express the overall semantic information of the image. An additional character perception path is introduced: the single-line text image extracted by the text detection module is fed into the pre-trained CRNN (Convolutional Recurrent Neural Network) feature extractor, and its output features (B×512×1×26) are mapped into 26 token sequences (each a 512-dimensional vector) through a nested mapping structure (CRNNFeature Extractor → Token Mapper). This sequence represents the character-level structural features in the text image. It is then fused and matched with image features extracted by the Vision Encoder through the CrossAttention module (CRNN token as the Query, Vision token as the Key & Value), forming a structurally aligned Aligned CRNN Token. These aligned character-aware tokens, along with the full-image Vision Token and the Prompt Token encoded by the Text Encoder, are concatenated as the input sequence for the multimodal model, uniformly fed into the language decoder to complete the ship name recognition task. This design offers several significant advantages: Firstly, by introducing a CRNN branch, it enhances the modeling ability for character-level information, enabling the multimodal large model to not only perceive the overall image semantics but also accurately capture local text details, compensating for its original shortcomings in character structure perception. Secondly, the CrossAttention mechanism achieves accurate alignment between character features and image semantics, effectively reducing missed recognition and misrecognition issues in scenarios with multi-line text, complex backgrounds, or font variations. Meanwhile, the structured prompt token embeds the coordinates and confidence scores of the text boxes detected by the small model, guiding the model to focus on high-confidence regions, significantly reducing the risk of hallucinations and enhancing recognition stability. Furthermore, this module exhibits excellent pluggability and compatibility; the CRNN branch, as a preprocessing pathway, can be deployed independently without affecting the Qwen2.5-VL backbone architecture, facilitating rapid integration and flexible replacement.In summary, by introducing a pre-trained CRNN character feature and structure alignment mechanism, this invention significantly improves the model's local perception and global semantic fusion capabilities in ship name recognition tasks without disrupting the original multimodal model framework. This results in higher accuracy and stronger robustness in recognition, making it particularly suitable for port video surveillance scenarios with diverse fonts, complex layouts, and dense interference text.
[0064] S7. The target identification result is matched with the database to update the ship identification status.
[0065] In this embodiment of the disclosure, step S7 specifically includes: The identified target ship names are matched with ship registration information in the ship information database to obtain matching results; If a match is successful, the vessel name, tracking ID, identification time, and confidence level are written into the structured database, and the tracking ID is marked as identified. If a match fails or some characters are uncertain, the identification record will be retained and marked as pending review.
[0066] In this embodiment of the disclosure, the ship information database includes information such as the ship's unique identifier (e.g., ship number), registered ship name, company, and ship type. The registered ship name supports precise and fuzzy matching of Chinese and English characters, numbers, and special symbols (e.g., "-" and "_").
[0067] In this embodiment of the disclosure, the edit distance algorithm is used to calculate the similarity between the target recognition result and all registered ship names in the database. The similarity calculation formula is: similarity = 1 - (edit distance / max(length of target recognition result, length of registered ship name)); Set a matching threshold (e.g., 0.85): If there is a registered ship name with a similarity ≥ the matching threshold, the match is considered successful, and the corresponding unique ship identifier is obtained; based on the actual tracking ID, update the ship identification status to "identified" in the ship name identification status table, and record the target identification result, the unique ship identifier, the identification time (accurate to the second), and the confidence of the large model inference; If there is no registered ship name with a similarity greater than or equal to the threshold, or if the target recognition result contains ambiguous characters (such as "*" or "?"), the matching is deemed to have failed. The ship recognition status is marked as "pending confirmation" in the ship name recognition status table. The target recognition result, ship hull thumbnail, and text detection information are retained for manual review by the back-end team.
[0068] Based on the above method, this disclosure also provides a ship name recognition system based on size model collaboration, corresponding to the above method, the system comprising: The extraction module is used to extract key image frames from surveillance videos based on a preset frequency. The information acquisition module is used to detect and track target ships in key image frames based on the YOLOv8 model, and to obtain the actual detection box and the actual tracking ID. The judgment module is used to determine whether the target vessel's name has been identified. Based on the judgment result, it determines that the vessel name will no longer be identified, or it determines the vessel name recognition result based on the actual information of the actual detection frame, the pan-tilt information of the monitoring camera, and the Laplace variance of the current hull area image. The processing module is used to determine whether to stop processing the ship in the current frame based on the ship name recognition result, or to perform text detection. The text detection module is used to construct a target text detection model, obtain a small image of the ship hull based on the actual detection box, perform text detection on the small image of the ship hull based on the target text detection model, obtain the actual text line position coordinates and actual confidence information, embed the actual text line position coordinates and actual confidence information into a preset prompt word template, construct a structured question template, and form an image-text pair with the ship hull image; The recognition module is used to perform ship name recognition reasoning on the input multimodal large model of the image and text pairs and output the target recognition result; The association module is used to associate and match the target identification results with the database and update the vessel identification status.
[0069] Based on the same inventive concept as the above disclosure, this disclosure also provides an electronic device. The electronic device of this disclosure includes at least one processor and at least one memory electrically connected to the processor. The memory is electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.
[0070] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. The indirect connection method can be applied to the embodiments of this disclosure as long as it achieves the purpose of this disclosure.
[0071] Based on the same inventive concept, this disclosure also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the above method.
[0072] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A ship name recognition method based on size model collaboration, characterized in that, The method includes, Key image frames are extracted from surveillance videos based on a preset frequency. The YOLOv8 model is used to detect and track target ships in key image frames, and the actual detection boxes and actual tracking IDs are obtained. Determine whether the target vessel's name has been identified. Based on the determination result, decide not to identify the vessel's name again. Alternatively, determine the vessel name identification result based on the actual information of the actual detection frame, the pan-tilt information of the monitoring camera, and the Laplace variance of the current hull area image. Based on the ship name recognition result, determine whether to stop processing the ship in the current frame, or perform text detection; A target text detection model is constructed. Based on the actual detection box, a small image of the ship hull is obtained. Text detection is performed on the small image of the ship hull based on the target text detection model to obtain the actual text line position coordinates and actual confidence information. The actual text line position coordinates and actual confidence information are embedded into a preset prompt word template to construct a structured question template and form an image-text pair with the ship hull image. The image and text pairs are input into a multimodal large model for ship name recognition and reasoning, and the target recognition result is output. The target identification results are matched with the database to update the vessel identification status.
2. The method according to claim 1, characterized in that, Determining whether the target vessel's name can be identified specifically includes: The matching result is obtained by matching the actual tracking ID with the ship name identification status table. If a valid ship name record corresponding to the actual tracking ID exists in the ship name recognition status table, then it is determined that the ship name will no longer be recognized. If there is no valid ship name record corresponding to the actual tracking ID in the ship name recognition status table, the ship name recognition result is determined based on the actual information of the actual detection box, the pan-tilt information of the monitoring camera, and the Laplace variance of the current ship hull area image.
3. The method according to claim 2, characterized in that, The ship name recognition result is determined based on the actual information of the detection frame, the pan-tilt information of the surveillance camera, and the Laplacian variance of the current hull area image. Specifically, it includes: Extract the center point coordinates, width, and height of the actual detection box as the actual information of the actual detection box; The actual horizontal angle, actual vertical angle, and actual zoom level of the surveillance camera are obtained as pan-tilt information. Calculate the Laplacian variance of the current hull region image; The actual information of the actual detection box, the gimbal information, and the Laplacian variance are normalized to obtain a normalized feature vector; The normalized feature vector is input into a preset binary classification neural network model to obtain the ship name recognition probability value; The ship name recognition probability value is compared with a preset probability value threshold, and the ship name recognition result is determined based on the comparison result.
4. The method according to claim 1, characterized in that, Based on the ship name recognition result, determine whether to stop processing the ship in the current frame, or perform text detection, specifically including: If the ship name recognition probability is greater than the preset probability threshold, then ship name recognition is determined, and the current frame is stopped from processing the ship. If the ship name recognition probability is less than or equal to the preset probability threshold, the ship name is determined to be unrecognizable, and text detection is performed.
5. The method according to claim 1, characterized in that, Constructing a target text detection model specifically includes: The classic DBNet++ architecture is structurally extended and trained and optimized to obtain an improved DBNet++ model as a text detector, namely the target text detection model. The improved DBNet++ model includes introducing an auxiliary supervision branch based on 1 / 8 scale feature maps during the basic model training phase, and adding a character center heatmap branch. The auxiliary supervision branch and the character center heatmap branch are only enabled during the training phase and are pruned during the inference phase.
6. The method according to claim 1, characterized in that, Text detection is performed on the ship hull thumbnail based on the target text detection model, specifically including: Based on the actual boundary coordinates of the actual detection box, an image region containing the complete hull is cropped from the key image frame as a small hull image. The ship hull image is input into the target text detection model. The model encodes the features of the image through the backbone feature extraction network and outputs the actual probability map, the actual threshold map, and the actual approximate binary map. The actual text region is determined based on the actual approximate binary image; The actual confidence level of each initial text line in the actual text region is calculated based on the actual probability map. The actual confidence level is compared with the preset confidence level, and the actual text line is determined based on the comparison result. Output the actual text line position coordinates and actual confidence information corresponding to the actual text line.
7. The method according to claim 1, characterized in that, Construct a structured question template and combine it with a ship hull image to form a text-image pair, specifically including: Generate natural language prompts based on the detected actual text lines; The actual location coordinates are embedded into the prompt text in a preset format; The actual confidence level information is embedded in the prompt words in numerical form; The prompt text is obtained by using a preset prompt template structure; The prompt text is combined with the cropped image of the ship's hull to form an image-text pair input sample.
8. The method according to claim 1, characterized in that, The process of performing ship name recognition and reasoning on the input multimodal large model of the image and text pairs specifically includes: The ship hull image is input into a visual encoder to extract the visual features of the entire image and convert them into a Vision Token sequence; Input the image of the text line into a pre-trained CRNN feature extractor to extract character-level features; Aligned CRNNToken is generated by aligning the character features extracted by CRNN with visual features through a cross-attention mechanism. Input the structured prompt text into the text encoder to generate a Prompt Token; The Vision Token, Aligned CRNN Token, and Prompt Token are concatenated to form a multimodal input sequence; The multimodal input sequence is fed into the language decoder for joint inference, and the ship name recognition result is output. The output results are post-processed to remove unreasonable character combinations, and the final ship name recognition result is output.
9. The method according to claim 1, characterized in that, The target identification results are associated and matched with the database, specifically including: The identified target ship names are matched with ship registration information in the ship information database to obtain matching results; If a match is successful, the vessel name, tracking ID, identification time, and confidence level are written into the structured database, and the tracking ID is marked as identified. If a match fails or some characters are uncertain, the identification record will be retained and marked as pending review.
10. A ship name recognition system based on size-model collaboration, characterized in that, The system includes, The extraction module is used to extract key image frames from surveillance videos based on a preset frequency. The information acquisition module is used to detect and track target ships in key image frames based on the YOLOv8 model, and to obtain the actual detection box and the actual tracking ID. The judgment module is used to determine whether the target vessel's name has been identified. Based on the judgment result, it determines that the vessel name will no longer be identified, or it determines the vessel name recognition result based on the actual information of the actual detection frame, the pan-tilt information of the monitoring camera, and the Laplace variance of the current hull area image. The processing module is used to determine whether to stop processing the ship in the current frame based on the ship name recognition result, or to perform text detection. The text detection module is used to construct a target text detection model, obtain a small image of the ship hull based on the actual detection box, perform text detection on the small image of the ship hull based on the target text detection model, obtain the actual text line position coordinates and actual confidence information, embed the actual text line position coordinates and actual confidence information into a preset prompt word template, construct a structured question template, and form an image-text pair with the ship hull image; The recognition module is used to perform ship name recognition reasoning on the input multimodal large model of the image and text pairs and output the target recognition result; The association module is used to associate and match the target identification results with the database and update the vessel identification status.