Navigation mark detection method and device, storage medium and program product

Abstract

The invention provides a navigation mark detection method, which comprises the following steps: taking a small amount of navigation mark sample images as input, and generating a plurality of navigation mark image data sets containing navigation mark multi-dimensional characterization characteristics through a plurality of data enhancement modes; aiming at different navigation mark detection sub-tasks, taking the navigation mark image data set as input to train corresponding sub-task execution models, and outputting detection results of the corresponding sub-tasks by each sub-task execution model; taking each sub-task execution model as a sub-task execution component of the reference multi-modal large-scale language model, and performing sub-task adaptation fine tuning on the reference multi-modal large-scale language model to obtain a task allocation agent; and inputting a navigation mark detection request and a to-be-detected navigation mark image into the task allocation agent, identifying the sub-task, calling the corresponding sub-task execution model, and outputting a navigation mark detection result. The navigation mark detection accuracy is improved, the memory consumption is reduced, and the method is suitable for edge deployment.

Description

Technical Field

[0001] This invention relates to the field of water transportation, and in particular to a method, apparatus, storage medium, and program product for detecting navigation marks. Background Technology

[0002] As critical aids to navigation (AtoNs), the operational integrity of navigational aids is essential for ensuring maritime safety. These aids are exposed to harsh marine environments for extended periods and are vulnerable to damage from severe weather or collisions with other vessels. Such incidents can impair the functionality of navigational aids, for example, by causing the loss of the top beacon, a core component that transmits specific navigational information.

[0003] In current technologies, research specifically targeting intelligent navigation mark analysis remains scarce, and this field still faces several challenges: First, the scarcity of open-source navigation mark image data results in a lack of relevant corpora for existing visual pre-trained models, leading to a weak task foundation; relying solely on manual data collection and annotation is too costly and lacks scalability. Second, navigation mark analysis is a typical multi-task learning problem, encompassing objectives such as state detection, weather classification, and color recognition; even large models with 7 billion parameters, while performing excellently in single tasks, struggle to maintain both accuracy and robustness in coupled multi-task scenarios. Third, edge devices have limited computing resources, making it impossible to deploy multiple large models or large-scale visual Transformers (ViTs). Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a method, device, storage medium, and program product for navigational aid detection, which effectively resolves the contradiction between high memory consumption and high accuracy requirements in navigational aid detection.

[0005] This invention provides a method for detecting navigational aids, comprising:

[0006] Using a small number of navigation mark sample images as input, several navigation mark image datasets containing multi-dimensional representation features of navigation marks are generated through various data augmentation methods;

[0007] For different navigation mark detection sub-tasks, the navigation mark image dataset is used as input to train the corresponding sub-task execution model, and each sub-task execution model outputs the detection result of the corresponding sub-task;

[0008] Each subtask execution model is used as a subtask execution component of the benchmark multimodal large language model, and the benchmark multimodal large language model is fine-tuned to adapt to the subtasks to obtain the task allocation agent.

[0009] The navigation mark detection request and the image of the navigation mark to be tested are input into the task allocation agent, which identifies the sub-tasks and calls the corresponding sub-task execution model to output the navigation mark detection results.

[0010] In one embodiment of the present invention, the multi-dimensional characterization features of the navigation mark include navigation mark failure morphology features, environmental scene features, and structural state features.

[0011] In one embodiment of the present invention, the step of generating a plurality of navigation mark image datasets containing multi-dimensional representation features of navigation marks through various data augmentation methods includes:

[0012] Simulates a missing top marker fault of a navigation beacon using region editing, including:

[0013] Using an image editing sub-model, the top marker region of the navigation mark in the navigation mark sample image is removed to generate a first sample image with missing top marker;

[0014] The data augmentation method includes a region editing method, and the navigation mark image dataset includes a first sample image.

[0015] In one embodiment of the present invention, the step of generating a plurality of navigation mark image datasets containing multi-dimensional representation features of navigation marks through various data augmentation methods includes:

[0016] Expanding diverse environmental scene samples through generative augmentation, including:

[0017] Using an image generation model, text prompts and navigation mark sample images of different meteorological scenes and / or observation perspectives are input to generate a second sample image containing different meteorological conditions and multiple observation perspectives;

[0018] The data augmentation method includes a generative augmentation method, and the navigation mark image dataset includes second sample images.

[0019] In one embodiment of the present invention, the step of generating a plurality of navigation mark image datasets containing multi-dimensional representation features of navigation marks through various data augmentation methods includes:

[0020] The deformation of the navigational aid structure is simulated using a controllable generation method, including:

[0021] Edge detection is performed on the navigation mark sample image to extract the initial navigation mark structure outline, and the initial navigation mark structure outline is deformed and adjusted according to preset deformation conditions to obtain a new navigation mark structure outline;

[0022] By combining the control network with the stable diffusion model, using the new beacon structure profile as the control signal and the intermediate feature map output by the stable diffusion model encoder as the input, a control map containing spatial deformation constraints is generated through the control network.

[0023] The control chart is input into the stable diffusion model to guide it in generating a third sample image of the beacon structure deformation;

[0024] The generation process is represented as follows:

[0025]

[0026] in, The third sample image of the generated beacon structure deformation is given, where T is the text prompt word, z is the initial noise vector, and C is the control graph output by the control network.

[0027] The data augmentation method includes a controllable generation method, and the navigation mark image dataset includes third sample images.

[0028] In one embodiment of the present invention, the navigation mark detection subtask includes at least a navigation mark fault type detection subtask and a navigation mark fine-grained classification subtask, and the corresponding subtask execution models are as follows:

[0029] A fault detection model is input into a navigation beacon image dataset and outputs navigation beacon fault type results, wherein the fault types include intact top beacon, missing top beacon, bent structure, and missing and bent top beacon.

[0030] A multimodal navigation mark recognition model takes a navigation mark image dataset as input and outputs fine-grained classification results for the navigation marks, including navigation mark type and color feature recognition results.

[0031] In one embodiment of the present invention, the fault detection model adopts a residual network and sets up two independent binary classification tasks: top marker existence judgment and structural bending detection, and configures an independent output head for each task.

[0032] The model is trained by minimizing the sum of the binary cross-entropy losses of the two tasks;

[0033] The fault detection model takes a single navigation mark image as input and outputs two sets of binary classification results: "top mark present or missing" and "structure normal or bent".

[0034] In one embodiment of the present invention, the multimodal navigation mark recognition model adopts a small-scale multimodal large-scale language model. The small-scale multimodal large-scale language model is subjected to small-sample adaptation training to learn the exclusive features of specific types of navigation marks. The trained model is then subjected to low-level quantization processing to obtain the multimodal navigation mark recognition model.

[0035] In one embodiment of the present invention, the step of performing subtask adaptation and fine-tuning on a benchmark multimodal large-scale language model to obtain a task allocation agent includes:

[0036] Low-rank adaptation is used to fine-tune the benchmark multimodal large language model;

[0037] The low-rank fine-tuning multimodal large language model is further fine-tuned using a multi-turn dialogue format to assign subtasks to the corresponding subtask execution models, thus obtaining the task allocation agent.

[0038] In one embodiment of the present invention, the step of fine-tuning the benchmark multimodal large language model using low-rank adaptation includes:

[0039] Freeze the original pre-trained weights of the benchmark multimodal large language model, inject trainable low-rank matrices into the model layers, and the model forward propagation process satisfies the formula:

[0040]

[0041] in, This represents the frozen original pre-trained weights, where B and A are trainable low-rank matrices. Output features for the model. Input images and / or text;

[0042] Model training is achieved by minimizing the standard cross-entropy loss for language modeling tasks, as follows:

[0043]

[0044] in, The loss is the standard cross-entropy loss, where x is the input image and / or text. Let i be the target token at position i. Let be the target token less than position i; θ represents the trainable parameters, which only contain low-rank matrices A and B.

[0045] Another aspect of the present invention provides a navigation mark detection device, comprising:

[0046] The dataset construction module is used to generate several navigation mark image datasets containing multi-dimensional representation features of navigation marks by taking a small number of navigation mark sample images as input and using various data augmentation methods.

[0047] The sub-model training module is used to train the corresponding sub-task execution model for different navigation mark detection sub-tasks, using the navigation mark image dataset as input. Each sub-task execution model outputs the detection results of the corresponding sub-task.

[0048] The agent training module is used to use the sub-task execution model as a sub-task execution component of the benchmark multimodal large language model, and to perform sub-task adaptation and fine-tuning on the benchmark multimodal large language model to obtain the task allocation agent.

[0049] The detection execution module is used to input the navigation mark detection request and the image of the navigation mark to be tested into the task allocation agent, identify the sub-task and call the corresponding sub-task execution model, and output the navigation mark detection result.

[0050] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned beacon detection method.

[0051] In another aspect, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the navigation mark detection method.

[0052] As can be seen from the above solutions, the advantages of the present invention are:

[0053] The navigation mark detection method provided by this invention expands the navigation mark image dataset through various data augmentation methods, solving the problem of scarce navigation mark image data in existing technologies. For different navigation mark detection sub-tasks, corresponding sub-task execution models are configured, and a multimodal large-scale language model is fine-tuned to obtain a task allocation agent. This addresses the shortcomings of existing large models being unsuitable for multi-task scenarios and the limitation of edge devices being unable to deploy multiple large models due to computational resource constraints, thus mitigating the performance degradation problem of a single large model in multi-scale tasks. This invention achieves high accuracy in navigation mark status detection and effectively resolves the contradiction between high memory consumption and high accuracy requirements, and is suitable for practical edge deployment. Attached Figure Description

[0054] Figure 1 A schematic diagram of the overall process of a navigation mark detection method provided in an embodiment of the present invention is shown;

[0055] Figure 2 It shows Figure 1 A detailed flowchart of step S1;

[0056] Figure 3 This demonstrates the effect of removing the top marker of the navigation beacon;

[0057] Figure 4 The effects of weather and enhanced perspective were shown;

[0058] Figure 5 The effect of simulating structural deformation is shown;

[0059] Figure 6 A schematic diagram of the overall structure of a navigation mark detection device provided in an embodiment of the present invention is shown.

[0060] The attached figures are labeled as follows:

[0061] 300: Navigational aid detection device;

[0062] 310: Dataset building module;

[0063] 320: Sub-model training module;

[0064] 330: Agent training module;

[0065] 340: Detection execution module. Detailed Implementation

[0066] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0067] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0068] As mentioned earlier, the scarcity of open-source navigational aid image data leads to a lack of relevant corpora for existing visual pre-trained models, resulting in a weak task foundation. Relying solely on manual data collection and annotation is too costly and lacks scalability. Secondly, navigational aid analysis is a typical multi-task learning problem, encompassing objectives such as state detection, weather classification, and color recognition. Even models with 7 billion parameters, while performing well in single tasks, struggle to maintain accuracy and robustness simultaneously in coupled multi-task scenarios. Furthermore, edge devices have limited computing resources, making it impossible to deploy multiple large models. To address this, this invention expands navigational aid image data through various data augmentation methods; configures dedicated models for different sub-tasks; and fine-tunes a large multimodal language model using a multi-turn dialogue format, enabling it to act as an agent and allocate sub-tasks to smaller, dedicated models. This achieves high accuracy in navigational aid state detection, effectively resolving the contradiction between high memory consumption and high accuracy requirements, and is suitable for practical edge deployment.

[0069] See Figure 1 As shown, Figure 1 A schematic diagram of the overall process of a navigation mark detection method provided in an embodiment of the present invention is shown.

[0070] A method for detecting navigational aids includes the following steps:

[0071] Step S1: Using a small number of navigation mark sample images as input, generate several navigation mark image datasets containing multi-dimensional representation features of navigation marks through various data augmentation methods.

[0072] Step S2: For different navigation mark detection sub-tasks, train the corresponding sub-task execution model with the navigation mark image dataset as input, and each sub-task execution model outputs the detection result of the corresponding sub-task.

[0073] Step S3: Use each subtask execution model as a subtask execution component of the benchmark multimodal large language model, and perform subtask adaptation fine-tuning on the benchmark multimodal large language model to obtain the task allocation agent.

[0074] Step S4: Input the navigation mark detection request and the image of the navigation mark to be tested into the task allocation agent, identify the sub-task and call the corresponding sub-task execution model, and output the navigation mark detection result.

[0075] In this embodiment, the navigation mark image dataset is expanded through various data augmentation methods, addressing the scarcity of navigation mark image data in existing technologies. Corresponding sub-task execution models are configured for different navigation mark detection sub-tasks, and the multimodal large-scale language model is fine-tuned to obtain a task allocation agent. This solves the problems of existing large models being unsuitable for multi-task scenarios and edge devices being limited by computing resources, thus mitigating the performance degradation problem of a single large model in multi-scale tasks. This invention achieves high accuracy in navigation mark status detection and effectively resolves the contradiction between high memory consumption and high accuracy requirements, making it suitable for practical edge deployment.

[0076] In one embodiment, in step S1, a navigation mark image dataset is generated using various data augmentation methods, including navigation mark fault morphology features, environmental scene features, structural state features, and other multi-dimensional representation features of navigation marks.

[0077] See Figure 2 As shown, Figure 2 The diagram shows a specific process flow for step S1.

[0078] Specifically, step S1 includes the following steps:

[0079] Step S11: Simulate the missing top marker fault of the navigation mark by using the area editing method;

[0080] Step S12: Expand the samples of diverse environmental scenarios through generative augmentation;

[0081] Step S13: Simulate the deformation of the navigation mark structure using a controllable generation method.

[0082] By using data augmentation methods such as region editing, generative augmentation, and controllable generation, samples are expanded from different dimensions such as fault morphology, environmental scenarios, and structural states to construct a high-quality navigation mark image dataset.

[0083] One common operational failure of navigation aids is the loss of their top label. However, inputting prompts such as "remove the top label of this navigation aid" into a general large-scale model often yields unsatisfactory results. This is because navigation aids are not common objects in massive pre-training corpora, and unlike ordinary objects, the model lacks the specific contextual understanding required to accurately execute the instruction. To address this, in one embodiment, in step S11, a region editing method is used to simulate the missing top label failure of a navigation aid, thereby achieving precise and targeted modification of the top label. Specifically, an image editing sub-model is used to remove the top label region from the navigation aid sample image, generating a first sample image with a missing top label, wherein the navigation aid image dataset contains the first sample image. In one specific implementation, the image editing sub-model is selected as the "gpt-image-1" model, which excels at modifying user-specified regions in images and can process and generate multimodal content such as text, audio, and images. Of course, in practical applications, the image editing sub-model is not limited to the "gpt-image-1" model.

[0084] Figure 3 This demonstrates the effectiveness of using the "gpt-image-1" model to precisely remove top markers from navigation marks through region-based editing. From left to right, the first and third images are the initial navigation mark sample images; the second and fourth images are the images after removing the top markers from the first and third images, respectively.

[0085] Environmental factors significantly affect the visual characteristics of navigation marks. To address this, in one embodiment, in step S12, a generative augmentation approach is used to expand the sample of diverse environmental scenes by synthesizing images from different weather scenarios and multiple perspectives, thus enriching the feature space of the dataset. Specifically, an image generation model is used, inputting text prompts and navigation mark sample images of different weather scenarios and / or observation perspectives, to generate a second sample image containing different weather conditions and multiple observation perspectives; wherein the navigation mark image dataset includes the second sample image. In one specific implementation, the image generation model selected is the nano-Banana model (formally named gemini-2.5-flash-image), which generates images under diverse weather conditions and observation angles. This model is currently the most advanced model in the field of image generation. Of course, in practical applications, image generation models are not limited to the "nano-Banana" model.

[0086] Figure 4 illustrates the weather and perspective enhancement effects based on Nano-Banana. From left to right, the figures show: a combination of sunny weather and the original perspective; a combination of cloudy weather and an upward-looking perspective; and a combination of rainy weather and an upward-looking perspective.

[0087] To simulate structural deformations such as bending and tilting, in one embodiment, a controllable generation method is used to simulate the deformation of the navigation mark structure in step S13. Specifically, edge detection is performed on the navigation mark sample image to extract the initial navigation mark structure outline, and the initial navigation mark structure outline is deformed and adjusted according to preset deformation conditions (such as bending, tilting, etc.) to obtain a new navigation mark structure outline.

[0088] Then, the ControlNet is combined with the Stable Diffusion model, using the new beacon structure outline as the control signal and the intermediate feature map output by the stable diffusion model encoder as input, to generate a control map containing spatial deformation constraints through the controlNet; and the control map is input to the stable diffusion model to guide it to generate a third sample image of the beacon structure deformation.

[0089] The generation process is represented as follows:

[0090]

[0091] in, The third sample image of the generated beacon structure deformation is given, where T is the text prompt word, z is the initial noise vector, and C is the control graph output by the control network.

[0092] The navigation mark image dataset includes a third sample image.

[0093] In this embodiment, the control network transforms the abstract constraints of navigation beacon structural deformation (such as bending and tilting) into spatial control features that can be recognized by the diffusion model. By injecting these control features into the diffusion generation process, the diffusion model is precisely guided to generate navigation beacon images that conform to the preset structural deformation morphology. This solves the technical problems of traditional generative data augmentation being unable to accurately control structural deformation and generating samples that differ greatly from the real damage scene.

[0094] Figure 5 shows the results of simulating structural deformation using a combination of a control network and a stable diffusion model, and compares the generation effects of image-generated and text-generated images. From left to right in the figure: initial beacon sample image, image-generated image result, and text-generated image result. The image-generated image is generated by combining the initial beacon sample image as model input with the modified structural contour control signal from ControlNet to produce a deformed beacon image. The text-generated image is generated by directly generating a deformed beacon image by taking text describing the beacon's structural deformation as model input (such as "beacon bent" or "beacon tilted").

[0095] In one embodiment, in step S2, for different navigation mark detection sub-tasks, the corresponding sub-task execution model is trained with the navigation mark image dataset as input, and each sub-task execution model outputs the detection result of the corresponding sub-task.

[0096] In a specific implementation, the navigational aid detection subtask includes at least a navigational aid fault type detection subtask and a navigational aid fine-grained classification subtask, with the corresponding subtask execution models as follows:

[0097] A fault detection model is input into a navigation beacon image dataset and outputs navigation beacon fault type results, wherein the fault types include intact top beacon, missing top beacon, bent structure, and missing and bent top beacon.

[0098] A multimodal navigation mark recognition model takes a navigation mark image dataset as input and outputs fine-grained classification results for the navigation marks, including navigation mark type and color feature recognition results.

[0099] The fault detection model employs a residual network. Based on fault types such as intact top marker, missing top marker, structural bending, and missing and bent top marker, it sets up two independent binary classification tasks: top marker presence determination and structural bending detection. The residual network is configured with an independent output head for each task. The model is trained by minimizing the sum of the binary cross-entropy losses of these two tasks. When a single navigation mark image is input, the fault detection model outputs two sets of binary classification results: "top marker present or missing" and "structure normal or bent." Furthermore, it should be noted that the fault detection model is not limited to detecting missing top markers and structural bending; it can also be used to detect faults in other parts of the navigation mark. The applicability of the network structure can be adjusted accordingly, and this invention does not impose specific limitations on this application.

[0100] Navigational aids are typically categorized by color, such as red, yellow, and green. Even with the aforementioned data augmentation, the dataset size for a specific color (e.g., yellow) in the constructed navigational aid image dataset may still be relatively small. Such small datasets are insufficient to train a dedicated small-scale model from scratch and pose a risk of overfitting. Therefore, to achieve fine-grained and accurate navigational aid classification on such small datasets, and considering the computational limitations of edge devices, the multimodal navigational aid recognition model employs a small-scale multimodal large-scale language model. This model undergoes small-sample adaptation training to learn the specific features of a particular navigational aid type (e.g., yellow navigational aids) (such as the cone shape at the top and the yellow-black color scheme). The trained model is then subjected to low-level quantization to obtain the multimodal navigational aid recognition model, which ultimately generates fine-grained classification results for navigational aids, including different types of navigational aids and the features contained within each type.

[0101] In one specific implementation, the small-scale multimodal large-scale language model adopts the "Qwen2.5-3B-Instruct" model. Subsequently, the fine-tuned model undergoes 8-bit quantization, enabling it to learn specific features of specific types of navigation marks from a limited set of samples. Simultaneously, quantization significantly reduces the model's memory footprint and computational requirements, allowing it to be deployed on resource-constrained edge hardware. It is important to note that this multimodal navigation mark recognition model is not limited to processing a small number of samples of a specific color; it can process a small number of samples of any type in the navigation mark image dataset, such as samples from specific weather conditions. This invention is not limited to this.

[0102] In addition, in practice, the navigation mark detection subtask is not limited to the navigation mark fault type detection subtask and the navigation mark fine-grained classification subtask, but also includes tasks such as weather recognition and navigation mark color classification.

[0103] In one embodiment, in step S3, each subtask execution model is used as a subtask execution component of the benchmark multimodal large language model, and the benchmark multimodal large language model is fine-tuned for subtask adaptation to obtain a task allocation agent; wherein, the task allocation agent takes the beacon detection request and the beacon image to be tested as input, identifies the subtask and calls the corresponding subtask execution model, and outputs the beacon detection result.

[0104] In one specific implementation, firstly, low-rank adaptation (LoRA) is used to fine-tune the benchmark multimodal large language model. Specifically, to adapt the multimodal large language model (MLLM) to a specific task, low-rank adaptation (LoRA), an efficient parameter fine-tuning technique, is employed. The original pre-trained weights of the benchmark MLLM are frozen, and trainable low-rank matrices are injected into the model layers, significantly reducing the number of trainable parameters and improving the computational efficiency of fine-tuning. The model's forward propagation process satisfies the following formula:

[0105]

[0106] in, This represents the frozen original pre-trained weights, where B and A are trainable low-rank matrices. Output features for the model. Input images and / or text;

[0107] Model training is achieved by minimizing the standard cross-entropy loss for language modeling tasks, as follows:

[0108]

[0109] in, The loss is the standard cross-entropy loss, where x is the input image and / or text. Let i be the target token at position i. Let be the target token less than position i; θ represents the trainable parameters, which only contain low-rank matrices A and B.

[0110] Furthermore, to improve the accuracy of multimodal large-scale language models in complex, multi-dimensional tasks, an agent-based approach is adopted. Instead of relying on a single model to handle all sub-tasks, the multimodal large-scale language model is trained as an agent to assign sub-tasks to dedicated sub-task execution models. Specifically, a multi-turn dialogue format is used to further fine-tune the low-rank multimodal large-scale language model to assign sub-tasks to corresponding sub-task execution models, resulting in the task-assignment agent. The structure of the multi-turn dialogue training data is designed to teach the model to recognize specific user intents and map them to corresponding sub-task execution components, which can be invoked via function calls. When faced with complex queries, the agent can call the correct sub-task execution model, obtain the detection results of the corresponding sub-tasks output by each sub-task execution model, and then aggregate and integrate these detection results into the final beacon detection result. This process utilizes both the reasoning capabilities of the multimodal large-scale language model and the high accuracy advantage of dedicated expert models.

[0111] The following is a verification of the effectiveness of the method of the present invention.

[0112] Fault detection includes four types of samples: intact beacons, beacons with missing top markers, bent beacons, and beacons with both missing and bent top markers. Each type contains 400 images, covering the original beacon sample images and first, second, and third sample images generated through various data augmentation methods. Table 1 shows the accuracy of ResNet18, ResNet34, ResNet50, ResNet101, ViT-Base16, QwenVL-7B (single-task), and QwenVL-7B (multi-task) models for top marker fault detection. As shown in Table 1, the ResNet series models perform excellently, with ResNet34 effectively adapting to this medium-sized dataset; in contrast, ViT fails to fully utilize its performance due to limited data. The fine-tuned QwenVL-7B-Instruct model shows a significant decrease in accuracy in multi-scale tasks compared to single-task scenarios.

[0113] Table 1. Performance comparison of different models on fault detection tasks

[0114]

[0115] Taking the relatively small size of the yellow navigation mark dataset in the constructed navigation mark image dataset as an example, color jitter and random rotation (within 30 degrees) were applied to some images. A balanced dataset of 250 images (50 images for each of the five classes: dedicated navigation marks, northern navigation marks, southern navigation marks, western navigation marks, and eastern navigation marks) was constructed through undersampling. The core distinguishing features of this class of navigation marks are the cone shape at the top and the yellow-black color scheme. Table 2 shows the accuracy of QwenVL-3B (zero-shot), QwenVL-3B (5-shot), and QwenVL-3B (10-shot) models as multimodal navigation mark recognition models for fine-grained navigation mark classification. As shown in Table 2, after providing classification rules, the accuracy of QwenVL-3B (5-shot) (1 example per class) reaches 97.2%, and the accuracy of QwenVL-3B (10-shot) (2 examples per class) reaches 100%. Importantly, the model accuracy did not decrease after quantization, confirming its suitability as a dedicated tool for intelligent agents. Experiments demonstrate that using the QwenVL-3B multimodal large-scale language model with 3 billion parameters for LoRA fine-tuning can effectively mitigate the risks of overfitting or training failure caused by small datasets.

[0116] Table 2. Few-sample performance of yellow buoy classification

[0117]

[0118] To further evaluate the effectiveness of the task-assignment agent, ablation experiments were conducted to measure overall accuracy (all subtasks being completed correctly was considered a success). Table 3 shows the ablation experiments for overall accuracy. As shown in Table 3, the complete agent achieved an overall accuracy of 81.6%; however, the agent's performance significantly decreased after omitting some subtask execution models: the baseline multimodal large language model without any subtask execution models achieved an accuracy of only 45.4%, while the accuracy was 57.0% and 75.0% respectively when only the multimodal beacon recognition model or fault detection model was used. This result highlights the necessity of the function call mechanism. Furthermore, when the complete agent was running, the individual accuracy of each subtask exceeded 90%.

[0119] Table 3 Ablation Experiment Overall Accuracy

[0120]

[0121] In summary, this invention expands a small number of navigation mark samples through various data augmentation methods, constructing a high-quality navigation mark image dataset and addressing the scarcity of navigation mark image data in existing technologies. For different navigation mark detection sub-tasks, corresponding sub-task execution models are configured, and the multimodal large language model is fine-tuned using low-rank and multi-turn dialogue to obtain a task allocation agent. This solves the problems of existing large models being unsuitable for multi-task scenarios and the inability to deploy multiple large models on edge devices due to computational resource limitations, thus mitigating the performance degradation problem of a single large model in multi-scale tasks. This invention achieves high accuracy in navigation mark state detection, with each sub-task model exceeding 90% accuracy and an overall accuracy of 81%. Furthermore, it effectively resolves the contradiction between high memory consumption and high accuracy requirements and is suitable for practical edge deployment.

[0122] In one embodiment, a navigational aid detection device is further provided. This embodiment is a device embodiment corresponding to the above method embodiment, such as... Figure 6 As shown, Figure 6 A schematic diagram of a navigation mark detection device according to an embodiment of the present invention is shown. This device embodiment can be implemented in conjunction with the above-described method embodiment. The relevant technical details mentioned in the above method embodiment remain valid in this device embodiment, and will not be repeated here to avoid repetition.

[0123] A navigational aid detection device 300, comprising:

[0124] The dataset construction module 310 is used to generate several navigation mark image datasets containing multi-dimensional representation features of navigation marks by taking a small number of navigation mark sample images as input and using various data augmentation methods.

[0125] The sub-model training module 320 is used to train the corresponding sub-task execution model for different navigation mark detection sub-tasks, using the navigation mark image dataset as input. Each sub-task execution model outputs the detection results of the corresponding sub-task.

[0126] The agent training module 330 is used to use each sub-task execution model as a sub-task execution component of the benchmark multimodal large language model, and to perform sub-task adaptation fine-tuning on the benchmark multimodal large language model to obtain the task allocation agent.

[0127] The detection execution module 340 is used to input the navigation mark detection request and the image of the navigation mark to be tested into the task allocation agent, identify the sub-task and call the corresponding sub-task execution model, and output the navigation mark detection result.

[0128] It should be noted that the relevant technical details mentioned in the above embodiments are still valid in this device embodiment, and will not be repeated here to avoid repetition.

[0129] This invention also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the above-described navigation mark detection method and achieve the same technical effect.

[0130] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described navigation mark detection method and achieves the same technical effect.

[0131] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for detecting navigational aids, characterized in that, include: Using a small number of navigation mark sample images as input, several navigation mark image datasets containing multi-dimensional representation features of navigation marks are generated through various data augmentation methods; For different navigation mark detection sub-tasks, the navigation mark image dataset is used as input to train the corresponding sub-task execution model, and each sub-task execution model outputs the detection result of the corresponding sub-task; Each subtask execution model is used as a subtask execution component of the benchmark multimodal large language model, and the benchmark multimodal large language model is fine-tuned to adapt to the subtasks to obtain the task allocation agent. The navigation mark detection request and the image of the navigation mark to be tested are input into the task allocation agent, which identifies the sub-tasks and calls the corresponding sub-task execution model to output the navigation mark detection results.

2. The method according to claim 1, characterized in that, The multi-dimensional characterization features of navigation marks include navigation mark failure morphology features, environmental scene features, and structural state features.

3. The method according to claim 2, characterized in that, The process involves generating several navigation mark image datasets containing multi-dimensional representational features of navigation marks through various data augmentation methods, including: Simulates a missing top marker fault of a navigation beacon using region editing, including: Using an image editing sub-model, the top marker region of the navigation mark in the navigation mark sample image is removed to generate a first sample image with missing top marker; The data augmentation method includes a region editing method, and the navigation mark image dataset includes a first sample image.

4. The method according to claim 2, characterized in that, The process involves generating several navigation mark image datasets containing multi-dimensional representational features of navigation marks through various data augmentation methods, including: Expanding diverse environmental scene samples through generative augmentation, including: Using an image generation model, text prompts and navigation mark sample images of different meteorological scenes and / or observation perspectives are input to generate a second sample image containing different meteorological conditions and multiple observation perspectives; The data augmentation method includes a generative augmentation method, and the navigation mark image dataset includes second sample images.

5. The method according to claim 2, characterized in that, The process involves generating several navigation mark image datasets containing multi-dimensional representational features of navigation marks through various data augmentation methods, including: The deformation of the navigational aid structure is simulated using a controllable generation method, including: Edge detection is performed on the navigation mark sample image to extract the initial navigation mark structure outline, and the initial navigation mark structure outline is deformed and adjusted according to preset deformation conditions to obtain a new navigation mark structure outline; By combining the control network with the stable diffusion model, using the new beacon structure profile as the control signal and the intermediate feature map output by the stable diffusion model encoder as the input, a control map containing spatial deformation constraints is generated through the control network. The control chart is input into the stable diffusion model to guide it in generating a third sample image of the beacon structure deformation; The generation process is represented as follows: in, The third sample image of the generated beacon structure deformation is given, where T is the text prompt word, z is the initial noise vector, and C is the control graph output by the control network. The data augmentation method includes a controllable generation method, and the navigation mark image dataset includes third sample images.

6. The method according to claim 1, characterized in that, The navigation aid detection subtask includes at least a navigation aid fault type detection subtask and a navigation aid fine-grained classification subtask, and the corresponding subtask execution models are as follows: A fault detection model is input into a navigation beacon image dataset and outputs navigation beacon fault type results, wherein the fault types include intact top beacon, missing top beacon, bent structure, and missing and bent top beacon. A multimodal navigation mark recognition model takes a navigation mark image dataset as input and outputs fine-grained classification results for the navigation marks, including navigation mark type and color feature recognition results.

7. The method according to claim 6, characterized in that, The fault detection model uses a residual network and sets up two independent binary classification tasks: top marker existence judgment and structural bending detection. Each task is configured with an independent output head. The model is trained by minimizing the sum of the binary cross-entropy losses of the two tasks; The fault detection model takes a single navigation mark image as input and outputs two sets of binary classification results: "top mark present or missing" and "structure normal or bent".

8. The method according to claim 6, characterized in that, The multimodal navigation mark recognition model adopts a small-scale multimodal large-scale language model. It performs small-sample adaptation training on the small-scale multimodal large-scale language model to enable it to learn the exclusive features of specific types of navigation marks. The trained model is then subjected to low-level quantization to obtain the multimodal navigation mark recognition model.

9. The method according to claim 1, characterized in that, The subtask adaptation and fine-tuning of the benchmark multimodal large-scale language model to obtain the task allocation agent includes: Low-rank adaptation is used to fine-tune the benchmark multimodal large language model; The low-rank fine-tuning multimodal large language model is further fine-tuned using a multi-turn dialogue format to assign subtasks to the corresponding subtask execution models, thus obtaining the task allocation agent.

10. The method according to claim 9, characterized in that, The step of fine-tuning the benchmark multimodal large language model using low-rank adaptation includes: Freeze the original pre-trained weights of the benchmark multimodal large language model, inject trainable low-rank matrices into the model layers, and the model forward propagation process satisfies the formula: in, This represents the frozen original pre-trained weights, where B and A are trainable low-rank matrices. Output features for the model. Input images and / or text; Model training is achieved by minimizing the standard cross-entropy loss for language modeling tasks, as follows: in, The loss is the standard cross-entropy loss, where x is the input image and / or text. Let i be the target token at position i. Let be the target token less than position i; θ represents the trainable parameters, which only contain low-rank matrices A and B.

11. A navigational aid detection device, characterized in that, include: The dataset construction module is used to generate several navigation mark image datasets containing multi-dimensional representation features of navigation marks by taking a small number of navigation mark sample images as input and using various data augmentation methods. The sub-model training module is used to train the corresponding sub-task execution model for different navigation mark detection sub-tasks, using the navigation mark image dataset as input. Each sub-task execution model outputs the detection results of the corresponding sub-task. The agent training module is used to use the sub-task execution model as a sub-task execution component of the benchmark multimodal large language model, and to perform sub-task adaptation and fine-tuning on the benchmark multimodal large language model to obtain the task allocation agent. The detection execution module is used to input the navigation mark detection request and the image of the navigation mark to be tested into the task allocation agent, identify the sub-task and call the corresponding sub-task execution model, and output the navigation mark detection result.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images