Navigation method and system for offshore wind power booster station based on multi-modal information interaction

Abstract

This invention discloses a navigation method and system for offshore wind power booster stations based on multimodal information interaction, belonging to the field of artificial intelligence and robot navigation technology. It achieves visual-language modality alignment through the CLIP model, constructing an end-to-end navigation parameter generation model. This model can adaptively generate AGV motion parameters based on natural language commands and scene images, enabling efficient and accurate inspection of offshore wind power booster stations. By establishing a multimodal dataset and associating language commands with AGV motion parameters, this invention solves the error accumulation problem caused by multi-stage processing in traditional methods, improving navigation accuracy and adaptability. Furthermore, this invention employs a multi-task loss function optimization model, including contrastive loss and motion parameter regression loss, enabling the AGV to adaptively adjust motion parameters according to environmental changes, thus improving the efficiency and safety of offshore wind power booster station inspections.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robot navigation technology, specifically relating to a navigation method and system for offshore wind power booster stations based on multimodal information interaction. Background Technology

[0002] Offshore wind turbine booster stations are critical facilities in wind farms, requiring regular inspections of instruments and equipment to ensure operational safety. Due to the high salt spray and humidity environment at sea, offshore wind turbine booster stations need to be relatively enclosed to prevent salt and moisture damage. Simultaneously, to withstand wind, waves, and erosion, the pile foundations, jacket structures, and other components need to be compact and modularized within a limited platform volume. Furthermore, considering the reliability of offshore wind turbine booster stations, high redundancy and low maintenance are required, posing significant challenges to inspection navigation. Traditional inspection methods often employ point-to-point navigation, where AGVs (Automated Guided Vehicles) move along a pre-set fixed path to a designated location. This method has the following drawbacks:

[0003] 1. Error accumulation caused by multi-stage processing: Existing methods usually decompose navigation tasks into multiple stages, such as target recognition and path planning, which leads to the accumulation of errors between stages.

[0004] 2. Reliance on manual programming: Each time the inspection target changes, the route needs to be replanned, which is inefficient and has weak language command comprehension ability: Existing methods have difficulty effectively understanding natural language commands and cannot perform accurate navigation based on user commands.

[0005] 3. Poor environmental adaptability: It is difficult to cope with the complex and ever-changing internal structure and instrument layout of the booster station.

[0006] In recent years, multimodal fusion technologies (such as vision-language models) have made progress in robot perception and decision-making. For example, models like CLIP can achieve cross-modal alignment of images and language for object localization and grasping tasks. However, existing technologies have not yet been applied to the field of AGV navigation, especially to achieve language-guided adaptive inspection in complex environments such as offshore wind power booster stations. Summary of the Invention

[0007] To address the shortcomings of existing technologies and achieve the goals of improving the accuracy of navigation parameter generation, reducing information processing efficiency, and enhancing environmental adaptability, this invention adopts the following technical solution:

[0008] The navigation method for offshore wind power booster stations based on multimodal information interaction includes the following steps:

[0009] Acquire images and commands for offshore wind power booster station scenarios;

[0010] A contrastive learning model is used to extract features from images and instructions based on similarity, resulting in image features and instruction text features that are mapped to a unified semantic space.

[0011] Multimodal feature fusion is performed on the mapped image features and the instruction text features to obtain the fused multimodal features;

[0012] A motion parameter generator is used to obtain multimodal features and inject the instruction text features obtained through position encoding into the attention mechanism of the decoder. This allows the decoder to capture the internal correlation of multimodal features through a self-attention mechanism. Through the interaction between the cross-attention mechanism and the instruction text features, the relative structure of the text symbol sequence in the semantic space affects the degree of attention the decoder pays to different multimodal feature regions, so as to generate semantically guided motion parameters in different station areas in a differentiated manner.

[0013] Construct a contrastive loss for the contrastive learning model and a motion parameter regression loss for the decoder, and train them together. The contrastive loss maximizes the similarity between the matching image features and the instruction text features, and the motion parameter regression loss minimizes the error between the predicted motion parameters and the true values ​​of the motion parameters.

[0014] Based on the trained contrastive learning model and the decoder, motion parameters are generated for the current offshore wind power booster station scenario to navigate the inspection task.

[0015] Furthermore, in the joint training, a dynamic weight adjuster is used to optimize the contrastive loss L. contrastive Regression loss L with motion parameters regression The synergistic effect of the dynamic weight adjuster dynamically adjusts the weight coefficients of the contrastive loss and the motion parameter regression loss by evaluating the instruction resolvability and environmental complexity of the current training sample in real time. For samples with low instruction resolvability, the weight of the contrastive loss is increased to improve semantic alignment capability, and for samples with high environmental complexity, the weight of the regression loss is increased to enhance the accuracy of motion parameter generation.

[0016] Furthermore, the dynamic weight adjuster is implemented by introducing an attention gating network;

[0017] Calculate the semantic ambiguity of the input sample, i.e., the entropy value based on the language instruction, and the environmental complexity, i.e., quantified by the texture entropy and obstacle density of the scene image;

[0018] By normalizing the data, the weights of semantic ambiguity and environmental complexity in the sum of the two are calculated separately, and the contrast loss weights and motion parameter regression loss weights are obtained respectively.

[0019] An exponential decay term based on the decay coefficient and training rounds is introduced to balance the optimization priorities in the early and later stages of training.

[0020] The total loss function is obtained by multiplying the value of the contrastive loss function by the contrastive loss weight and the exponential decay term, and by multiplying the value of the motion parameter regression loss function by the difference between the motion parameter regression loss weight and the exponential decay term.

[0021] Furthermore, the multimodal feature fusion is achieved by using the top-down and lateral connection interaction mechanism of the feature pyramid network to jointly model the extracted image features and instruction text features in the spatial and semantic dimensions to generate fused multimodal features. The top-down instruction text features provide global semantic guidance for the image features, while the lateral connection image features retain local details.

[0022] Furthermore, physical constraints are constructed for the motion parameters. If the motion parameters exceed the constraint range, they are clipped to the boundary value. If an obstacle is encountered during the motion, an adjustment coefficient is calculated based on the distance to the obstacle and the maximum safe distance to adjust the motion parameters.

[0023] Furthermore, to avoid drastic fluctuations in motion parameters that could cause motion trajectory jitter, an exponential smoothing algorithm is introduced to smooth the motion parameters of consecutive frames. Based on the smoothing coefficient, weights are constructed for the motion parameters of the previous moment and the adjusted motion parameters to generate stable motion parameters, which can significantly improve navigation stability in complex marine environments (such as wave disturbances or equipment obstruction).

[0024] Furthermore, the extraction of image features involves using a visual encoder to segment the image into a set of image blocks and performing linear projection to convert them into serialized tokens. Then, a multi-layer encoder is used to extract global semantic features to obtain image features that include local image texture and overall scene semantic structure.

[0025] The extraction of the instruction text features involves using a text encoder to segment and position-encode the instructions, adding position embeddings to each token to preserve the sequential information of the language sequence, and then extracting context-related instruction text features through a multi-layer encoder.

[0026] Furthermore, the image includes an RGB image sequence and a depth map. The RGB images include, but are not limited to, covering the main transformer, GIS bay, high-voltage switch, step-up inverter, refrigeration equipment, fire-fighting equipment, and cabin passageway, and spatial geometric information is obtained by collecting point cloud data.

[0027] The instruction text data is a natural language instruction;

[0028] The motion parameters include linear velocity, angular velocity, and acceleration.

[0029] Furthermore, scene structure constraint sampling is performed on the offshore wind power booster station, including path decoupling sampling, obstacle proximity sampling, and language-trajectory alignment sampling;

[0030] Based on multi-layer topology graph-based path decoupling sampling, the offshore wind power booster station is divided into equipment block nodes, and multi-style inspection paths are sampled from the topology graph.

[0031] Obstacle proximity sampling enhances the sampling of equipment edges, pipe bottoms, and narrow corridors to improve the model's sensitivity to boundary safety.

[0032] Language-track alignment sampling automatically generates a corresponding instruction template for each track segment.

[0033] The offshore wind power booster station navigation system based on multimodal information interaction includes an image acquisition module, a command input module, a multimodal fusion module, a motion parameter generation module, and a guidance control module. It adopts the aforementioned offshore wind power booster station navigation method based on multimodal information interaction, and sequentially performs image and command acquisition, fusion of image features and command text features, and generation of motion parameters based on fused features for guidance control.

[0034] The advantages and beneficial effects of this invention are as follows:

[0035] This invention constructs an end-to-end multimodal model capable of robust semantic understanding and motion parameter generation in complex marine environments. Utilizing the visual-language alignment capability of the CLIP model, it achieves deep fusion of images and language, providing high-quality feature representations with both global semantics and local details for subsequent motion parameter generation. The end-to-end model avoids error accumulation caused by traditional multi-stage processing, achieving direct mapping from the joint input of natural language commands and scene images to AGV motion parameters. Simultaneously, to ensure high reliability and strong generalization ability of the model in real-world booster station scenarios, this invention also designs a multi-stage training process, scene enhancement strategy, cross-modal optimization mechanism, and safety constraint learning method tailored to the characteristics of offshore wind power booster stations. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the system structure according to an embodiment of the present invention.

[0037] Figure 2 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0038] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0039] This invention proposes a navigation system for offshore wind power booster stations based on multimodal information interaction. It innovatively introduces a CLIP (Contrastive Language-Image Pretraining) model as the core component for multimodal feature alignment. This model fuses the semantic features of natural language commands with the visual features of scene images across modalities, generating multimodal feature vectors with environmental awareness. Based on this, an improved Transformer decoder architecture is used, combined with a dual-task optimization strategy of contrastive learning and motion parameter regression, to generate a navigation system that includes linear velocity... angular velocity and acceleration AGV motion parameter vector ,like Figure 1 As shown, the system includes an image acquisition module, a language command input module, a multimodal fusion module, a motion parameter generation module, and an AGV control module.

[0040] Image acquisition module: used to acquire RGB-D images of offshore wind power booster station scenes, including RGB image I∈R{H×W×3} and depth image D∈R{H×W}.

[0041] Language instruction input module: used to receive and process natural language instructions T∈V, where V is the language vocabulary space.

[0042] Multimodal fusion module: By constructing a multimodal fusion model, image and language information are fused to generate fused features of AGV motion parameters.

[0043] Specifically, this invention employs the CLIP model as the core framework for multimodal feature extraction. Its core idea is to establish a shared semantic space for visual-linguistic modalities through large-scale image-text pair pre-training. For the input image... and natural language commands The CLIP model extracts its corresponding feature representations through two independent but co-optimized encoders.

[0044]

[0045]

[0046] in, and These are CLIP's visual encoder and text encoder, respectively. and These are the dimensions of visual and textual features, respectively.

[0047] Visual encoder Employing the Vision Transformer (ViT) architecture based on Transformer, the input image... Divide the image into fixed-size blocks (e.g.) (pixels), and convert them into serialized tokens through linear projection. Then, they are processed by a multi-layer Transformer encoder (typically containing...). The visual encoder extracts global semantic features using a self-attention layer. Finally, the visual encoder outputs a fixed-dimensional feature vector. This feature vector not only contains local texture information of the image, but also captures the overall semantic structure of the scene, such as key features like device layout and ambient lighting conditions, through the global attention mechanism of the Transformer.

[0048] Text encoder Employing a Transformer-based architecture (such as BERT or RoBERTa) to process input natural language instructions The process involves word segmentation and embedding. The specific steps include:

[0049] 1) Word segmentation: This refers to the instruction... It can be broken down into subword units, for example, by using the Byte Pair Encoding (BPE) algorithm to generate a token sequence;

[0050] 2) Positional encoding: Add positional embedding to each token to preserve the order information of the language sequence;

[0051] 3) Semantic modeling: Extract context-related semantic features through a multi-layer Transformer encoder.

[0052] Finally, the text encoder outputs a feature vector. ,in The dimensions of text features (usually with) Same, for example This feature vector can accurately express the semantic intent of the instruction, such as task-related instructions like "go to the main transformer" or "go around the obstacle".

[0053] The CLIP model uses a contrastive learning strategy to incorporate visual features. and text features This is mapped to a unified semantic space. The specific training objective is to maximize the similarity of matching image-text pairs while minimizing the similarity of non-matching pairs. This process is achieved through a contrastive loss function. The driving force, its mathematical expression is:

[0054]

[0055] in, This represents cosine similarity. This is a temperature parameter used to control the smoothness of the distribution. The loss function, which minimizes the number of negative samples, enhances the model's understanding of natural language instructions by maximizing the similarity of matched image-text pairs. Through this mechanism, the CLIP model achieves deep alignment of visual and language modalities, enabling the integration of image features... Text features They are comparable in a shared semantic space.

[0056] After CLIP pre-training, visual features and text features It can align to the same semantic space; for example, the features of an image of "wind turbine" are closer in spatial distance to the textual features of "wind turbine." Simultaneously, the CLIP model is pre-trained on a large scale of image-text pairs (e.g., 400 million pairs), endowing the encoder with powerful cross-domain generalization capabilities, enabling it to adapt to the feature extraction needs of special scenarios such as offshore wind power booster stations. In this invention, the visual encoder... and text encoder The parameters can be frozen to preserve the pre-trained semantic structure, or fine-tuned according to the specific task to enhance scene adaptability.

[0057] Through the above mechanism, the CLIP model provides high-quality visual-linguistic feature representations for subsequent multimodal fusion and motion parameter generation, laying the foundation for adaptive navigation of AGVs.

[0058] To achieve deep fusion of visual and linguistic features, this invention designs a multimodal fusion layer based on a Feature Pyramid Network (FPN). This fusion layer integrates visual features through a top-down and lateral connection interaction mechanism. Text features Joint modeling is performed in both spatial and semantic dimensions to generate fused multimodal features. ,in This represents the spatial resolution of the feature map (usually determined by the downsampling operation of the CLIP visual encoder). This represents the number of feature channels after fusion.

[0059] Traditional FPN is mainly used for multi-scale feature fusion in convolutional neural networks (such as the FPN architecture in object detection). Its core idea is to upsample high-level semantic features to low-level resolution through a top-down path and concatenate them with low-level features through lateral connections, thereby preserving rich semantic and detailed information at different scales. In this invention, the FPN structure is extended into a cross-modal feature interaction framework, and the specific implementation process is as follows:

[0060] First, the input features are aligned by upsampling and channel adjustment to ensure that the visual-linguistic features are strictly aligned in the spatial dimension, thus avoiding feature misalignment caused by modal differences.

[0061] Specifically, visual features (shape is) ) and text features (shape is) To align in the spatial dimension, since text features are global semantic vectors, they need to be upsampled to the same spatial resolution as visual features. Implemented using deconvolution:

[0062]

[0063] Secondly, channel dimension adjustment is required. To achieve cross-modal feature fusion, visual features need to be adjusted. Compared with upsampled text features The number of channels is unified, and the channels of both are mapped using a 1×1 convolution:

[0064]

[0065] Then, the top-down and lateral connectivity fusion mechanism is implemented. Top-down text features provide global semantic guidance for visual features (such as semantic constraints for the "bypass obstacles" instruction), while lateral connectivity visual features preserve local details (such as device outlines and occluded areas). The top-down path and lateral connectivity of FPN are redesigned as a cross-modal feature interaction module, and the specific execution process is as follows:

[0066] Top-down approach: incorporating high-level semantic text features The text features are passed down to lower resolutions via a top-down path. Since the text features have been upsampled and aligned with the visual features, this path primarily enhances the guiding role of text features on visual features through element-wise addition or attention gating mechanisms.

[0067] For example: using attention weights The contribution of modulated text features to visual features:

[0068]

[0069] Lateral connectivity: Enhancing text features from top to bottom With original visual features Concatenation or element-wise addition is performed along the channel dimension to form multimodal fusion features:

[0070]

[0071] By using a concatenation method, the number of channels is compressed to the target dimension through convolution operations. :

[0072]

[0073] Final multimodal fusion features It can be represented as:

[0074]

[0075] in, Channel compression for visual features, Channel mapping and spatial expansion for text features.

[0076] Through the above design, the multimodal fusion layer of the present invention can efficiently integrate visual-linguistic information, providing a high-quality feature representation that combines global semantics and local details for subsequent motion parameter generation.

[0077] Motion parameter generation module: used to convert the fused features into specific AGV motion parameters.

[0078] The objective of this invention is to generate motion parameters for an AGV based on an input image I and a language command T. To this end, this invention extends the CROG (Contrastive Retrieval and Optimization for Guidance) model by introducing a motion parameter generation head to construct an end-to-end multimodal navigation parameter generation framework. The following is a detailed description of the specific implementation and optimization strategies:

[0079] Multimodal features after fusion Based on this, the present invention uses a Transformer decoder architecture to generate motion parameters. As input features to the decoder, while also incorporating language instructions Text features obtained through positional encoding This is incorporated into the decoder's attention mechanism. The decoder captures multimodal features through a self-attention mechanism. The internal relationships are analyzed and linked with text features through a cross-attention mechanism. To enhance semantic alignment, an interactive mechanism is implemented. A fully connected network (FCN) is added to the final layer of the decoder to map the feature vectors output by the decoder to motion parameters.

[0080]

[0081] in, Indicates Transformer decoder, It includes two layers of linear layers and the ReLU activation function.

[0082] To optimize model performance, this invention employs a multi-task total loss function to jointly optimize the contrastive learning and motion parameter regression tasks:

[0083]

[0084] in, and is the weighting coefficient, used to balance the optimization priorities of the two tasks; the motion parameter regression loss uses Smooth L1 Loss to optimize the generation accuracy of motion parameters, and its calculation formula is:

[0085]

[0086] in, The model predicts the first Each motion parameter corresponds to a ground truth value. Smooth L1 Loss combines the advantages of L1 and L2 losses: when the error is small ( L2 loss is used to improve convergence stability, while L1 loss is used to avoid gradient explosion when the error is large. Its mathematical expression is:

[0087]

[0088] To ensure the safe operation of AGVs in complex environments, the following physical constraints must be met:

[0089]

[0090] in, , and These are the AGV's maximum linear velocity, maximum angular velocity, and maximum acceleration, respectively. These constraints are implemented through the post-processing module, including trimming operations and dynamic adjustment mechanisms.

[0091] Clipping operation: If the generated motion parameters exceed the constraint range, they are clipped to the boundary values. For example, if... Then scale proportionally. and to .

[0092] Dynamic adjustment mechanism: When the AGV encounters obstacles or sudden environmental changes, the system obtains the distance to the obstacles through real-time sensing modules (such as LiDAR or depth cameras). And calculate the adjustment factor. ,in To achieve the maximum safe distance, the adjusted motion parameters are as follows:

[0093]

[0094] To avoid AGV trajectory jitter caused by drastic fluctuations in motion parameters, this invention introduces an exponential smoothing algorithm to smooth the motion parameters of consecutive frames:

[0095]

[0096] in, For smoothing coefficients, Using the motion parameters from the previous moment, this method uses weighted historical data to make the AGV's movement smoother, especially in complex marine environments (such as wave disturbances or equipment obstruction), significantly improving navigation stability.

[0097] AGV control module: Used to control the movement of AGVs based on motion parameters.

[0098] like Figure 2 As shown, this invention also proposes a navigation method for offshore wind power booster stations based on multimodal information interaction, which is particularly suitable for autonomous navigation of AGVs (Automated Guided Vehicles) in complex marine environments. The method specifically includes the following steps:

[0099] Step 1: Acquire RGB-D images and voice commands for the offshore wind power booster station scene; the RGB-D image includes RGB images. and depth images The language instructions are natural language instructions. , specifically:

[0100] First, multimodal data was collected for offshore wind power booster stations. To fully reflect the structural complexity, humid and corrosive environment, and diverse equipment layout of the booster station scenario, the training data consisted of the following multimodal data:

[0101] RGB image sequence: covering scenes such as main transformer, GIS bay, high voltage switch, step-up frequency converter, refrigeration equipment, fire protection equipment, and cabin passageway;

[0102] Depth map (D): Acquired by an industrial depth camera, supplementing structural information in narrow spaces;

[0103] LiDAR point cloud (P): Used to enhance spatial geometry modeling capabilities;

[0104] Equipment category and location labeling: used to supervise the vision module's identification of key equipment;

[0105] Natural language commands (T): such as "go to the main transformer maintenance area", "go around the front of the cooling system", "pass through the cable corridor on the left", etc.; real motion trajectory and control parameters The trajectory is recorded by manual remote control or teaching.

[0106] Secondly, the structured paths of the booster station are sampled, taking into account typical narrow compartments, equipment gaps, and irregular obstacles in the booster station. This invention proposes a scene structure constraint sampling method:

[0107] 1) Path decoupling sampling based on multi-layer topology graph, that is, dividing the substation into equipment block nodes (such as "main transformer area", "GIS area" and "UPS room"), and sampling multi-style inspection paths from the topology graph;

[0108] 2) Obstacle proximity sampling involves enhanced sampling at equipment edges, pipe bottoms, and narrow corridors to improve the model's sensitivity to boundary safety.

[0109] 3) Language-trajectory alignment sampling automatically generates corresponding instruction templates for each trajectory segment, such as: "go around the back of the transformer", "pass through the cooler passage", "pass through the main road on the right".

[0110] Step 2: Feature Extraction; Extract features using CLIP's visual encoder and text encoder:

[0111]

[0112]

[0113] Using the collected data, the proposed model is trained with multimodal semantic alignment. The visual encoder adopts the ViT-L / 14 structure and loads large-scale CLIP pre-trained weights. The text encoder uses Transformer and adds domain terms of booster stations, such as "GIS", "cooler", and "circulation duct", to the lexicon. The linear mapping is unified to the same semantic space.

[0114] Considering the dim lighting, strong equipment reflection, and narrow passages inside the substation, this invention adopts a fine-tuning strategy: Pre-training weight loading: Load the visual encoder weights of CLIP ViT-L / 14 into the model and freeze all visual encoder parameters except for the last layer; Domain adaptation fine-tuning: Perform 20 rounds of fine-tuning on the substation's domain data, with the learning rate set to 1e-5.

[0115] Step 3: Multimodal fusion; using FPN to fuse visual and linguistic features:

[0116]

[0117] The multimodal fusion process is the same as the navigation method for offshore wind power booster stations based on multimodal information interaction mentioned above, and will not be repeated here.

[0118] Step 4: Motion parameter generation; using the Transformer decoder to... Converted into motion parameters.

[0119] Specifically, after completing multimodal feature fusion, this invention constructs a motion parameter generation decoder specifically for offshore wind power booster station scenarios and proposes a joint training strategy for strong disturbance environments and cross-modal semantic consistency, enabling the model to stably generate the motion parameters required for autonomous inspection in complex scenarios. This decoder uses fused features as input and further incorporates instruction text features. As a modulated signal, a deep cross-modal semantic association is achieved through an adaptive attention mechanism. An end-to-end optimization approach is employed during the training phase to meet the requirements for highly robust and accurate motion parameter generation in the substation scenario.

[0120] During the training phase, this invention designs a dynamic weight adjuster to optimize the contrast loss L, taking into account the specific characteristics of offshore wind power booster station scenarios. contrastive Regression loss L with motion parameters regressionThe regulator dynamically adjusts the weight coefficients λ1 and λ2 of the "instruction resolvability" and "environmental complexity" of the current training samples in real time. Specifically, for samples with low instruction resolvability (such as ambiguous instructions like "approach the right cabinet"), the system prioritizes increasing the weight of the contrast loss, making the model focus more on improving semantic alignment capabilities; while for samples with high environmental complexity (such as densely populated areas or scenes with occlusions), the system increases the weight of the regression loss to enhance the accuracy of motion parameter generation. This dynamic adjustment mechanism is achieved by introducing an attention-gated network: first, the semantic ambiguity S (based on the entropy value of the language instruction) and environmental complexity E (quantized by the texture entropy and obstacle density of the scene image) of the input sample are calculated, and then the weight allocation factor is obtained through normalization.

[0121]

[0122] Based on this, an exponential decay term γ is further introduced. t =e -αt (α is the decay coefficient, t is the training epoch), to balance the optimization priorities in the early and later stages of training, the final total loss function is:

[0123]

[0124] The training process first inputs the fused multimodal feature sequence into the multi-layer self-attention structure of the decoder. The decoder uses a global self-attention mechanism to establish a mapping relationship between visual and linguistic signals, thereby capturing implicit constraints of semantic and spatial structure such as "approaching the left channel of the main transformer" and "continuing to move forward after bypassing the safety fence".

[0125] To enhance the control of text instructions over the directionality of motion generation, this invention incorporates position-encoded instructions into the cross-attention structure. This allows the relative structure of the text symbol sequence in the semantic space to influence the degree of attention the decoder pays to different fusion feature regions, thereby enabling semantically guided differentiated generation of motion parameters in different station areas such as transformer high-voltage cabinets, inverter areas, and surge arrester overhead lines.

[0126] To ensure optimal performance in both cross-modal semantic understanding and motion parameter generation, this invention proposes a coupled multimodal constraint training mechanism. During training, both cross-modal contrastive loss and motion parameter regression loss are minimized simultaneously. A dynamic weight adjuster is incorporated within the loss function, automatically adjusting the weights of the two losses based on the "instruction resolvability" and "environment complexity" of the current batch. For samples with semantic ambiguity or vague descriptions in the instructions (e.g., "near the right cabinet"), the system increases the weight of the contrastive loss, enhancing the decoder's semantic parsing capabilities. Conversely, in samples with clear scene structures but difficult motion prediction (e.g., areas with dense obstacles), the training mechanism increases the weight of the regression loss, enabling the model to more accurately fit the motion parameter distribution.

[0127] Step 5: Motion parameter constraints and adaptive adjustment; apply constraints and calculate adjustment coefficients. Generate adjusted motion parameters :

[0128]

[0129]

[0130] The constraints and adaptive adjustments of motion parameters are the same as those of the above-mentioned navigation method for offshore wind power booster stations based on multimodal information interaction, and will not be repeated here.

[0131] Step 6: Motion parameter smoothing; apply smoothing to generate smoothed motion parameters. :

[0132]

[0133] The smoothing process is the same as the navigation method for offshore wind power booster stations based on multimodal information interaction mentioned above, and will not be repeated here.

[0134] Step 7: AGV control executes inspection tasks; according to Control the movement of AGVs to complete inspection tasks.

[0135] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A navigation method for offshore wind power booster stations based on multimodal information interaction, characterized in that: Acquire images and commands for offshore wind power booster station scenarios; A contrastive learning model is used to extract features from images and instructions based on similarity, resulting in image features and instruction text features that are mapped to a unified semantic space. The extraction of instruction text features involves using a text encoder to segment and position the instructions, adding position embeddings to each token, and then extracting context-related instruction text features through a multi-layer encoder. Multimodal feature fusion is performed on the mapped image features and the instruction text features to obtain the fused multimodal features; A motion parameter generation decoder is employed to acquire multimodal features and inject the instruction text features obtained through positional encoding into the decoder's attention mechanism. This allows the decoder to capture the internal correlations of multimodal features through a self-attention mechanism. The interaction between the decoder and instruction text features via a cross-attention mechanism allows the relative structure of the text symbol sequence in the semantic space to influence the decoder's attention to different multimodal feature regions, thereby generating semantically guided motion parameters in different regions. A contrastive loss for the contrastive learning model and a motion parameter regression loss for the decoder are constructed and jointly trained. The contrastive loss maximizes the similarity between matching image features and instruction text feature pairs, while the motion parameter regression loss minimizes the error between the predicted motion parameters and the ground truth motion parameters. Based on the trained contrastive learning model and the decoder, motion parameters are generated for the current offshore wind power booster station scenario to navigate the inspection task.

2. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: In the joint training, a dynamic weight adjuster is used to dynamically adjust the weight coefficients of the contrastive loss and the motion parameter regression loss by evaluating the instruction resolvability and environmental complexity of the current training sample in real time. For samples with low instruction resolvability, the weight of the contrastive loss is increased to improve semantic alignment capability. For samples with high environmental complexity, the weight of the regression loss is increased to enhance the accuracy of motion parameter generation.

3. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 2, characterized in that: The dynamic weight adjuster is implemented by introducing an attention gating network; Calculate the semantic ambiguity and environmental complexity of the input sample; By normalizing the data, the weights of semantic ambiguity and environmental complexity in the sum of the two are calculated separately, and the contrast loss weights and motion parameter regression loss weights are obtained respectively. An exponential decay term based on the decay coefficient and training rounds is introduced to balance the optimization priorities in the early and later stages of training. The total loss function is obtained by multiplying the value of the contrastive loss function by the contrastive loss weight and the exponential decay term, and by multiplying the value of the motion parameter regression loss function by the difference between the motion parameter regression loss weight and the exponential decay term.

4. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: The multimodal feature fusion is achieved by using the top-down and lateral connection interaction mechanism of the feature pyramid network to jointly model the extracted image features and instruction text features in the spatial and semantic dimensions to generate fused multimodal features. The top-down instruction text features provide global semantic guidance for the image features, while the lateral connection image features retain local details.

5. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: Physical constraints are constructed for the motion parameters. If the motion parameters exceed the constraint range, they are clipped to the boundary value. If an obstacle is encountered during the motion, an adjustment coefficient is calculated based on the distance to the obstacle and the maximum safe distance to adjust the motion parameters.

6. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: An exponential smoothing algorithm is introduced to smooth the motion parameters of consecutive frames. Based on the smoothing coefficient, the weights of the previous motion parameters and the adjusted motion parameters are constructed to generate stable motion parameters.

7. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: The extraction of image features involves using a visual encoder to segment the image into a set of image blocks and performing linear projection to convert them into serialized tokens. Then, a multi-layer encoder is used to extract global semantic features, resulting in image features that include local image texture and the overall semantic structure of the scene.

8. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: The images include RGB image sequences and depth maps. The RGB images include, but are not limited to, covering the main transformer, GIS bay, high-voltage switch, step-up inverter, refrigeration equipment, fire-fighting equipment, and cabin passageways. Spatial geometric information is obtained by collecting point cloud data. The instruction text data is a natural language instruction; The motion parameters include linear velocity, angular velocity, and acceleration.

9. The navigation method for offshore wind power booster stations based on multimodal information interaction according to claim 1, characterized in that: Scene structure constraint sampling is performed on the offshore wind power booster station, including path decoupling sampling, obstacle proximity sampling, and language-trajectory alignment sampling; Based on multi-layer topology graph-based path decoupling sampling, the offshore wind power booster station is divided into equipment block nodes, and multi-style inspection paths are sampled from the topology graph. Obstacle proximity sampling enhances sampling at equipment edges, pipe bottoms, and narrow passageways. Language-track alignment sampling automatically generates a corresponding instruction template for each track segment.

10. A navigation system for offshore wind power booster stations based on multimodal information interaction, comprising an image acquisition module, a command input module, a multimodal fusion module, a motion parameter generation module, and a guidance and control module, characterized in that: The navigation method for offshore wind power booster stations based on multimodal information interaction as described in any one of claims 1 to 9 sequentially performs image and command acquisition, image feature and command text feature fusion, and motion parameters generated based on the fused features for guidance control.

Images