Island road condition image segmentation method based on generated text-driven hybrid expert

Abstract

The application discloses an island road condition image segmentation method based on a generated text-driven hybrid expert, and comprises the following steps: inputting an island road condition image into a multimodal large model to generate a road condition description text corresponding to the image, and respectively extracting visual features and text semantic features of the image; obtaining ground distribution of the text semantic features on the visual features and information of consistency between the text and the image according to a matching relationship between the two; constructing a hybrid expert network comprising multiple expert branches, and generating dynamic weights of the expert branches at each spatial position according to the visual features, the ground distribution and the information of consistency between the text and the image; inputting the visual features into each expert branch to obtain output features, weighting and fusing the output features of each expert at each spatial position based on the dynamic weights to obtain initial fusion features; determining a fusion adjustment coefficient according to the information of consistency between the text and the image, multiplying the initial fusion features by the fusion adjustment coefficient, adding the initial fusion features and the visual features to obtain fusion features, and inputting the fusion features into a segmentation decoding network to obtain a segmentation result.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method for segmenting island road condition images based on generated text-driven hybrid experts. Background Technology

[0002] Difficulties in extracting road areas and inaccurate road condition information identification in island scenarios are key issues hindering island traffic management and emergency response. Island road condition image segmentation, as an important method to address these problems, utilizes image segmentation algorithms to automatically extract road areas from remote sensing images, UAV images, or surveillance images of islands. This provides crucial support for road trafficability analysis, route planning, post-disaster assessment, and emergency rescue, and is of great significance in island disaster monitoring and traffic inspection scenarios.

[0003] Addressing the main challenge of island road condition image segmentation—namely, how to fully utilize road structure and high-level semantic information within the image—most existing methods primarily rely on visual features for segmentation and attempt to fuse image features with textual semantics to improve segmentation accuracy and semantic understanding. In existing technologies, one type of method jointly trains a large multimodal model and segmentation / decoding end-to-end to directly output a segmentation mask. However, this relies on large-scale image-text-mask pairing data and requires overall fine-tuning of the large model, resulting in high annotation and training costs. Furthermore, semantic judgment and segmentation results are coupled in a black box, making error localization difficult. Another type of method uses a frozen large multimodal model to generate descriptive text as a semantic prior, combined with an independent visual segmentation backbone, offering advantages such as low data dependency and interpretability.

[0004] However, existing technologies have significant shortcomings in practical applications. On the one hand, island images contain a large amount of interfering information such as vegetation, water accumulation, shadows, and complex terrain, making it difficult for models to accurately identify road areas relying solely on visual features. On the other hand, if image features and text semantics are coarsely fused, effective semantic information related to road conditions is easily lost. Furthermore, the frozen multimodal large model is not aligned for the segmentation task, and its generated text has biases that may not match the predictions of the visual backbone. Direct fusion would introduce text bias into the segmentation results. Therefore, in schemes that use frozen multimodal large models to generate text priors and combine them with independent visual backbones, how to utilize reliable text semantics while suppressing the contamination of segmentation results by text bias has become an important task that urgently needs to be addressed. Summary of the Invention

[0005] This invention proposes a method for segmenting island road condition images based on generative text-driven hybrid experts, which solves the problem that existing technologies are unable to effectively align and adaptively fuse image features and text semantics while suppressing a large amount of interference information in island images.

[0006] To address the aforementioned technical problems, this invention provides a method for island road condition image segmentation based on generated text-driven hybrid experts, comprising the following steps: Step S1: Input the island road condition image into the multimodal large model to generate road condition description text, and extract the visual features of the island road condition image and the textual semantic features of the road condition description text respectively; Step S2: Based on the matching relationship between the visual features and the text semantic features, obtain the grounding distribution of the text semantic features on the visual features and the image-text consistency information; Step S3: Construct a hybrid expert network including multiple expert branches, and generate dynamic weights for each expert branch at each spatial location based on the visual features, grounding distribution, and image-text consistency information; Step S4: Input the visual features into each expert branch to obtain output features, and perform spatial position-wise weighted fusion of the output features of each expert branch based on the dynamic weights to obtain initial fused features; Step S5: Determine the fusion adjustment coefficient based on the image-text consistency information. The fusion adjustment coefficient increases as the image-text consistency information increases. Multiply the initial fusion feature by the fusion adjustment coefficient and add it to the visual feature to obtain the fusion feature. Input the fusion feature into the segmentation decoding network to obtain the segmentation result.

[0007] Preferably, step S1, extracting visual features from the island road condition image, includes the following steps: inputting the island road condition image into a visual encoder for hierarchical feature extraction, and extracting the initial visual features from the visual encoder output... Add position encoding ,get : ; In the formula, The learnable fusion coefficient; Will After layer normalization, the input is given to the self-attention module and the enhanced visual features are obtained through residual connections. : ; In the formula, This represents a self-attention operation; Representation layer normalization operation.

[0008] Preferably, step S2 includes the following steps: Step S21: Apply the enhanced visual features Expanded into a sequence of visual features according to spatial dimensions , with the first i Text semantic features corresponding to each text field As a query, using the visual feature sequence As keys and values, the first value is obtained through a cross-attention mechanism. i Visual response features corresponding to each text field and the first in the cross-attention mechanism i The attention weight distribution of the text field at each spatial location in the visual feature sequence is used as the first... i Grounding distribution of text fields ; Step S22: Calculation and The cosine similarity after linear mapping, and then the Sigmoid activation function to obtain the first... i Field-level text-image consistency coefficient for each text field : ; In the formula, For the Sigmoid function; , These are the text feature mapping matrix and the visual feature mapping matrix, respectively; Step S23: When Below the preset consistency threshold At that time, Multiply by the attenuation coefficient Get the first i Corrected consistency coefficient of each text field ,otherwise Pick The image-text consistency information is composed of the corrected consistency coefficients of each text field, wherein... The attenuation coefficient has a value greater than 0 and less than 1.

[0009] Preferably, step S3 generates dynamic weights for each expert branch at each spatial location based on the visual features, grounding distribution, and image-text consistency information, including the following steps: Step S31: Weight the grounding distribution of the text fields based on the corrected consistency coefficient of each text field to obtain the reliable grounding distribution of each text field. Stack the reliable grounding distributions of each text field according to the field dimension to obtain the text grounding vector at each spatial location. Perform a weighted average of the text semantic features based on the image-text consistency information to obtain reliable text semantic features. Step S32: At each spatial location, the enhanced visual features of the spatial location, the text grounding vector, and the trusted text semantic features are concatenated to obtain the concatenated features of the spatial location; Step S33: Input the splicing features of each spatial location into the gating network to obtain the gating score of each expert branch at each spatial location. All spatial locations share the parameters of the gating network. Step S34: Perform Softmax normalization with temperature coefficient on the gating scores of each expert branch at each spatial location to obtain the dynamic weight of each expert branch at each spatial location.

[0010] Preferably, the gating network includes a first fully connected layer, a GELU activation layer, and a second fully connected layer connected in sequence, and the expression for the gating score of each expert branch at each spatial location in step S33 is: ; In the formula, Spatial location The gate control score; Spatial location The splicing characteristics at the location; , The weights and biases of the first fully connected layer; , These are the weights and biases for the second fully connected layer.

[0011] Preferably, before step S32, the method further includes determining the visual uncertainty of each spatial location based on the enhanced visual features, including the following steps: [The text abruptly ends here, so the translation stops here as well.] The feature input at the location is used to classify the data, and the predicted distribution of the spatial location for each road condition category is obtained. The normalized entropy of the predicted distribution is calculated as the visual uncertainty of the spatial location. : ; In the formula, This represents the total number of road condition categories. Spatial location Belongs to the first c The predicted probability of a class; The value ranges from 0 to 1; In step S32, both the text grounding vector of the spatial location and the semantic features of the reliable text are multiplied by the modulation coefficient. Then, it is spliced ​​with the enhanced visual features of the spatial location to obtain the spliced ​​features of the spatial location.

[0012] Preferably, step S34 further includes a step of correcting the gating score, including: determining the text field corresponding to each expert branch according to a preset field-expert correspondence, and distributing the reliable grounding of the text field corresponding to each expert branch in spatial location. The value at that point is multiplied by the binding strength coefficient. Then, it is added to the gating score of the expert branch at the spatial location to obtain the corrected gating score. : ; In the formula, For the first k Each expert branch in spatial location The gate control score; For the first k The index of the text field corresponding to each expert branch; For the first The reliable grounding distribution of each text field is located in spatial location. The value at; The binding strength coefficient; After replacing the gated score in step S34 with the modified gated score, the modified gated score is... Softmax normalization with a temperature coefficient is performed to obtain the dynamic weights of each expert branch at each spatial location: ; In the formula, For the first k Each expert branch in spatial location Dynamic weights on; Temperature coefficient; K This refers to the number of expert branches.

[0013] Preferably, the expression for the fusion adjustment coefficient in step S5 is: ; ; In the formula, The mean of the corrected consistency coefficients for each text field; This represents the upper limit of fusion strength. This is the sensitivity coefficient; The number of preset road condition fields; For the first i Corrected consistency coefficient for each text field.

[0014] Preferably, step S1, which involves inputting the island road condition image into a multimodal large model to generate road condition description text, includes the following steps: inputting the island road condition image and a preset island road condition prompt template into the multimodal large model to obtain road condition description text corresponding to the island road condition image; the island road condition prompt template is used to guide the multimodal large model to describe the island road condition image from at least two dimensions, including road spatial location, road morphology, environmental interference, traffic status, and boundary adjacency; the road condition description text is structured text, which is used to characterize at least one of road area distribution, road boundary features, coastal background interference, and traffic anomaly information.

[0015] Preferably, step S5 further includes optimizing the parameters of the hybrid expert network and the segmentation-decoding network using a joint loss function, the expression of which is: ; In the formula, The Dice loss function is used to measure the degree of region overlap between the segmentation result and the true annotation. The cross-entropy loss function is used to measure the pixel-level classification error between the segmentation result and the true annotation. The orthogonality constraint loss is applied to the output features of each expert branch. , and These are the weighting coefficients.

[0016] The advantages of this invention include at least the following: 1. Based on the island road condition prompt template, a structured road condition description text is generated, which enables the text to express information such as road location, road shape, environmental interference, traffic status and boundary adjacency. Compared with ordinary description text, this structured text is more suitable as a semantic prior for island road condition image segmentation, reduces the dependence on manual text annotation, and improves the scenario adaptability of the method. 2. After extracting the visual features of the image and the semantic features of the text, the image-text consistency information is further calculated to determine the degree of matching between the generated text and the image content. When the text and image content are consistent, the segmentation guidance role is enhanced. When the text and image content are inconsistent, the interference of the text on the subsequent expert weight allocation is reduced, thereby effectively reducing the adverse effects of the generated text deviation on the segmentation results. 3. Construct a hybrid expert network for island road conditions. By using a gated network to combine text semantic features, image-text consistency information and image visual features to generate dynamic weights, adaptive modeling and fusion of features such as road continuity, coastal background interference, water accumulation and reflection, occlusion boundaries and traffic anomalies are performed. This improves the accuracy, continuity and robustness of road area and traffic anomaly area segmentation in complex island environments. 4. By utilizing the grounding distribution of each text field on the image and performing spatial modulation according to field-level consistency, expert dynamic weights are generated for each spatial location. This allows the generated text to participate in expert routing only in the areas where it is trusted to be grounded. It can locally utilize and suppress regionally unreliable generated text, and the segmentation granularity is better than global weight allocation. 5. The residual strength of expert fusion changes monotonically with the overall consistency between the generated text and the image. When the overall consistency between the generated text and the image content is low, the residual strength is small and the fusion features are dominated by visual features, providing a safe performance lower bound for the segmentation results and reducing the impact of multimodal large model generation bias on the segmentation results. Furthermore, visual uncertainty is introduced to spatially modulate the text prior, suppressing the influence of text grounding vectors and text semantics on expert routing in areas where visual discrimination is ambiguous, avoiding blindly accepting the generated text. At the same time, the credibility of the generated text is explicitly decomposed according to the road condition field and spatial location, so that the source of text bias can be located to specific fields and regions. Compared with the end-to-end scheme that couples semantic judgment with the segmentation results, it has better interpretability and robustness. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the acquisition of visual features, text semantic features, and image-text consistency information according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the hybrid expert network and dynamic weight generation structure in an embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-expert branch fusion process in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for island road condition image segmentation based on generated text-driven hybrid experts, including the following steps: Step S1: Input the island road condition image and the preset island road condition prompt template into the multimodal large model to generate road condition description text corresponding to the island road condition image.

[0020] This embodiment first constructs and obtains an island road condition image dataset. This dataset includes road scene images and corresponding road area annotation information, covering two or more types of images of island roads, coastal roads, waterfront roads, flooded roads, obstructed roads, and roads with traffic abnormalities, and is used to train and validate the island road condition image segmentation model.

[0021] This embodiment uses images of island road conditions. With the preset island road condition prompt template Inputting into the Qwen2.5-VL-7B-Instruct multimodal large model, semantic understanding and road condition element inference are performed on island road condition images to generate structured road condition description text corresponding to the image content. .

[0022] The island traffic condition prompt template guides multimodal large models to describe island traffic conditions from at least two dimensions: road spatial location, road morphology, environmental disturbances, traffic conditions, and boundary adjacency. For example, the island traffic condition prompt template could be: "Focus on two aspects: the spatial location of the road in the image, and whether there are abnormal conditions affecting traffic, such as water accumulation, strong light reflection, damage, collapse, occlusion, or blockage. Output a structured description related to road region segmentation."

[0023] The road condition description text is structured text, used to represent at least one of the following: road area distribution, road boundary features, coastal background interference, and traffic anomaly information. The structured text generated through the above-mentioned prompt template can provide high-level semantic prior information for subsequent image-text consistency calculations and expert weight allocation.

[0024] Step S2: Extract the visual features of the road condition image and the textual semantic features of the road condition description text respectively. Based on the matching relationship between the visual features and the textual semantic features at each spatial location, obtain the grounding distribution of the textual semantic features on the visual features. Based on the degree of matching between the textual semantic features and the visual features, obtain the image-text consistency information.

[0025] like Figure 2 As shown, in this embodiment, the island road condition image is input into a visual encoder for hierarchical feature extraction to obtain visual features. Simultaneously, the structured road condition description text is input into a text encoder for text semantic feature extraction to obtain text semantic features. Then, based on the semantic matching relationship between the image visual features and the text semantic features, image-text consistency information is calculated. This image-text consistency information is used to characterize the degree of matching between the structured road condition description text and the island road condition image in terms of road spatial location, road morphology, environmental interference, traffic status, and boundary adjacency. The following provides a detailed explanation of the acquisition process for visual features, text semantic features, and image-text consistency information.

[0026] In the visual feature extraction stage, the island road condition image is input into a visual encoder. In this embodiment, a ResNet-50 network is used to downsample the image through multi-stage feature extraction, extracting visual features at different scales to obtain visual features. The image visual features include one or more of the following: road area features, road boundary features, coastal background features, and traffic anomaly area features.

[0027] Positional encoding is performed on the initial visual features output by the visual encoder. Specifically, positional encodings weighted by learnable fusion coefficients are superimposed on the initial visual features to obtain the visual features with added positional encodings. Its expression is: ; In the formula, This represents the initial visual features output by the visual encoder. This represents the location encoding corresponding to the visual features. This represents the fusion coefficient of the positional encoding. It is a learnable parameter and is initialized to a small value of 0.1, so that the model mainly relies on the original image features extracted by the visual encoder in the initial stage and gradually learns the contribution strength of the positional encoding during the training process.

[0028] The visual features, after incorporating positional encoding, are normalized by layers and then input into the self-attention module. Enhanced visual features are obtained through residual connections, and their expression is as follows: ; In the formula, This represents enhanced visual features that contain long-range dependencies. Through a self-attention mechanism, the model can model global dependencies between different spatial locations in an image, enhancing the ability of visual features to express the continuous structure of road areas.

[0029] In the text semantic feature extraction stage, the structured road condition description text is parsed according to preset road condition fields to obtain multiple text fields. These preset road condition fields include road spatial location, road morphology, environmental interference, traffic status, and boundary adjacency. When the multimodal large model outputs field titles or segmented content according to the prompt template, the corresponding text fields are extracted directly based on the field titles or segmentation markers. When the output text does not strictly conform to the preset structure, semantic matching is performed on the output text. Specifically, a small number of keywords are pre-set for each preset road condition field to assist in identifying obvious field content. The output text is segmented into multiple text fragments according to sentences, punctuation marks, and keywords. Each text fragment is input into the BERT-base encoder to obtain fragment vectors. The cosine similarity between each fragment vector and each preset road condition field is calculated, and each text fragment is assigned to the field with the highest cosine similarity. For missing fields or unmatched text content, they are treated as low-confidence fields, and their corresponding weights are reduced in subsequent text-image consistency calculations.

[0030] Each text field is input into a text encoder for semantic modeling. In this embodiment, the text encoder uses the BERT-based model to obtain text semantic features. Its expression is: ; ; In the formula, Indicates the first i One text field; Indicates the relationship with the first i The text semantic features corresponding to each text field. Each text field includes at least two of the following: road spatial location, road morphology, environmental interference, traffic status, and boundary adjacency. Multiple text fields refer to valid fields parsed from a fixed preset set of fields, with a fixed total number of 5 fields. When a field is missing or cannot be reliably matched in the output text of the multimodal large model, or when the image itself does not possess the descriptive features of that field, that field is set to an empty field or a low-confidence field, and its contribution to the generation of reliable text semantic features and gating weights is reduced through effective field labeling and image-text consistency coefficients.

[0031] In the image-text consistency information calculation stage, the enhanced visual features are expanded into a visual feature sequence according to the spatial dimension, and its expression is: ; In the formula, Indicates the number of spatial locations; Indicating the enhancement of visual features, the first Visual representation of a spatial location.

[0032] Using textual semantic features as queries and visual feature sequences as keys and values, a cross-attention mechanism is employed to obtain the visual response features corresponding to each text field. Specifically, textual semantic features serve as semantic guidance information, and response weights are calculated for visual features at each spatial location in the visual feature sequence. Based on these response weights, the visual features are weighted and aggregated to obtain the visual response features corresponding to the first text field. i Visual response features corresponding to each text field Its expression is: ; In the formula, Represents a text query mapping matrix; Represents the visual key mapping matrix; Represents the visual response mapping matrix; This represents the mapped feature dimension. The Softmax function is used to obtain the ... i The response weights of each text field at different spatial locations of the enhanced visual features.

[0033] This yields the visual response feature set. Its expression is: .

[0034] The response weight is the first i The distribution of attention weights for a text field at each spatial location in the visual feature sequence is denoted as the grounding distribution of that text field. : ; Grounding distribution Characterizing the first i The spatial location of the road condition elements described by each text field in the island road condition image; This indicates the spatial location of the text field. The grounding strength at that location.

[0035] After obtaining the visual response features, the matching degree between the text semantic features and the corresponding visual response features is calculated to obtain the field-level image-text consistency coefficient, which is expressed as follows: ; In the formula, Indicates the relationship with the first i Visual response features corresponding to each text field; and These represent the text feature mapping matrix and the visual feature mapping matrix, respectively; This represents the Sigmoid normalization function; Indicates the first i The consistency coefficient between the text field and the visual content of the image.

[0036] The consistency coefficients of the text and images at each field level are combined to obtain the text and image consistency information. Let the first field be... i The text-image consistency coefficient corresponding to each text field is the original score of that field, with a preset consistency threshold of [value missing]. When the image-text consistency coefficient is not lower than the consistency threshold, it remains unchanged. When the image-text consistency coefficient is lower than the consistency threshold, it is multiplied by an attenuation coefficient for attenuation processing to obtain the corrected consistency coefficient. The expression for this segmented calculation rule is: ; ; In the formula, This is the attenuation coefficient, ranging from 0 to 1, excluding 1. In practice, it is determined based on the segmentation performance on the validation set. and The value is searched using the Dice coefficient as an evaluation metric. In this embodiment... We use 0.5 to distinguish between highly consistent and low-consistency fields. A value of 0.2 is used to softly suppress low-consistency fields, that is, to retain a small amount of their semantic information while reducing their impact on the gating weight generation process.

[0037] The consistency information between the text and images is composed of the corrected consistency coefficients corresponding to each field, and its expression is: ; In the formula, The preset number of road condition fields is used. Image-text consistency information is used to adjust the influence of each text field on the generation of expert weights in the subsequent gating network, so that the contribution of text fields with low matching degree with image content is weakened in the gating weight generation process.

[0038] Step S3: Construct a hybrid expert network that includes multiple expert branches, and generate dynamic weights for each expert branch at each spatial location based on visual features, grounding distribution, and image-text consistency information.

[0039] like Figure 3 As shown, this embodiment constructs a hybrid expert network for island road condition features, which includes multiple parallel expert branches and a gating network. The number of expert branches is determined based on the types of road condition features that need to be prioritized for modeling in the island road condition images. Considering five key features commonly found in island road segmentation—road continuity, coastal background interference, water reflection, occlusion boundaries, and traffic anomalies—this embodiment preferably sets the number of expert branches to five: a road continuity expert, a coastal background suppression expert, a water reflection expert, an occlusion boundary recovery expert, and a traffic anomaly identification expert. Each expert branch employs a different feature modeling structure, and their parameters are not shared.

[0040] The number and type of expert branches are determined based on the key road condition features that need to be modeled in the island road condition images. This embodiment considers five common key road condition features in island road segmentation: road continuity, coastal background interference, water accumulation reflection, occlusion boundaries, and traffic anomalies. Correspondingly, five expert branches are set: a road continuity expert, a coastal background suppression expert, a water accumulation reflection expert, an occlusion boundary restoration expert, and a traffic anomaly identification expert. Each expert branch uses a different feature modeling structure, and their parameters are not shared. In other embodiments, the number and type of expert branches can be increased or decreased according to the actual road condition feature types appearing in the target scene.

[0041] The gating network generates dynamic weights for each expert branch at each spatial location based on the grounding distribution of text semantic features, image-text consistency information, and enhanced visual features. This ensures that text fields consistent with the image content participate in expert weight allocation within their grounding regions, while reducing the impact of text fields inconsistent with the image content on expert weight allocation. The dynamic weight generation process includes steps S31 to S34.

[0042] Step S31: Based on the corrected consistency coefficient of each text field, the grounding distribution is modulated to obtain a reliable grounding distribution. i The expression for the reliable grounding distribution of each text field is: ; In the formula, For the first i Grounding distribution of each text field; For the first i Corrected consistency coefficient for each text field.

[0043] Stacking the reliable grounding distributions of each text field along the field dimension yields the text grounding vector at each spatial location: In the formula, The number of preset road condition fields; Characterizing the spatial location of each road condition field The grounding strength after confidence modulation.

[0044] Based on the consistency information between the text and the image, a weighted average of the text semantic features is used to obtain the reliable text semantic features: ; In the formula, Indicates the first i The text semantic features corresponding to each text field; To prevent constants with a denominator of zero.

[0045] To avoid blindly accepting generated text in areas where vision itself is difficult to discern, the visual uncertainty of each spatial location is determined based on enhanced visual features before step S32. The enhanced visual features are then applied to the spatial location... The feature input at that location is used to classify the header, and the predicted distribution of each road condition category for that spatial location is obtained. Using its normalized entropy as visual uncertainty : ; In the formula, This represents the total number of road condition categories. Spatial location Belongs to the first c The predicted probability of a class; The value ranges from 0 to 1. The larger the value, the more blurred the visual features become in terms of distinguishing road condition categories at that spatial location.

[0046] The classification head is used to estimate visual uncertainty at the feature layer and is independent of the pixel classification convolutional layers in the subsequent segmentation and decoding network.

[0047] Step S32: At each spatial location, ground the text vector of that spatial location. With trustworthy text semantic features Multiply by the modulation coefficient Then, it is combined with the enhanced visual features of that spatial location. By splicing, the splicing features of that spatial location are obtained. .

[0048] When the visual uncertainty of a certain spatial location is high, the influence of the text grounding vector and the semantic features of the credible text on the expert routing of that spatial location is suppressed. Thus, a conservative routing is adopted in areas where the text cannot be independently verified by vision, and the generated text is only accepted in areas where the vision can serve as independent evidence to correct its deviation.

[0049] Step S33: Combine the splicing features of each spatial location Input a gated network with parameters shared by all spatial locations. The gated network includes a first fully connected layer, a GELU activation layer, and a second fully connected layer connected in sequence. The gated scores of each expert branch at that spatial location are obtained. ; In the formula, for A dimensional vector, whose dimensional vector is the first dimensional vector. k Each component For the first k Each expert branch's gating score at this spatial location; , The weights and biases of the first fully connected layer; , The weights and biases of the second fully connected layer; This refers to the number of expert branches.

[0050] According to the preset field-expert correspondence, the reliable grounding distribution of each text field is used as the spatial bias term of the corresponding expert branch gating score, and superimposed on the gating score to obtain the corrected gating score. : ; In the formula, Indicates the first k The index of the text field corresponding to each expert branch, for example, the road spatial location field corresponds to the road continuity expert, the environmental interference field corresponds to the water accumulation and reflection expert, and the boundary adjacency field corresponds to the occlusion boundary recovery expert. For the first The reliable grounding distribution of each text field is located in spatial location. The value at; This is a learnable binding strength coefficient.

[0051] This bias term enhances the gating score of the corresponding expert branch of a text field in its trusted grounded spatial location, thereby structurally ensuring that the text field only enhances the routing weight of the corresponding expert branch in its trusted grounded region.

[0052] Step S34: Apply a temperature coefficient to the corrected gating score at each spatial location. Softmax normalization yields the dynamic weights of each expert branch at that spatial location: ; In the formula, Indicates the first k Each expert branch in spatial location Dynamic weights on; This represents the temperature coefficient, set to be limited to preset upper and lower limits. The learnable parameters within are initialized to 1 in this embodiment. and Take 0.5 and 2.0 respectively.

[0053] Step S4: Input the visual features into each expert branch respectively, and perform spatial position-wise weighted fusion of the output features of each expert branch based on the dynamic weights at each spatial position to obtain the initial fused features. Perform residual fusion of the initial fused features and the visual features to obtain the fused features.

[0054] like Figure 4As shown, in this embodiment, the visual features of the image are input into multiple expert branches to obtain multiple expert output features. Each expert branch adopts a different feature modeling structure, and each expert output feature corresponds to at least one road condition feature among island road continuity, coastal background interference, water reflection, occlusion boundary, and traffic anomalies. The specific network structure expression of each expert branch is as follows.

[0055] The output characteristics of the road continuity expert are: ; The output features of the coastal background suppression expert are: ; The output characteristics of the water reflection expert are: ; The output characteristics of the occlusion boundary restoration expert are: ; The output features of the passage anomaly identification expert are: ; In the above expression, Indicates the kernel size as And the void ratio is The convolution operation, when Time indicates convolution; Indicates feature concatenation operation; This indicates the channel attention module; This represents a spatial attention module; This indicates a boundary-enhancing convolutional module; This represents a self-attention module. Parameters are not shared between different expert branches, and the spatial size and number of channels of the output features of each expert branch remain consistent.

[0056] Road continuity experts through Convolution and different void ratios The structure of cascaded dilated convolutions enhances the representation of long, elongated road structures. Coastal background suppression experts, through... Convolution and The convolutional stitching, followed by the channel attention module and spatial attention module structure, suppresses background interference from sea surfaces, beaches, and shorelines. Water reflection experts utilize parallel processing with different void ratios... The structure of dilated convolutional splicing is used to extract multi-scale reflective and water accumulation features. Occlusion boundary restoration experts... A concatenated structure of boundary enhancement convolutional modules and spatial attention modules is used to recover road boundaries obscured by vegetation, shadows, or vehicles. Traffic anomaly detection experts use a structure of dilated convolutional modules concatenated with self-attention modules to identify abnormal areas such as road damage, collapses, and blockages.

[0057] The expression for the boundary-enhancing convolution module is: ; This module performs 3x3 convolutions on the input features and 3x3 convolutions on the difference between the input features and their 3x3 mean pooling. Then, the two feature paths are concatenated and fused by 1x1 convolutions, thereby achieving the joint expression of ordinary features and boundary enhancement features and unifying the number of output channels.

[0058] To further ensure that different expert branches learn differentiated features, an orthogonality constraint loss for expert outputs is introduced during training. Global average pooling and L2 normalization are then applied to the output features of each expert to obtain a normalized semantic representation. Its expression is: ; In the formula, Indicates the first Normalized semantic representation of each expert branch; Indicates the first Each expert branch in spatial location The eigenvector at that location; and These represent the height and width of the expert output feature, respectively; the L2 norm is used for normalization. To prevent constants with zero denominators, this embodiment takes... .

[0059] The expression for the orthogonality constraint loss is: ; By minimizing the orthogonality constraint loss, the normalized semantic representations of different expert branches are made as orthogonal as possible, thereby promoting the learning of differentiated island road condition features by different expert branches.

[0060] Based on the dynamic weights of each expert branch at each spatial location, the output features of each expert are weighted and summed at each spatial location to obtain the initial fusion features.

[0061] Spatial location The expression for the initial fusion feature at the given location is: ; In the formula, For the first k Each expert branch in spatial location Dynamic weights on; For the first k Each expert outputs a feature vector at that location.

[0062] Because the dynamic weights change with spatial location, an expert branch only receives a larger weight in the area where its corresponding road condition field is reliably grounded, thus achieving spatial adaptive routing of the expert branch; for example, when the text environment interference field is reliably grounded to the water accumulation area in the image, the water reflection expert receives a larger weight in that area and a smaller weight in other areas.

[0063] The initial fused features and the enhanced visual features are residually fused using a fusion adjustment coefficient to obtain the fused features, the expression of which is: ; In the formula, To enhance visual features; This represents the fusion adjustment coefficient.

[0064] Fusion adjustment coefficient As the degree of aggregation of consistent information between text and images changes monotonically, its expression is: ; ; In the formula, The mean of the consistency coefficient for each text field is adjusted to characterize the overall consistency aggregation degree between the generated text and image content; This represents the upper limit of fusion strength. Both are sensitivity coefficients and learnable parameters, as described in this embodiment. Initialize to 0.5. Initialize to 1; Set the preset number of road condition fields.

[0065] When the generated text and image content have a high overall consistency Larger Approaching Hybrid expert fusion features contribute significantly to the segmentation results; however, when the overall consistency of text generated by a multimodal large model is low... Smaller Smaller, fusion features To enhance visual features The segmentation result is primarily determined by the visual backbone features, thus providing a safe lower bound for segmentation performance when the generated text as a whole is unreliable, and reducing the impact of generated text bias on the segmentation result.

[0066] Step S5: Input the fused features into the segmentation decoding network to obtain the island road condition image segmentation result.

[0067] In this embodiment, the segmentation decoding network includes a first transposed convolutional upsampling layer, a first convolutional reconstruction layer, a second transposed convolutional upsampling layer, a second convolutional reconstruction layer, and a pixel classification convolutional layer connected in sequence. The visual encoder in this embodiment uses a ResNet-50 network, and in its final feature extraction stage, dilated convolution replaces downsampling, making the total downsampling factor of the visual encoder relative to the input image 16. The first and second transposed convolutional upsampling layers each increase the spatial resolution of the feature map by 4 times. The product of the two upsampling factors is equal to the total downsampling factor of 16, thus restoring the spatial resolution of the fused features to be consistent with the input island road condition image. After each upsampling stage, road boundary details and traffic anomaly area features are reconstructed through convolutional layers to obtain a segmentation feature map corresponding to the size of the input island road condition image. In other embodiments, the features from each downsampling stage of the visual encoder can also be introduced into the corresponding upsampling layer through skip connections to supplement high-resolution road boundary details.

[0068] The processing steps of the segmentation decoding network include the following steps.

[0069] The fused features are input into the first transposed convolutional upsampling layer to obtain the first upsampling features. Its expression is: ; In the formula, This indicates the first transposed convolution upsampling operation.

[0070] The first upsampled feature is input into the first convolutional reconstruction layer to obtain the first reconstructed feature. Its expression is: ; In the formula, This represents a convolution operation used to enhance road boundary details and local spatial information.

[0071] The first reconstructed feature is input into the second transposed convolutional upsampling layer to obtain the second upsampling feature. Its expression is: ; In the formula, This indicates the second transposed convolution upsampling operation.

[0072] The second upsampled feature is input into the second convolutional reconstruction layer to obtain the second reconstructed feature. Its expression is: ; In the formula, This represents a convolution operation used to recover the spatial structure of road areas and the features of traffic anomaly areas.

[0073] The second reconstructed feature is input into the pixel classification layer, and the softmax function outputs the island road condition image segmentation result. Its expression is: ; In the formula, Represents a pixel classification convolutional layer; This represents the image segmentation results for road conditions on the island. The image segmentation results for road conditions on the island include road areas and areas with abnormal traffic.

[0074] During model training, a loss function is calculated based on the island road condition image segmentation results output by the decoding network and the corresponding ground truth annotations to optimize the model parameters. The loss functions include the Dice loss function, the cross-entropy loss function, and the expert output orthogonality constraint loss.

[0075] Dice loss function The expression used to measure the degree of overlap between the segmentation result and the true annotation is: ; In the formula, The total number of categories to be divided; This represents the total number of pixels in the image. Indicates the first i The pixel belongs to the first The predicted value of the class; Indicates the first i The pixel belongs to the first The actual labeled value of the class, when the first i The value is 1 if a pixel belongs to the target category, and 0 otherwise. This indicates the smoothing term.

[0076] Cross-entropy loss function The expression for the pixel-level classification error used to measure the difference between the segmentation result and the true annotation is: .

[0077] The Dice loss function, cross-entropy loss function, and expert output orthogonality constraint loss are weighted and combined to obtain the joint loss function, the expression of which is: ; In the formula, , and This is the weighting coefficient. In this embodiment... and The value is 0.5. The value is 0.01.

[0078] This invention also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the various steps of the above-described method for island road condition image segmentation based on generative text-driven hybrid experts.

[0079] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0080] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for island road condition image segmentation based on generative text-driven hybrid experts, characterized in that, Includes the following steps: Step S1: Input the island road condition image into the multimodal large model to generate road condition description text, and extract the visual features of the island road condition image and the textual semantic features of the road condition description text respectively; Step S2: Based on the matching relationship between the visual features and the text semantic features, obtain the grounding distribution of the text semantic features on the visual features and the image-text consistency information; including the following steps: Step S21: Expand the visual features into a visual feature sequence according to the spatial dimension. , with the first i Text semantic features corresponding to each text field As a query, using the visual feature sequence As keys and values, the first value is obtained through a cross-attention mechanism. i Visual response features corresponding to each text field and the first in the cross-attention mechanism i The attention weight distribution of the text field at each spatial location in the visual feature sequence is used as the first... i Grounding distribution of text fields ; Step S22: Calculation and The cosine similarity after linear mapping, and then the Sigmoid activation function to obtain the first... i Field-level text-image consistency coefficient for each text field : ； In the formula, For the Sigmoid function; , These are the text feature mapping matrix and the visual feature mapping matrix, respectively; Step S23: When Below the preset consistency threshold At that time, Multiply by the attenuation coefficient Get the first i Corrected consistency coefficient of each text field ,otherwise Pick The image-text consistency information is composed of the corrected consistency coefficients of each text field, wherein... The attenuation coefficient has a value greater than 0 and less than 1; Step S3: Construct a hybrid expert network comprising multiple expert branches, and generate dynamic weights for each expert branch at each spatial location based on the visual features, grounding distribution, and image-text consistency information, including the following steps: Step S31: Weight the grounding distribution of the text fields based on the corrected consistency coefficient of each text field to obtain the reliable grounding distribution of each text field. Stack the reliable grounding distributions of each text field according to the field dimension to obtain the text grounding vector at each spatial location. Perform a weighted average of the text semantic features based on the image-text consistency information to obtain reliable text semantic features. Step S32: At each spatial location, the visual features of the spatial location, the text grounding vector, and the semantic features of the trusted text are concatenated to obtain the concatenated features of the spatial location; Step S33: Input the splicing features of each spatial location into the gating network to obtain the gating score of each expert branch at each spatial location. All spatial locations share the parameters of the gating network. Step S34: Perform Softmax normalization with temperature coefficient on the gating scores of each expert branch at each spatial location to obtain the dynamic weight of each expert branch at each spatial location. Step S4: Input the visual features into each expert branch to obtain output features, and perform spatial position-wise weighted fusion of the output features of each expert branch based on the dynamic weights to obtain initial fused features; Step S5: Determine the fusion adjustment coefficient based on the image-text consistency information. The fusion adjustment coefficient increases as the image-text consistency information increases. Multiply the initial fusion feature by the fusion adjustment coefficient and add it to the visual feature to obtain the fusion feature. Input the fusion feature into the segmentation decoding network to obtain the segmentation result.

2. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: Step S1 extracts visual features from the island road condition image, including the following steps: inputting the island road condition image into a visual encoder for hierarchical feature extraction, and extracting the initial visual features from the visual encoder output. Add position encoding ,get : ； In the formula, The learnable fusion coefficient; Will After layer normalization, the input is given to the self-attention module and visual features are obtained through residual connections. : ； In the formula, This represents a self-attention operation; Representation layer normalization operation.

3. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: The gated network comprises a first fully connected layer, a GELU activation layer, and a second fully connected layer connected in sequence. The expression for the gated score of each expert branch at each spatial location in step S33 is as follows: ； In the formula, Spatial location The gate control score; Spatial location The splicing characteristics at the location; , The weights and biases of the first fully connected layer; , These are the weights and biases for the second fully connected layer.

4. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: Before step S32, the method further includes determining the visual uncertainty of each spatial location based on the visual features, including the following steps: [The text abruptly ends here, so the translation stops as well.] The feature input at the location is used to classify the data, and the predicted distribution of the spatial location for each road condition category is obtained. The normalized entropy of the predicted distribution is calculated as the visual uncertainty of the spatial location. : ； In the formula, This represents the total number of road condition categories. Spatial location Belongs to the first c The predicted probability of a class; The value ranges from 0 to 1; In step S32, both the text grounding vector of the spatial location and the semantic features of the reliable text are multiplied by the modulation coefficient. Then, it is spliced ​​with the visual features of the spatial location to obtain the spliced ​​features of the spatial location.

5. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: Step S34 further includes a step of correcting the gating score, including: determining the text field corresponding to each expert branch according to a preset field-expert correspondence, and distributing the reliable grounding of the text field corresponding to each expert branch in spatial location. The value at that point is multiplied by the binding strength coefficient. Then, it is added to the gating score of the expert branch at the spatial location to obtain the corrected gating score. : ； In the formula, For the first k Each expert branch in spatial location The gate control score; For the first k The index of the text field corresponding to each expert branch; For the first The reliable grounding distribution of each text field is located in spatial location. The value at; The binding strength coefficient; After replacing the gated score in step S34 with the modified gated score, the modified gated score is... Softmax normalization with a temperature coefficient is performed to obtain the dynamic weights of each expert branch at each spatial location: ； In the formula, For the first k Each expert branch in spatial location Dynamic weights on; Temperature coefficient; K This refers to the number of expert branches.

6. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: The expression for the fusion adjustment coefficient mentioned in step S5 is: ； ； In the formula, The mean of the corrected consistency coefficients for each text field; This represents the upper limit of fusion strength. This is the sensitivity coefficient; The number of preset road condition fields; For the first i Corrected consistency coefficient for each text field.

7. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: Step S1, which involves inputting the island road condition image into a multimodal large model to generate road condition description text, includes the following steps: inputting the island road condition image and a preset island road condition prompt template into the multimodal large model to obtain road condition description text corresponding to the island road condition image; the island road condition prompt template is used to guide the multimodal large model to describe the island road condition image from at least two dimensions, namely, road spatial location, road morphology, environmental interference, traffic status, and boundary adjacency relationship; the road condition description text is structured text, which is used to characterize at least one of road area distribution, road boundary features, coastal background interference, and traffic anomaly information.

8. The island road condition image segmentation method based on generative text-driven hybrid experts according to claim 1, characterized in that: Step S5 further includes optimizing the parameters of the hybrid expert network and the segmentation-decoding network using a joint loss function, the expression of which is: ； In the formula, The Dice loss function is used to measure the degree of region overlap between the segmentation result and the true annotation. The cross-entropy loss function is used to measure the pixel-level classification error between the segmentation result and the true annotation. The orthogonality constraint loss is applied to the output features of each expert branch. , and These are the weighting coefficients.

Images