A language guide-based saliency and camouflage target unified detection method and system

Abstract

The application relates to the technical field of computer vision, and discloses a language-guided saliency and camouflage target unified detection method and system. The method takes a natural language description as a high-level semantic anchor point, injects text semantics into a visual space through bimodal coding and cross-modal fusion, and divides the fusion features into a language-guided segmentation branch and an attribute scoring branch. The segmentation branch locates a target region and repairs a boundary through multi-granularity saliency perception attention, and the attribute scoring branch dynamically calibrates text semantics through word-sentence matrix interaction and visual-sentence cross-attention, and scores the saliency of the target region in combination with continuous attribute regression. The application uniformly processes saliency targets, camouflage targets, intermediate states and coexisting scenes in a single framework, and simultaneously outputs a segmentation mask and a saliency score map.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a language-guided unified detection method and system for salient and camouflaged targets. Background Technology

[0002] Salient object detection (SOD) is a fundamental task in computer vision, aiming to locate and segment foreground objects in an image that are most likely to attract human visual attention. These methods typically utilize color, texture, brightness, spatial location, and high-level semantic differences between the object and the background to generate foreground masks.

[0003] Camouflaged object detection (COD) also falls under the category of foreground object extraction tasks, but its target objects are often highly integrated with the background. For example, the target may have similar color, texture, edge contours, or geometric structures to its surroundings. Therefore, camouflaged object detection places greater emphasis on fine-grained texture differences, weak boundary cues, and structural reasoning capabilities against complex backgrounds.

[0004] Visual foundational models (such as segmentation foundational models) can provide strong general visual representation capabilities. By freezing the visual backbone network and inserting low-rank adaptation (LoRA) matrices or adapter modules into its attention layers, downstream task adaptation can be achieved with a relatively low number of trainable parameters.

[0005] Visual-language models can leverage textual descriptions to provide high-level semantic priors for visual recognition tasks. Textual information can describe target category, quantity, color, texture, pose, spatial location, and the relationship between the target and the background. For unified detection tasks of salient and camouflaged targets, language descriptions can serve as cross-modal semantic guidance, helping the model focus on the true core target in complex scenes and reducing the influence of background noise and similar distracting objects.

[0006] Existing technical solutions mainly include a unified detection method for purely visual salience / camouflage, an image segmentation method based on visual basic model adaptation, and a visual-language segmentation method that incorporates text prompts.

[0007] Existing purely visual unified detection methods typically employ a unified encoder and decoder structure to extract features from the input image and directly output a foreground target mask or target attribute category. Its main steps include:

[0008] Step 1: Input the image to be detected, and perform scale normalization and preprocessing on the image.

[0009] Step 2: Extract multi-scale image features using a visual encoder.

[0010] Step 3: The decoder fuses visual features from different levels to generate a pixel-level foreground prediction map.

[0011] Step 4: Determine whether the current target is a salient target or a camouflaged target by identifying the attribute branch or category head.

[0012] Existing vision-language segmentation methods typically input images and text into a visual encoder and a text encoder, respectively, and then fuse visual and linguistic features through cross-modal attention or feature concatenation, finally outputting a target segmentation result related to the text description. Its main steps include:

[0013] Step 1: Obtain the image and a text description related to the target.

[0014] Step 2: Extract image features using a visual encoder and extract text semantic features using a text encoder.

[0015] Step 3: Establish the association between image regions and text words through the visual-language fusion module.

[0016] Step 4: Predict the target segmentation mask based on the fusion features.

[0017] The above-mentioned approach struggles to simultaneously segment and attribute-scoring salient targets, camouflaged targets, and intermediate-state targets within a unified scene. Specific drawbacks are as follows:

[0018] First, existing salient object detection models and camouflaged object detection models are typically trained for a single task. Since salient and camouflaged objects have opposite visual representations, single-task models develop specific feature preferences. Salient object detection models tend to look for high-contrast regions, thus easily missing detections in camouflaged scenes; camouflaged object detection models tend to look for weak texture differences and hidden boundaries, potentially introducing edge noise or falsely detecting background regions in salient scenes. Therefore, when both salient and camouflaged objects exist in the same scene, single-task models struggle to consistently achieve unified detection.

[0019] Second, existing pure vision-based unified detection models primarily rely on the color, texture, shape, and edge cues of the image itself. In scenarios with multiple instances coexisting and uneven feature distribution, the model tends to prioritize objects with the strongest visual response, while ignoring weaker instances that are still core targets. In scenarios with complex backgrounds or where distractions are similar to the target, pure vision models lack high-level semantic constraints, easily mistaking background textures for target boundaries, leading to missing target regions, fragmented boundaries, or attribute confusion.

[0020] Third, existing attribute discrimination methods mostly use hard labels in a salient / camouflaged binary classification. This method divides target attributes into discrete categories, failing to represent the continuous change of a target from salient to camouflaged. When a target has a certain visibility but is also blended into the background, hard labels make it difficult for the model to learn intermediate states, resulting in uneven scoring, unstable ranking, and affecting the model's ability to interpret semi-hidden targets.

[0021] Fourth, existing text-guided methods typically use text features as fixed semantic input, lacking a mechanism to dynamically update text semantics based on the current visual scene. When the text description cannot be adequately aligned with local targets, interfering backgrounds, or multi-target relationships in a complex visual scene, the ability of language information to assist in target localization is limited. Summary of the Invention

[0022] To address the aforementioned problems, the present invention aims to provide a language-guided unified detection method and system for salient and camouflaged targets, thereby improving the accuracy of core target localization, the integrity of segmentation boundaries, and the stability of explicit and implicit attribute scoring in complex open scenes. The technical solution is as follows:

[0023] A language-guided unified detection method for salient and camouflaged targets includes the following steps:

[0024] Step 1: Obtain the image to be detected and a natural language description related to the core target;

[0025] Step 2: Input the image to be detected into the visual encoder to extract multi-scale visual features; input the natural language description into the text encoder to extract word-level embeddings and sentence-level embeddings; the visual encoder uses a pre-trained segmentation base model as the backbone network, and inserts a low-rank adaptive matrix and adapter module into its attention layer for task adaptation;

[0026] Step 3: Input the multi-scale visual features, the word-level embeddings, and the sentence-level embeddings into the visual-language fusion module, and inject the text semantics into the visual space through cross-modal attention to obtain cross-modal fusion features;

[0027] Step 4: Split the cross-modal fusion features into the language-guided segmentation branch and the attribute scoring branch; wherein,

[0028] The language-guided segmentation branch uses a multi-granularity saliency-aware attention module to locate and repair the target region and generate a pixel-level segmentation mask. The multi-granularity saliency-aware attention module uses the word-level embedding as semantic guidance, performs attention aggregation on the multi-scale visual context, and extracts features of uncertain regions to repair missed target regions and boundary details.

[0029] Step 5: The attribute scoring branch inputs the sentence-level embedding and the word-level embedding into the language update module, and dynamically calibrates the text semantics through the following sub-steps:

[0030] Step 5.1: Perform self-attention enhancement on the sentence-level embedding to obtain enhanced sentence-level embedding;

[0031] Step 5.2: Perform matrix interaction between the enhanced sentence-level embedding and the word-level embedding to generate a transition matrix and a fine-grained attention matrix, and obtain the preliminarily updated sentence-level semantic vector based on the fine-grained attention matrix;

[0032] Step 5.3: Use the preliminarily updated sentence-level semantic vector as the query feature, and the deep visual features as the key and value features. Perform secondary calibration on the sentence-level semantic vector through visual-language cross-attention to obtain the language guidance feature.

[0033] Step 6: In the attribute scoring branch, the language-guided features and the visual features activated by the language-guided segmentation branch are input into the core target prediction head, and a core target segmentation mask is generated through matrix multiplication and upsampling;

[0034] Step 7: In the attribute scoring branch, based on the salient-camouflage continuous attribute regression, the visibility of the target region corresponding to the core target segmentation mask is scored, and a visibility score map is generated.

[0035] A language-guided unified detection system for salient and camouflaged targets includes:

[0036] A dual-modal input module is used to acquire the image to be detected and a natural language description related to the core target;

[0037] The dual-modal coding module includes a visual encoder and a text encoder. The visual encoder uses a pre-trained segmentation base model as the backbone network and inserts a low-rank adaptive matrix and an adapter module into its attention layer to extract multi-scale visual features. The text encoder is used to extract word-level embeddings and sentence-level embeddings.

[0038] The visual-language fusion module is used to inject textual semantics into the visual space through cross-modal attention to obtain cross-modal fusion features, and to branch the cross-modal fusion features to the language-guided segmentation branch and the attribute scoring branch;

[0039] The language-guided segmentation branch includes a language-guided segmentation module, which is used to locate and repair the target region by passing the cross-modal fusion features through a multi-granularity saliency-aware attention module, and generate a segmentation mask;

[0040] The attribute scoring branches include:

[0041] The language update module is used to perform self-attention enhancement on the sentence-level embedding and matrix interaction with the word-level embedding to obtain a preliminary updated sentence-level semantic vector. Then, the module uses the preliminary updated sentence-level semantic vector as the query feature and deep visual features as the key and value features to perform visual-language cross-attention calibration to obtain language guidance features.

[0042] The core target prediction module is used to fuse and predict the language-guided features with the visual features activated by the language-guided segmentation branch to generate a core target segmentation mask.

[0043] The attribute scoring module is used to score the visibility of a target region based on salient-camouflage continuous attribute regression and generate a visibility score map.

[0044] The beneficial effects of this invention are:

[0045] This invention addresses the problems of existing tasks involving the disconnect between salient target detection and camouflaged target detection, the susceptibility of pure visual models to complex background interference, and the difficulty in continuously characterizing the explicit and implicit attributes of targets. It introduces natural language description as a high-level semantic anchor and, through visual-language fusion, a multi-granularity saliency-aware attention module, and a language update strategy, enables the model to no longer rely solely on low-level visual cues such as color, texture, and edges when locating targets. Therefore, in scenarios with multiple instances coexisting, complex background textures, weak target visual cues, or the simultaneous presence of salient and camouflaged targets, this invention can constrain and calibrate visual features using target category, spatial relationships, local attributes, and explicit and implicit semantics, thereby reducing the probability of missed detections, false detections, and attribute confusion, and improving the completeness and boundary accuracy of core target segmentation.

[0046] Meanwhile, this invention extends the salience and camouflage attributes from traditional discrete classification to continuous regression modeling, and through intermediate state soft label supervision, enables the model to learn the smooth change pattern of the target from concealment and camouflage to visual salience. Therefore, for semi-concealed, weakly salient, or ambiguous targets, this invention can not only output pixel-level segmentation masks, but also provide a target visibility score map, improving the interpretability and ranking stability of the detection results. Since this invention can train only the low-rank adaptive matrix, adapter module, and task-related branches on the basis of freezing the visual / language base model, it reduces the number of training parameters and deployment costs while ensuring detection accuracy and robustness, making it suitable for applications such as target segmentation, visibility and camouflage attribute evaluation, and intelligent decision-making assistance in complex scenarios. Attached Figure Description

[0047] Figure 1 This is a diagram of the overall architecture for language-guided unified detection of salient and camouflaged targets.

[0048] Figure 2 This is a structural diagram of the multi-granularity saliency perception attention module.

[0049] Figure 3 A structural diagram of language update strategies. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0051] This invention aims to introduce high-level semantic anchors through natural language descriptions, enabling the model to correct visual features based on target category, quantity, texture, spatial relationships, and salient / camouflaged attribute descriptions, thereby reducing false negatives and false positives of pure visual models in complex backgrounds.

[0052] The present invention also aims to aggregate visual features from different receptive fields through a multi-granularity saliency-aware module (MGSM) and combine it with an uncertain region extraction strategy to repair missing target regions and boundary details, thereby improving the integrity of pixel-level segmentation results.

[0053] This invention also aims to dynamically update sentence-level semantic vectors based on word-level semantics and deep visual features through a Language Updating Strategy (LUS), enabling text semantics to adapt to the target state and scene interference in the current image, thereby enhancing visual-language co-localization capabilities.

[0054] This invention also aims to improve the model's ability to quantitatively predict the visibility of targets by using an intermediate state data supervision strategy to model salience and camouflage attributes as a continuous regression problem and construct a soft-label supervision signal consisting of camouflage, camouflage-side intermediate states, salience-side intermediate states, and salience states.

[0055] This invention provides a language-guided unified detection method for salient and camouflaged targets. The method takes the image to be detected and a natural language description as input, and outputs a core target segmentation mask, a target visibility score map, and corresponding visualized detection results. This method can be applied to scenarios such as autonomous driving, field exploration, intelligent monitoring, robot environmental perception, remote sensing image analysis, and target-assisted recognition in complex scenes.

[0056] like Figure 1As shown, the language-guided unified target detection framework proposed in this invention comprises a bimodal input module, a bimodal encoding module, a visual-language fusion module, a language-guided segmentation branch, an attribute scoring branch, and a result output module. The bimodal input module includes an image input terminal and a natural language description input terminal. The image input terminal is used to input the image of the scene to be detected, and the natural language description input terminal is used to input semantic prompts related to the core target. These semantic prompts may include target category, target quantity, color and texture, spatial location, and the salience or camouflage relationship between the target and the background.

[0057] (1) Dual-modal input module, used to acquire the image to be detected and natural language descriptions related to the core target.

[0058] (2) A dual-modal coding module, including a visual encoder and a text encoder. The visual encoder uses a pre-trained segmentation base model as the backbone network and inserts a low-rank adaptive matrix and an adapter module into its attention layer to extract multi-scale visual features. The text encoder is used to extract word-level embeddings and sentence-level embeddings.

[0059] (3) Visual-Language Fusion Module, which injects text semantics into visual space through cross-modal attention to obtain cross-modal fusion features, and diverts the cross-modal fusion features to language-guided segmentation branch and attribute scoring branch.

[0060] (4) The language-guided segmentation branch includes a language-guided segmentation module, which is used to locate and repair the target region by using the cross-modal fusion features through a multi-granularity saliency perception attention module, and generate a segmentation mask.

[0061] (5) Attribute scoring branches include:

[0062] 1) The language update module is used to perform self-attention enhancement on the sentence-level embedding and matrix interaction with the word-level embedding to obtain the initially updated sentence-level semantic vector, and to perform visual-language cross-attention calibration using the initially updated sentence-level semantic vector as the query feature and deep visual features as the key and value features to obtain the language guidance feature;

[0063] 2) Core target prediction module, used to fuse and predict the language guidance features with the visual features activated by the language guidance segmentation branch to generate a core target segmentation mask;

[0064] 3) Attribute scoring module, used to score the visibility of the target area based on salient-camouflage continuous attribute regression and generate a visibility score map.

[0065] The input image to be detected first enters the visual encoder. The visual encoder uses a pre-trained segmentation base model as the backbone network and inserts a low-rank adaptive matrix into the multi-head self-attention layer. It also incorporates an adapter module for efficient parameter adaptation for the unified saliency and camouflage detection task, thereby outputting multi-scale visual features. The input natural language description enters the text encoder and is converted into word-level embeddings and sentence-level embeddings. Word-level embeddings provide local target attributes and fine-grained semantic cues, while sentence-level embeddings provide global task semantic constraints. Subsequently, the multi-scale visual features, word-level embeddings, and sentence-level embeddings are jointly input into the vision-language fusion module, where the language prior is initially injected into the visual space, forming cross-modal fusion features.

[0066] The cross-modal fusion features are further branched into two task branches. The language-guided segmentation branch performs core target region discovery and repairs missed regions and boundary details through a multi-granularity saliency-aware attention module, ultimately outputting a pixel-level segmentation mask. The attribute scoring branch receives dynamically updated language semantic vectors and combines visual features to perform continuous regression modeling of target visibility, ultimately outputting a visibility score map. During training, the language-guided segmentation branch is supervised using weighted intersection-union loss and weighted binary cross-entropy loss, while the scoring branch is supervised using a Smooth L1 regression loss within the foreground region, and learns smooth attribute changes between saliency and camouflage using continuous soft labels composed of 0, 0.4, 0.6, and 1. Finally, this method outputs a target segmentation mask, a visibility score map, and a visual heatmap or detection report.

[0067] like Figure 2 As shown, the multi-granularity saliency-aware attention module is used for multi-scale localization and detail restoration of target regions under the guidance of linguistic semantics. The input to this module includes visual features from the previous stage, local visual features, and word-level linguistic embeddings. The visual features from the previous stage are first fed into a multi-scale average pooling unit, which uses pooling operations with multiple receptive fields to extract contextual information of different granularities, forming a multi-granularity pooled feature map. For the first decoding stage, since there is no visual input from the previous stage, the local visual features can be input into a dilated convolution initialization unit. Initial features are constructed through dilated convolutions with different kernel sizes, and then generated through concatenation and convolutional mapping to produce an initial visual representation.

[0068] Multi-granularity pooling feature maps are flattened and tokenized before being concatenated into a query vector. Simultaneously, word-level language embeddings are processed with self-attention to generate key and value vectors. The query vector, key vector, and value vector are then input into a multi-head cross-attention unit, allowing target categories, local attributes, and explicit / implicit semantics from the language description to be injected into the global visual labeling, thereby enhancing the visual features' ability to discriminate core targets. To address the issues of target region loss and boundary fragmentation, this module also includes an uncertain region extraction path: first, an initial activation map is generated from visual features; then, the inverse background attention map is multiplied with local visual features to extract uncertain region features located at target edges or weak response areas. Finally, the uncertain region features are fused with the global visual labeling to obtain a global attention map, which is then processed by convolution and activation functions to output an updated saliency prediction map. Through this structure, the module can suppress non-target noise in complex backgrounds and recover target details ignored by pure visual features.

[0069] like Figure 3 As shown, the language update strategy addresses the problem of fixed text semantics struggling to adapt to the current visual scene. The strategy's input includes sentence-level embeddings, word-level embeddings, and deep visual features. Sentence-level embeddings first enter a self-attention enhancement unit to supplement the global semantic context and form a more stable task-level semantic representation. Subsequently, sentence-level and word-level embeddings jointly enter a word-sentence matrix interaction unit, generating a transition matrix through matrix multiplication. This transition matrix is ​​used to decompose fine-grained semantic information such as target local attributes, spatial location, and explicit / implicit descriptions from the word-level embeddings. This transition matrix is ​​further used to calculate a fine-grained attention matrix with the sentence-level embeddings, resulting in a preliminarily updated sentence-level semantic vector.

[0070] After completing the semantic update at the word / sentence level, the updated sentence-level semantic vector is used as the query feature, and the deep visual features are used as key and value features, input into the vision-speech cross-attention unit. This unit performs secondary calibration of the language semantics based on the target appearance, background interference, and local response in the current image, making the language features more closely match the target to be detected. Finally, the calibrated language-guided features and the visual features activated by the language-guided segmentation branch are jointly input into the core target prediction head, and prediction results are generated through matrix multiplication and bilinear upsampling. Through this language update strategy, this invention can dynamically adjust the direction of text semantics in guiding the visual space when the target's saliency changes, the background texture is complex, or the target coexists significantly with a camouflaged target, thereby improving the stability of core target localization and attribute discrimination.

[0071] The method of the present invention constitutes a complete technical solution through the following step-by-step process.

[0072] Step 1: Image and speech dual-modal input.

[0073] This method receives the image to be detected. and natural language description Natural language descriptions can be input by the user or pre-generated by a large visual-language model and then manually verified. The text description includes, but is not limited to, target category, number of targets, color, texture, size, pose, spatial location, degree of integration between the target and background, and salience / camouflage attribute determination.

[0074] During the dataset construction phase, one or more text descriptions can be generated for each image. To ensure the accuracy of the text descriptions, manual review should be conducted in conjunction with pixel-level ground truth masks to remove incorrect target categories, incorrect target counts, misinterpretations of salient / camouflaged attributes, and background illusion descriptions. For intermediate state samples, degree descriptive words can be added to the text to ensure that the text semantics are consistent with soft label supervision.

[0075] Step Two: Visual Encoding and Text Encoding.

[0076] On the vision side, a pre-trained segmentation base model is used as the visual encoder. To reduce training costs and retain general visual knowledge, the backbone parameters of the visual encoder are frozen, with low-rank adaptive matrices inserted only into its multi-head self-attention layers, and an adapter module is used for task adaptation. The visual encoder outputs multi-scale visual features for subsequent cross-modal fusion and decoding prediction.

[0077] The text-side uses a text encoder to encode the natural language description and outputs word-level embeddings. and sentence-level embedding Word-level embeddings provide local attributes, positional relationships, and fine-grained semantic information of the target; sentence-level embeddings provide global semantic constraints at the sentence level. Visual and textual features are then fed into the visual-language fusion module to complete the initial semantic injection.

[0078] Step 3: Visual-Language Fusion.

[0079] The visual-language fusion module is used to establish the correspondence between image regions and text semantics. The input multi-scale visual features serve as visual spatial representations, while word-level embeddings and sentence-level embeddings serve as semantic spatial representations. Through cross-modal attention or equivalent feature interaction structures, text semantics are injected into visual features, enabling visual features to contain not only low-level information such as color, texture, and edges, but also high-level semantic information such as target category, target location, and target explicit / implicit attributes.

[0080] The purpose of this step is to provide clearer target semantic anchors for subsequent language-guided segmentation branches, and to reduce the interference of background noise on the localization of the core target.

[0081] Step 4: Multi-granularity saliency-aware attention segmentation.

[0082] The multi-granularity saliency perception attention module receives visual features from the previous stage. Word-level embedding and local visual features First, regarding visual features Perform average pooling at multiple scales to obtain multi-granularity pooled feature maps:

[0083] ;

[0084] In one specific embodiment, the average pooling receptive field size can be set to 4, 8, and 16 to capture contextual information at different scales. When this module is in the first decoding stage and lacks input features from the previous stage, it can focus on local visual features. Multiple sets of dilated convolutions were applied, with kernel sizes of [sizes to be filled in]. , and Then perform feature splicing and Convolution generates initial visual features.

[0085] Subsequently, the pooled feature maps are flattened and labeled, and then concatenated into a query vector for multi-head cross-attention. :

[0086] ;

[0087] in, This indicates a tokenization operation. Represents a linear transformation matrix. Word-level embedding. Key vectors are generated after self-attention processing. Sum value vector Global visual tags for injecting language-discriminative semantics are generated through multi-head cross-attention. :

[0088] ;

[0089] in, This indicates multiple points of cross-attention. Representation layer normalization, This represents a feedforward network.

[0090] To further repair missing target regions, this module extracts uncertain areas of the target. First, visual features... through Convolution and activation operations generate the initial activation map Then through a reverse mask With local visual features Multiplying yields the characteristics of the uncertain region. Finally, the characteristics of the uncertain region will be determined. With global visual markers By merging, a global attention map is obtained. :

[0091] ;

[0092] in, These are trainable parameters used to adaptively adjust the activation intensity of salient regions. express Convolution operation, This indicates element-wise multiplication. Indicates an upsampling operation; This is a saliency prediction map. Global attention map. An updated saliency prediction map is generated after convolution and sigmoid activation. This results in a more complete target segmentation outcome.

[0093] Step 5: Language Updates and Core Goal Prediction.

[0094] A language update strategy is used to dynamically update sentence-level embeddings, adapting the text semantics to the target state in the current image. First, the sentence-level embeddings are... Self-attention enhancement is performed to capture global semantic information. Then sentence-level embeddings are applied. With word-level embedding Perform matrix interactions to generate a transition matrix and a fine-grained attention matrix. This is used to decompose fine-grained information in word embeddings. Next, a fine-grained attention matrix is ​​obtained by calculating the product of the transition matrix and the transpose of the sentence embedding. This reflects the focus of the sentence embedding. Finally, the attention matrix and transition matrix are combined to generate an updated sentence embedding matrix. :

[0095] ;

[0096] in, Represents the learnable parameter matrix, Represents matrix multiplication. This represents the sentence-level semantic vector after the initial update.

[0097] Subsequently, the updated sentence-level semantic vector is used as the query vector, and the deep visual features are used as the key and value vectors, respectively. Fine-grained alignment of the visual and semantic features is achieved through cross-attention. The final generated language guidance features... Visual features of language-guided segmentation branch activation The core target prediction head is input together to generate prediction results. :

[0098] ;

[0099] in, Represents matrix multiplication. This represents a bilinear interpolation operation with a scaling factor of 4. This step dynamically calibrates the semantics of the language based on visual feedback, improving the ability to identify core targets in scenarios where salience and camouflage targets coexist.

[0100] Step 6: Monitoring and scoring of intermediate state data and visibility.

[0101] In order to enable the model to predict the continuous degree of visibility of a target from camouflage to salience, this invention models attribute scoring as a regression task and constructs intermediate state samples.

[0102] First, morphological operations are used to obtain the foreground region based on the foreground mask and the background region. and background area :

[0103] ;

[0104] in, This represents the foreground mask obtained through etching. This represents the background mask obtained after dilation processing. The input image is represented. A block search strategy is used to retrieve similar texture blocks from the background region and generate a foreground reconstruction image. Then, the texture reconstruction fidelity score is calculated:

[0105] ;

[0106] in, Indicates the number of pixels in the foreground region. This represents a function that determines the local similarity between the reconstructed result and the original foreground. This represents the texture reconstruction fidelity score calculated from the input image I under the ground truth mask M. If the target foreground can be reconstructed from background image patches with high quality, it indicates that the target and background are highly integrated and have strong camouflage capabilities.

[0107] Secondly, construct the boundary transition mask:

[0108] ;

[0109] Extract the predicted contour C and the true contour using an edge detection network. Calculate the boundary visibility score:

[0110] ;

[0111] in, This represents the contour consistency evaluation function. This represents the boundary visibility score calculated for the input image I under the truth mask M. If... A higher value indicates that the target boundary is more difficult to recover from background noise.

[0112] Finally, the texture reconstruction fidelity and boundary visibility are weighted and fused to obtain the comprehensive perceptual camouflage score:

[0113] ;

[0114] in, As a weighting factor, in one embodiment it can be taken as... .

[0115] For the camouflaged target training set, a dual screening process is used, combining the perceived camouflage score and the back-prediction error of the salient target detection model. Samples with a significant tendency but still retaining camouflage features are selected as camouflage-side intermediate state data, and a soft label of 0.4 is assigned. For the salient target training set, samples with a certain degree of camouflage or recognition difficulty are selected as salient-side intermediate state data, using both the perceived camouflage score and the model prediction error, and a soft label of 0.6 is assigned. Finally, camouflaged samples, camouflage-side intermediate samples, salient-side intermediate samples, and salient samples constitute a continuous supervision signal with labels of 0, 0.4, 0.6, and 1.

[0116] Step 7: Loss Function and Training Method.

[0117] Language-guided splitting branches use binary truth graphs For prediction mask Supervision is performed. To balance pixel-level accuracy and global structural consistency, weighted intersection-over-union loss and weighted binary cross-entropy loss are employed:

[0118] ;

[0119] The attribute scoring branch uses Smooth L1 loss for regression supervision. Let the predicted score map output by the scoring branch be... The corresponding attribute soft tag is The rating loss is:

[0120] ;

[0121] By comparing with the binary truth map Multiplication is performed, and the scoring loss is calculated only in the foreground object region to avoid interference from the background region in the modeling of explicit and implicit attributes. The overall training loss is:

[0122] .

[0123] In one embodiment, a phased training strategy can be adopted: first, the language-guided segmentation branch is trained, enabling the model to fully learn the core target localization ability under language guidance; then, the parameters or some parameters of the language-guided segmentation branch are fixed, and the attribute scoring branch is trained separately, enabling it to learn the smooth prediction of the explicitness and implicitness of the target. The training images can be uniformly adjusted to... Data augmentation is performed using random horizontal and vertical flipping; during the testing phase, the image can be adjusted to... The output will be restored to the original image size.

[0124] Method output and application implementation:

[0125] This invention provides a unified detection method for both salient and camouflaged targets. In practical applications, the image to be detected and a natural language description can be used as inputs. The method then undergoes processing steps including visual encoding, text encoding, visual-language fusion, language-guided segmentation, dynamic language updates, and attribute scoring. Finally, a target segmentation mask, a salience / camouflage score map, and the corresponding visualization results are obtained.

[0126] The target segmentation mask is used to indicate the pixel-level location of the core target in the image; the visibility score map is used to characterize the continuous attribute change of the target from camouflage to prominence; the visualization results can generate heatmaps or detection reports based on the segmentation mask and visibility score map, which can be used to assist in the analysis of target discovery, target attribute evaluation, and subsequent decision-making in complex scenes. This method is not limited to a specific operating platform and can be implemented in a computing environment with image processing and model inference capabilities.

[0127] In addition to the specific embodiments described above, the overall technical concept of this invention can also be implemented using other equivalent technical solutions. All such solutions, without departing from the core concept of this invention, fall within the scope of protection of this invention. For the vision and language encoding parts, the pre-trained segmentation base model, low-rank adaptive matrix, adapter module, and text encoder used in this invention can be replaced with other network structures capable of image feature extraction, text semantic encoding, and efficient parameter adaptation. Examples include visual Transformer networks, convolutional neural networks, hybrid vision backbone networks, other vision-language model text branches, cue learning structures, prefix tuning structures, gating fine-tuning structures, or partial-layer fine-tuning structures. These are used to achieve unified feature representation and task adaptation between the image to be detected and the natural language description.

[0128] For the multi-granularity saliency perception part, the multi-scale average pooling, dilated convolution initialization, word-level language cross-attention, and uncertain region extraction mechanisms used in this invention can be replaced with spatial pyramid pooling, multi-scale convolution, window attention, deformable attention, gated attention, or other feature interaction structures that can simultaneously model global context, local boundary details, and linguistic semantic constraints. These can be used to locate core target regions, repair missed detection regions, and enhance boundary details in complex backgrounds. For the language update part, the word-sentence matrix interaction and visual feedback cross-attention mechanism used in this invention can be replaced with bidirectional attention, graph attention, dynamic cue selection, semantic memory update, or other cross-modal semantic calibration structures. These can be used to achieve dynamic enhancement and adaptive updating of text semantic information as the current visual scene changes.

[0129] For the continuous supervision of explicit and implicit attributes, the method of filtering intermediate state samples based on texture reconstruction fidelity, boundary visibility, and model prediction error used in this invention can be replaced by structural similarity, feature distance, edge confidence, attention entropy, manually ranked labels, or other evaluation methods that can characterize the degree of fusion between the target and the background, the degree of boundary visibility, and the difficulty of model recognition, to construct a continuous supervision signal between salience and camouflage. The Smooth L1 regression loss used in the attribute scoring branch can also be replaced by L1 loss, L2 loss, Huber loss, ranking loss, or a combination thereof. As long as the model can be trained to output a target explicitness / implicitness score map, it can be considered an equivalent implementation of this invention.

[0130] Furthermore, this invention is not only applicable to the unified detection of salient and camouflaged targets in ordinary RGB images, but can also be extended to core target detection, segmentation, and explicit / implicit attribute scoring tasks in video, remote sensing images, medical images, RGB-Depth images, multispectral images, infrared images, and other multi-source visual data. Therefore, any equivalent substitutions or modifications made based on the core ideas of this invention to the network backbone structure, visual-language feature interaction methods, multi-granularity perception structures, language dynamic update mechanisms, intermediate state supervision methods, or applied data types should be considered to fall within the protection scope of this invention.

Claims

1. A unified detection method for salient and camouflaged targets based on language guidance, characterized in that, Includes the following steps: Step 1: Obtain the image to be detected and a natural language description related to the core target; Step 2: Input the image to be detected into the visual encoder to extract multi-scale visual features; input the natural language description into the text encoder to extract word-level embeddings and sentence-level embeddings; the visual encoder uses a pre-trained segmentation base model as the backbone network, and inserts a low-rank adaptive matrix and adapter module into its attention layer for task adaptation; Step 3: Input the multi-scale visual features, the word-level embeddings, and the sentence-level embeddings into the visual-language fusion module, and inject the text semantics into the visual space through cross-modal attention to obtain cross-modal fusion features; Step 4: Split the cross-modal fusion features into the language-guided segmentation branch and the attribute scoring branch; wherein, The language-guided segmentation branch uses a multi-granularity saliency-aware attention module to locate and repair the target region and generate a pixel-level segmentation mask. The multi-granularity saliency-aware attention module uses the word-level embedding as semantic guidance, performs attention aggregation on the multi-scale visual context, and extracts features of uncertain regions to repair missed target regions and boundary details. Step 5: The attribute scoring branch inputs the sentence-level embedding and the word-level embedding into the language update module, and dynamically calibrates the text semantics through the following sub-steps: Step 5.1: Perform self-attention enhancement on the sentence-level embedding to obtain enhanced sentence-level embedding; Step 5.2: Perform matrix interaction between the enhanced sentence-level embedding and the word-level embedding to generate a transition matrix and a fine-grained attention matrix, and obtain the preliminarily updated sentence-level semantic vector based on the fine-grained attention matrix; Step 5.3: Use the preliminarily updated sentence-level semantic vector as the query feature, and the deep visual features as the key and value features. Perform secondary calibration on the sentence-level semantic vector through visual-language cross-attention to obtain the language guidance feature. Step 6: In the attribute scoring branch, the language-guided features and the visual features activated by the language-guided segmentation branch are input into the core target prediction head, and a core target segmentation mask is generated through matrix multiplication and upsampling; Step 7: In the attribute scoring branch, based on the salient-camouflage continuous attribute regression, the visibility of the target region corresponding to the core target segmentation mask is scored, and a visibility score map is generated.

2. The language-guided unified detection method for salient and camouflaged targets according to claim 1, characterized in that, In step 4, the specific process of the multi-granularity saliency perception attention module performing attention aggregation on multi-scale visual context is as follows: Step 4.1: The multi-granularity saliency perception attention module receives visual features from the previous stage. Word-level embedding and local visual features ; Visual features of the previous stage Perform average pooling at multiple scales to obtain multi-granularity pooled feature maps: ； in, , This is a multi-granularity pooling feature map; , Average pooling across multiple scales; Step 4.2: In the first decoding stage, when visual features from the previous stage are missing, multiple sets of dilated convolutions with different kernel sizes are applied to the local visual features. After feature concatenation and convolution mapping, initial visual features are generated to replace the visual features from the previous stage. The multi-granularity pooling feature maps are flattened and tokenized, and then concatenated into a query vector. : ； in, This indicates a tokenization operation. Represents a linear transformation matrix; Step 4.3: Embed the word-level words After self-attention processing, it becomes a key vector. Sum value vector Global visual tags for injecting language-discriminative semantics are generated through multi-head cross-attention. : ； in, This indicates multiple points of cross-attention. Representation layer normalization, This represents a feedforward network.

3. The language-guided unified detection method for salient and camouflaged targets according to claim 2, characterized in that, The specific process of extracting features of uncertain regions to repair missed target regions and boundary details in step 4 is as follows: Visual features through Convolution and activation operations generate the initial activation map Then through a reverse mask With local visual features Multiplying yields the characteristics of the uncertain region. Finally, the features of the uncertain region are compared with global visual labels. By merging, a global attention map is obtained. : ； in, These are trainable parameters used to adaptively adjust the activation intensity of salient regions. express Convolution operation, This indicates element-wise multiplication. Indicates an upsampling operation; For saliency prediction; global attention map An updated saliency prediction map is generated after convolution and sigmoid activation. This results in a more complete target segmentation outcome.

4. The language-guided unified detection method for salient and camouflaged targets according to claim 1, characterized in that, The specific process of matrix interaction described in step 5.2 is as follows: For the sentence-level embedding Perform self-attention enhancement to capture global semantic information; Then embed sentence levels With word-level embedding Perform matrix interactions to generate a transition matrix and a fine-grained attention matrix. This is used to decompose fine-grained information in word embeddings; then, a fine-grained attention matrix is ​​obtained by calculating the product of the transition matrix and the transpose of the sentence embedding. This reflects the focus of sentence embedding; finally, the attention matrix and transition matrix are combined to generate an updated sentence embedding matrix, i.e., sentence-level semantic vectors. : ； in, Represents the learnable parameter matrix, Represents matrix multiplication. This represents the sentence-level semantic vector after the initial update.

5. The language-guided unified detection method for salient and camouflaged targets according to claim 1, characterized in that, In step 5.3, the specific process of secondary calibration through visual-verbal cross-attention is as follows: The updated sentence-level semantic vector is used as the query vector, and the deep visual features are used as the key and value vectors. Cross-attention is used to achieve fine alignment of visual and semantic features; the final generated language guidance features are... Visual features of language-guided segmentation branch activation The core target prediction head is input together to generate prediction results. : ； in, Represents matrix multiplication. This represents a bilinear interpolation operation with a scaling factor of 4.

6. The language-guided unified detection method for salient and camouflaged targets according to claim 1, characterized in that, The salient-masked continuous attribute regression described in step 7 includes: Step 7.1: Extract the foreground region from the foreground mask and background region using morphological operations. and background area ; ； in, This represents the foreground mask obtained through etching. This represents the background mask obtained after dilation processing. Indicates the input image; Step 7.2: Retrieve similar texture blocks from the background region based on a block search strategy to generate a foreground reconstruction image. Calculate the texture reconstruction fidelity score ; ； in, Indicates the number of pixels in the foreground region. A function representing the local similarity determination between the reconstructed result and the original foreground; This represents the texture reconstruction fidelity score calculated for the input image I under the truth mask M; Step 7.3: Construct the boundary transition mask : ； Extract the predicted contour C and the true contour using an edge detection network. Calculate the boundary visibility score ; ； in, This represents the contour consistency evaluation function; This represents the boundary visibility score calculated for the input image I under the truth mask M; Step 7.4: Weight and fuse the texture reconstruction fidelity score and the boundary visibility score to obtain the comprehensive perceptual camouflage score. : ； in, This is the weighting factor.

7. The language-guided unified detection method for salient and camouflaged targets according to claim 6, characterized in that, The salient-masked continuous attribute regression uses a four-valued soft-label sequence {0, 0.4, 0.6, 1} as the supervision signal; where: For the training set of camouflaged targets, the comprehensive perception camouflage score and the reverse prediction error of the salient target detection model are used for dual screening. Samples with significant tendency and camouflage features are selected as intermediate state data of the camouflage side, and a soft label of 0.4 is set. For the salient target training set, the comprehensive perception camouflage score and model prediction error are used for dual screening to select samples with camouflage or recognition difficulty as salient side intermediate state data, and a soft label of 0.6 is set. The camouflaged samples, camouflaged side intermediate samples, salient side intermediate samples, and salient samples are respectively assigned soft labels 0, 0.4, 0.6, and 1 to form a continuous supervision signal. The Smooth L1 loss is used to perform regression supervision on the visibility score map, and the regression supervision is only applied to the foreground target region.

8. The language-guided unified detection method for salient and camouflaged targets according to claim 1, characterized in that, The pixel-level segmentation mask generated in step 4 and the visibility score map generated in step 7 are output simultaneously by a single forward inference, forming a dual-branch collaborative prediction structure; the method can handle salient targets, camouflaged targets, scenes where salient and camouflaged coexist, and intermediate state targets simultaneously without manual switching of task modes.

9. The language-guided unified detection method for salient and camouflaged targets according to claim 1, characterized in that, The method employs the following combination of loss functions during the training phase: Language-guided splitting branches use binary truth graphs For prediction mask Supervision was conducted using a weighted average loss ratio. Weighted binary cross-entropy loss The segmentation mask is monitored; segmentation loss. As shown in the following formula: ； The attribute scoring branch uses Smooth L1 loss for regression supervision. Let the predicted score map output by the scoring branch be... The corresponding attribute soft tag is Then the score loss for: ； Total loss for: 。 10. A unified detection system for salient and camouflaged targets based on language guidance, characterized in that, include: A dual-modal input module is used to acquire the image to be detected and a natural language description related to the core target; The dual-modal coding module includes a visual encoder and a text encoder. The visual encoder uses a pre-trained segmentation base model as the backbone network and inserts a low-rank adaptive matrix and an adapter module into its attention layer to extract multi-scale visual features. The text encoder is used to extract word-level embeddings and sentence-level embeddings; The visual-language fusion module is used to inject textual semantics into the visual space through cross-modal attention to obtain cross-modal fusion features, and to branch the cross-modal fusion features to the language-guided segmentation branch and the attribute scoring branch; The language-guided segmentation branch includes a language-guided segmentation module, which is used to locate and repair the target region by passing the cross-modal fusion features through a multi-granularity saliency-aware attention module, and generate a segmentation mask; The attribute scoring branches include: The language update module is used to perform self-attention enhancement on the sentence-level embedding and matrix interaction with the word-level embedding to obtain a preliminary updated sentence-level semantic vector. Then, the module uses the preliminary updated sentence-level semantic vector as the query feature and deep visual features as the key and value features to perform visual-language cross-attention calibration to obtain language guidance features. The core target prediction module is used to fuse and predict the language-guided features with the visual features activated by the language-guided segmentation branch to generate a core target segmentation mask. The attribute scoring module is used to score the visibility of a target region based on salient-camouflage continuous attribute regression and generate a visibility score map.

Images