A semantic and spatial mutual driving open vocabulary camouflage target segmentation method, system, device and medium

Abstract

The application discloses a semantic and spatial mutual driving open vocabulary camouflage target segmentation method and system, equipment and medium, wherein the method comprises the following steps: constructing a learnable text prompt module to generate candidate category semantic information; then, based on an initial segmentation mask, a category rearranger with spatial structure perception is used to reorder the candidate categories to obtain refined category semantics; further, a segmentation modulator based on semantic context is used to inject the refined category semantics into a segmentation model to realize iterative optimization of category semantics and spatial structure information, and finally, target category prediction results and corresponding segmentation results are output. The application can effectively reduce semantic bias caused by background interference, and improve the recognition accuracy, segmentation precision and generalization ability of unseen category camouflage targets under the open vocabulary condition.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to an open-vocabulary camouflaged target segmentation method, system, device and medium that are driven by semantics and space. Background Technology

[0002] Camouflaged target segmentation aims to segment targets that are highly similar to the background from complex images, and has important application value in fields such as object detection, scene understanding, and intelligent perception. Most existing camouflaged target segmentation methods are based on closed-class settings, meaning that the same class space is shared during the training and testing phases. Therefore, when faced with new class targets not present in the training set, the model's generalization ability is limited.

[0003] With the development of visual language models, open-vocabulary semantic segmentation technology leverages the cross-modal alignment capability between image and text features, enabling models to identify and segment open categories based on free text descriptions. Existing open-vocabulary segmentation methods generally perform category semantic matching first, then combine the segmentation model to output a target mask, or simultaneously perform category prediction and region segmentation within a single-stage framework. While these methods are effective in general scenarios, when applied to camouflaged scenarios, the high similarity between the foreground target and the background in texture, color, and structure makes semantic recognition and spatial localization susceptible to background interference, leading to semantic confusion and inaccurate spatial segmentation.

[0004] In open-vocabulary camouflage target segmentation tasks, existing methods typically inject category semantics as a one-time prior into the segmentation process, lacking a bidirectional interaction module between category semantics and spatial structure. Since the category determination and spatial localization of camouflaged targets are strongly coupled, any deviation in the initial category prediction can be amplified in subsequent segmentation stages, affecting the completeness and accuracy of the final segmentation result. Furthermore, existing methods often lack a process for reverse correction of category semantics based on spatial masks, making it difficult to achieve continuous semantic correction and spatial refinement in complex camouflage scenarios.

[0005] Therefore, how to construct an open lexical masquerading target segmentation technology that can achieve bidirectional feedback between category semantics and spatial structure, and iteratively optimize category prediction results and spatial segmentation results, has become an urgent technical problem to be solved in this field. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to construct an open lexical masquerading target segmentation method, system, device and medium that can realize bidirectional feedback of category semantics and spatial structure, and iteratively optimize the category prediction results and spatial segmentation results.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a semantically and spatially driven open-lexical masquerading target segmentation method, comprising the following steps: Obtain the input image to be segmented and the category text; The input image is fed into an open-vocabulary camouflage target segmentation model for prediction, and the corresponding target category prediction result and camouflage target segmentation result are output. The open-vocabulary camouflage target segmentation model includes a learnable text prompt module, an initial segmentation model, a category rearranger, a segmentation modulator, and an iterative refinement module. The open-vocabulary camouflage target segmentation model performs predictions and outputs corresponding target category prediction results and camouflage target segmentation results, specifically including the following operation steps: Obtain the category name of the category text and construct a learnable text prompt module for camouflage scenarios. The learnable text prompt module adaptively reconstructs the category text expression and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented to generate candidate category semantic information. The input image to be segmented is fed into the initial segmentation model to model the target region and generate an initial segmentation mask corresponding to the semantic information of the candidate category; The candidate category semantic information, the initial target segmentation mask, and the image visual features extracted by the visual language model are input into the spatial structure-aware category reorderer to reorder the candidate categories and obtain refined category semantics. The refined category semantics are injected as modulation information into the initial segmentation model to guide the segmentation of the target region and generate an updated segmentation mask. The updated segmentation mask is fed back to the iterative refinement module, and the iterative refinement module and segmentation modulator are repeatedly executed until the preset iteration termination condition is met. Finally, the target category prediction result and its corresponding camouflaged target segmentation result are output.

[0008] Preferably, as one possible implementation, the learnable prompting module adaptively reconstructs the category text representation and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented, generating candidate category semantic information, specifically including the following steps: Get the category name from the category text; A learnable text prompt module for camouflage scenarios is constructed, wherein the learnable text prompt module includes multiple learnable context vectors and category word embeddings corresponding to category names; The category text representations reconstructed by the learnable text prompting module are input into the text encoder of the visual language model to obtain the text semantic feature representations corresponding to each candidate category. The input image is fed into the image encoder of the visual language model to extract the visual feature representation corresponding to the input image; Based on the similarity between the textual semantic feature representations corresponding to each candidate category and the visual feature representations corresponding to the input image, open-vocabulary semantic matching is performed to obtain the semantic information of the candidate categories.

[0009] Preferably, as one possible implementation, the initial segmentation model models the target region and generates an initial segmentation mask corresponding to the semantic information of the candidate categories, specifically including the following steps: The input image is fed into the segmentation model to perform preliminary spatial modeling of the target region in the image and generate initial segmentation features. Based on the initial segmentation features, the segmentation model outputs the corresponding initial segmentation mask for the target. The segmentation model is Segment Anything Model 2, and the initial target segmentation mask is used to characterize the initial spatial distribution of the camouflaged target region in the input image. The initial segmentation mask of the target is associated with the candidate category semantic information generated in the semantic construction stage, and used as the spatial structure prior of the subsequent iterative refinement module.

[0010] Preferably, as one possible implementation, the category reordering device re-orders candidate categories to obtain refined category semantics, specifically including the following steps: Image encoders using visual language models extract normalized visual features from input images. and the normalized visual features Similarity is calculated between the text semantic features corresponding to the candidate categories to obtain category-level prediction scores. The calculation formula is as follows: ;in, This represents a category semantic matrix composed of the textual semantic features of candidate categories; Select the category-level predicted score with the highest confidence level. There are 10 candidate categories, forming a candidate category set. ; Visual features of a pre-defined intermediate layer are extracted from the image encoder of a visual language model and then pooled to obtain the intermediate layer visual representation. ; against Each candidate category in the candidate categories Based on its prediction confidence, relative ranking position and its correlation with Construct a ranking feature vector based on the score differences between candidate categories: .

[0011] The sorting feature vector Intermediate layer visual representation And the spatial mask representing the initial segmentation mask of the target generated by the initial segmentation model. The common input spatial structure-aware category reorderer yields the reordering score for the corresponding candidate categories, calculated using the following formula: ;in, This represents a category reorderer, based on the reordering scores corresponding to each candidate category. right The candidate categories are reordered to obtain refined category semantics.

[0012] Preferably, as one possible implementation, the segmentation modulator, in performing guided segmentation of the target region by injecting the refined category semantics as modulation information into the initial segmentation model and generating an updated segmentation mask, specifically includes the following steps: Extracting intermediate visual features from the image encoder output of the segmentation model and the textual semantic features corresponding to the candidate categories The gating visual features are obtained by adaptive filtering and alignment using visual gating units and semantic gating units, respectively. and gated semantic features It is represented as: , The gated visual features With gated semantic features The fusion process is performed to obtain the fusion features, calculated using the following formula: ; Based on a multi-head attention mechanism, semantic-spatial modulation is applied to the fused features to obtain attention-enhanced features, calculated using the following formula: Meanwhile, through gating visual features With gated semantic features Bilinear interaction yields cross-modal correlation graphs The attention-enhancing features are then semantically consistent and reweighted using the cross-modal correlation graph to obtain semantically enhanced features. The calculation formula is as follows: The refined category semantics output by the category reorderer are then encoded into category-level semantic vectors. And based on the category-level semantic vector The semantic enhancement features Class conditional modulation is performed to obtain class consistency enhancement features, calculated using the following formula: ; The attention enhancement feature Enhanced consistency with categories Weighted fusion is performed, where the fusion weights are determined by learnable semantic assignment parameters, and the calculation formula is as follows: and modulated features Input the mask decoder of the segmentation model to generate an updated segmentation mask.

[0013] Preferably, as one possible implementation, the execution process of the iterative refinement module specifically includes the following steps: Based on the input image Initialize the semantic state of the candidate category set The target initial segmentation mask generated by the initial segmentation model is used as the initial spatial state. The initial semantic state is obtained by weighting and aggregating the text embeddings of candidate categories according to their corresponding confidence scores, and the calculation formula is as follows: ,in, Indicates candidate category Text embedding, This represents the confidence weight for the corresponding candidate category.

[0014] In the t-th iteration, the current space state is... Semantic information of candidate categories and visual features of the input image The spatial structure-aware category rearranger in the common input iterative refinement module updates the semantic state, resulting in the updated semantic state. The calculation formula is: ,in, This represents the semantic update operator corresponding to the category reorderer. Represents the set of candidate categories; Updated semantic state The semantic context-based segmentation modulator in the input segmentation modulator updates the spatial state to obtain the updated segmentation mask. The calculation formula is: ,in, This represents the spatial update operator corresponding to the segmented modulator. The input image is represented; semantic state update and spatial state update are repeatedly performed until the preset iteration termination condition is met, and the target category prediction result and its corresponding camouflaged target segmentation result are output according to the final semantic state and the final spatial state.

[0015] This invention provides an open-lexical spoofing target segmentation system driven by semantics and space, comprising: A semantic construction module is used to obtain the category name of the category text and construct a learnable text prompt module for camouflage scenarios. The learnable text prompt module adaptively reconstructs the category text expression and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented to generate candidate category semantic information. The initial segmentation module is used to feed the input image to be segmented into the initial segmentation model to model the target region and generate an initial segmentation mask corresponding to the semantic information of the candidate category; The category refinement module is used to construct a spatial structure-aware category reorderer. It uses the regional spatial structure information and image visual features represented by the initial segmentation mask of the target to reorder the candidate categories and obtain refined category semantics. The segmentation modulation module is used to construct a segmentation modulator based on semantic context, inject the refined category semantics as modulation information into the segmentation model, guide the segmentation of the target region, and generate an updated segmentation mask. The iterative optimization module is used to feed back the updated segmentation mask to the category refinement module, and repeatedly perform category refinement and segmentation modulation until the preset iteration termination condition is met, and output the target category prediction result and its corresponding camouflaged target segmentation result.

[0016] Furthermore, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program to enable the electronic device to perform a semantically and spatially driven open-vocabulary masquerading target segmentation method as described in any of the preceding claims.

[0017] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the semantically and spatially driven open-vocabulary masquerading target segmentation method described in any of the preceding claims.

[0018] The beneficial effects of this invention are as follows: This invention addresses the challenges of collaborative modeling of category semantics and spatial structure in open-vocabulary camouflage target segmentation, and the lack of bidirectional feedback mechanisms in existing methods which typically employ one-time semantic injection. It proposes an open-vocabulary camouflage target segmentation method based on a semantic-spatial mutual-driven mechanism. By constructing a learnable text prompt module, a spatial structure-aware category rearranger, and a semantic context-based segmentation modulator, a closed-loop iterative optimization between category semantics and spatial structure information is achieved. Compared to existing technologies, this invention can continuously correct early semantic judgment results using spatial structure information formed during segmentation. Simultaneously, it uses refined category semantics to guide the segmentation model to generate more accurate target masks, effectively mitigating semantic confusion, strong background interference, and blurred target boundaries in camouflage scenarios. This improves category recognition accuracy, target localization ability, and segmentation completeness under unseen category conditions. Furthermore, the iterative optimization mechanism proposed in this invention gradually enhances the consistency between semantic and spatial representations, improves the stability of mask and category label alignment, and makes the model more robust, generalizable, and practical in complex camouflage scenarios. Attached Figure Description

[0019] To more clearly illustrate the implementation of the present invention or existing technical solutions, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. The drawings mainly illustrate various embodiments of the present invention by way of example rather than limitation, and the implementation of the present invention is described in conjunction with the claims and specification. Where appropriate, the same or similar parts in the drawings are represented by the same reference numerals. It should be noted that these embodiments are merely illustrative and are not intended to exhaustively or exclusively limit all possible implementations of this device or method.

[0020] Figure 1 A schematic diagram of the overall framework of an open lexical camouflage target segmentation method driven by semantics and space provided in an embodiment of this application; Figure 2 A flowchart illustrating an open-lexical camouflage target segmentation method driven by semantics and space, provided for embodiments of this application; Figure 3 A schematic diagram of the category rearranger structure in an open lexical camouflage target segmentation method driven by semantics and space provided in an embodiment of this application; Figure 4 A schematic diagram of the segmentation modulator structure in an open vocabulary camouflage target segmentation method driven by semantics and space provided in an embodiment of this application; Figure 5 A schematic diagram showing a qualitative comparison between a semantically and spatially driven open-vocabulary camouflage target segmentation method provided in this application embodiment and the existing state-of-the-art methods; Figure 6 This is a schematic diagram of the image to be segmented for an open vocabulary camouflage target segmentation method driven by semantics and space in an embodiment of this application. Figure 7 This is a pixel-level label diagram of an open vocabulary camouflage target segmentation method driven by semantics and space in an embodiment of this application; Figure 8 This is a schematic diagram of the open-vocabulary camouflaged target segmentation results of the existing method CAT-Seg. Figure 9 A schematic diagram of the open-vocabulary camouflaged target segmentation results of the existing method SAN; Figure 10 This is a schematic diagram of the open-vocabulary masquerading target segmentation results of the existing method OVCoser.

[0021] Figure 11 This is a schematic diagram of the open vocabulary camouflage target segmentation result obtained by using the present invention in the embodiments of this application; Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] This application provides a semantically and spatially driven open-lexical camouflage target segmentation method, such as... Figures 1 to 4 As shown, the method includes a learnable text prompt module, an initial segmentation model, a category rearranger, a segmentation modulator, and an iterative refinement module. Based on a visual language model and a segmentation model, this method establishes a bidirectional feedback relationship between category semantics and spatial structure by introducing a learnable text prompt module, a spatial structure-aware category rearranger, and a semantic context-based segmentation modulator. This enables the collaborative optimization of category prediction and segmentation results for open-vocabulary camouflage targets.

[0025] S101: Obtain the category name of the category text and construct a learnable text prompt module for camouflage scenes. The learnable text prompt module adaptively reconstructs the category text expression and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented to generate candidate category semantic information. S102: The input image to be segmented is fed into the initial segmentation model to model the target region and generate an initial segmentation mask corresponding to the semantic information of the candidate category; S103: Construct a spatial structure-aware category reorderer, and use the regional spatial structure information and image visual features represented by the initial segmentation mask of the target to reorder the candidate categories and obtain refined category semantics; S104: Construct a segmentation modulator based on semantic context, inject the refined category semantics as modulation information into the segmentation model, guide the segmentation of the target region, and generate an updated segmentation mask; S105: Feed back the updated segmentation mask to step S103, and repeat steps S103 and S104 until the preset iteration termination condition is met, and finally output the target category prediction result and its corresponding camouflaged target segmentation result.

[0026] In step S101, the category name of the category text is obtained, and a learnable text prompt module for camouflage scenes is constructed. The learnable text prompt module adaptively reconstructs the category text expression and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented to generate candidate category semantic information. Specifically, it includes the following steps: S1011: Obtain the category name of the category text and construct a learnable text prompt module, wherein the learnable text prompt module includes multiple learnable context vectors and category word embeddings corresponding to the category name; S1012: The multiple learnable context vectors are concatenated with the category word embeddings corresponding to the category names to form a category text prompt representation, so as to achieve adaptive reconstruction of the category text expression; S1013: Input the category text prompt representation into the text encoder of the visual language model to obtain the text semantic feature representation corresponding to each candidate category; S1014: Input the input image into the image encoder of the visual language model to extract the visual feature representation corresponding to the input image; S1015: Based on the similarity between the text semantic feature representations corresponding to each candidate category and the visual feature representations corresponding to the input image, perform open-vocabulary semantic matching to generate candidate category semantic information.

[0027] In a preferred embodiment, for any category The category text prompt can be represented as follows: ,in, Indicates length is Learnable context vectors Indicate category The corresponding category word embeddings. The learnable context vectors are updated through backpropagation during training and shared among different categories, so that the model can construct corresponding cue representations by replacing the category word embeddings when faced with an unseen category.

[0028] In a preferred embodiment, the category text prompt indicates that the semantic feature representation of the text obtained after inputting the text encoder can be represented as follows: ,in, A text encoder representing a visual language model.

[0029] In step S102, the input image is fed into the segmentation model to model the target region and generate an initial segmentation mask for the target. This specifically includes the following steps: S1021: Input the input image into the segmentation model to perform preliminary spatial modeling of potential target regions in the image and generate initial segmentation features; S1022: Based on the initial segmentation features, the segmentation model outputs the corresponding initial target segmentation mask; S1023: Associate the initial segmentation mask of the target with the candidate category semantic information generated in step S101, as a spatial structure prior for subsequent iterative refinement modules.

[0030] In a preferred embodiment, the segmentation model is Segment Anything Model 2, and the initial target segmentation mask is used to characterize the initial spatial distribution of the camouflaged target region in the input image.

[0031] In step S103, a spatial structure-aware category reorderer is constructed to reorder candidate categories, specifically including the following steps: S1031: Extract normalized visual features from the input image using an image encoder based on a visual language model, calculate similarity between these features and the textual semantic features corresponding to the candidate categories to obtain category-level prediction scores, and select the category with the highest confidence score based on these scores. There are 10 candidate categories, which constitute a candidate category set; S1032: Extract visual features of a preset intermediate layer from the image encoder of the visual language model, and perform pooling processing to obtain the intermediate layer visual representation; For For each candidate category, a corresponding ranking feature vector is constructed based on its prediction confidence, relative ranking position, and score difference with the Top-1 candidate category. S1033: The sorted feature vector, the intermediate layer visual representation, and the spatial mask representing the initial target segmentation mask generated in step S102 are jointly input into the spatial structure-aware category reordering unit to obtain the reordering score of each candidate category, and the reordering score is used to... The candidate categories are reordered to obtain refined category semantics.

[0032] In a preferred embodiment, the category-level prediction score can be represented as: ,in, Represents the normalized visual features of the input image. This represents a category semantic matrix composed of the textual semantic features of candidate categories.

[0033] In a preferred embodiment, for each candidate category Its sorting feature vector can be represented as: ,in, Indicates candidate category The prediction confidence level Indicates that it is in The relative sorting position among the candidates Indicates its relationship with The score difference between candidate categories.

[0034] In a preferred embodiment, the re-ranking score corresponding to each candidate category can be expressed as: ,in, Indicates a category rearranger. This represents the visual representation of the intermediate layer. This represents the spatial mask that represents the initial segmentation mask of the target generated in step S102.

[0035] In step S104, a segmentation modulator based on semantic context is constructed, and refined category semantics are injected into the segmentation model to guide the segmentation of the target region. Specifically, this includes the following steps: S1041: Extract the intermediate visual features output by the image encoder of the segmentation model and the text semantic features corresponding to the candidate categories, and perform adaptive filtering and alignment through visual gating units and semantic gating units respectively to obtain gated visual features and gated semantic features. S1042: The gated visual features and gated semantic features are fused together, and semantic-spatial modulation is performed based on multi-head attention mechanism and cross-modal correlation interaction. At the same time, the category conditions are enhanced by combining the refined category semantics output in step S103 to obtain category consistency enhancement features. S1043: Use learnable semantic allocation parameters to perform weighted fusion of attention enhancement features and category consistency enhancement features to obtain modulation features, and input the modulation features into the mask decoder of the segmentation model to generate an updated segmentation mask.

[0036] In a preferred embodiment, the gated visual feature and gated semantic features They can be represented as: ,in, This represents the intermediate visual features output by the image encoder of the segmentation model. Representing the semantic features of the text. and These represent visual gating units and semantic gating units, respectively.

[0037] In a preferred embodiment, the fusion feature can be represented as: .

[0038] In a preferred embodiment, the attention enhancement feature can be represented as: in, This represents a multi-head attention mechanism.

[0039] In a preferred embodiment, the cross-modal correlation diagram can be represented as: The attention-enhancing features are then reweighted using the cross-modal correlation graph to obtain semantic-enhancing features: .

[0040] In a preferred embodiment, the learnable semantic assignment parameters satisfy: The final modulation features are obtained in the following manner: .

[0041] In step S105, the updated segmentation mask is fed back to the iterative refinement module, and steps S103 and S104 are repeated until the preset iteration termination condition is met. Specifically, this includes the following steps: S1051: Based on the input image The candidate category set is initialized with the semantic state, and the target initial segmentation mask generated in step S102 is used as the initial spatial state. S1052: In the In each iteration, the current spatial state, candidate category semantic information, and input image visual features are input together into the spatial structure-aware category rearranger in step S103 to update the semantic state and obtain the updated semantic state. S1053: Input the updated semantic state into the semantic context-based segmentation modulator in step S104 to update the spatial state, obtain the updated segmentation mask, and continue to feed the updated segmentation mask back to step S103 until the preset iteration termination condition is met, and finally output the target category prediction result and its corresponding disguised target segmentation result.

[0042] In a preferred embodiment, the initial semantic state can be represented as: in, Indicates candidate category Text embedding, This represents the confidence weight for the corresponding candidate category.

[0043] In a preferred embodiment, the first The space state update in the round iteration can be represented as: ,in, This represents the spatial update operator corresponding to the segmented modulator. This represents the input image.

[0044] In a preferred embodiment, the preset iteration termination condition includes at least one of the following: reaching a preset number of iterations; the category prediction results of two adjacent iterations remain unchanged; the difference in segmentation masks between two adjacent iterations is lower than a preset threshold.

[0045] like Figure 1 As shown, Embodiment 2 of the present invention also discloses an open lexical masquerading target segmentation system driven by semantics and space, including a semantic construction module, an initial segmentation module, a category refinement module, a segmentation modulation module, and an iterative optimization module.

[0046] Among them, the semantic construction module is used to build a learnable text prompt module for camouflage scenarios, adaptively reconstruct the category text expression, and combine the visual language model to perform open-vocabulary semantic matching on the input image to generate candidate category semantic information; The initial segmentation module is used to feed the input image into the segmentation model to model the target region and generate an initial segmentation mask for the target corresponding to the semantic information of the candidate categories; The category refinement module is used to construct a spatial structure-aware category reorderer. It uses the regional spatial structure information and image visual features represented by the initial segmentation mask of the target to reorder the candidate categories and obtain refined category semantics. The segmentation modulation module is used to construct a segmentation modulator based on semantic context, inject the refined category semantics as modulation information into the segmentation model, guide the segmentation of the target region, and generate an updated segmentation mask; The iterative optimization module feeds back the updated segmentation mask to the category refinement module and repeatedly performs category refinement and segmentation modulation until the preset iteration termination condition is met, and outputs the target category prediction result and its corresponding camouflaged target segmentation result.

[0047] The specific implementation methods, input-output relationships, and technical effects of each module of the system can be referred to the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0048] This invention also discloses an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program to enable the electronic device to perform a semantically and spatially driven open-vocabulary camouflage target segmentation method as described in any of the foregoing embodiments.

[0049] This invention also discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the semantically and spatially driven open vocabulary spoofing target segmentation method described in any of the foregoing embodiments.

[0050] To further verify the feasibility and effectiveness of the semantically and spatially driven open-vocabulary camouflage target segmentation method proposed in this invention, specific experiments are described below. It should be noted that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0051] The specific experimental data for Example 1 are explained below: Example

[0052] This embodiment 1 describes the experimental dataset, training configuration, loss function, and implementation details of the method in embodiment 1 of the present invention.

[0053] 1. Dataset Setup; To verify the effectiveness and applicability of the proposed method, this embodiment conducts experiments on both open-vocabulary camouflage target segmentation and traditional camouflage target segmentation tasks. In the open-vocabulary camouflage target segmentation task, the OVCamo dataset is used as the experimental dataset. The OVCamo dataset is a benchmark dataset constructed for the open-vocabulary camouflage target segmentation task, containing 11,483 images involving 75 camouflage target categories. According to the open-vocabulary setting, the 75 categories are divided into two parts: visible categories during the training phase and invisible categories during the testing phase, with no overlap. The training set consists of 14 visible categories, containing 7,713 images; the test set consists of 61 invisible categories, containing 3,770 images. This dataset allows for the verification of the open-vocabulary recognition and segmentation capabilities of the proposed method in scenarios with invisible categories.

[0054] In traditional camouflaged target segmentation tasks, experiments were conducted using the CAMO, COD10K, and NC4K datasets. The CAMO dataset contains 1,250 images, divided into 1,000 training images and 250 test images according to a standard partitioning method. The COD10K dataset contains 5,066 camouflaged images, covering 5 superclasses and 69 subclasses, with 3,040 images used for training and 2,026 images used for testing. The NC4K dataset contains 4,121 camouflaged images and is typically used only as a test set. This embodiment uses 4,040 training images from CAMO and COD10K to construct a training set, and tests are performed on the CAMO and COD10K test sets as well as the complete NC4K dataset, thereby comprehensively evaluating the segmentation performance of the proposed method under various camouflaged scenarios.

[0055] In this embodiment, all experiments were implemented using the PyTorch deep learning framework and trained on a single NVIDIA RTX3090 GPU. During the training phase, the parameters of the pre-trained CLIP image-to-text encoder and the SegmentAnything Model 2 encoder were frozen, and only the remaining network parameters were randomly initialized and end-to-end optimization training was performed. By freezing the pre-trained encoder parameters, the general visual semantic representation capabilities and spatial prior knowledge of the large-scale pre-trained model can be fully utilized, while reducing the scale of parameter updates during training and improving training stability. Regarding the optimizer settings, this embodiment uses the AdamW optimization algorithm to update the network parameters, with an initial learning rate of 1e-6 and a weight decay coefficient of 5e-4. The batch size was set to 4. To unify the network input size and balance computational efficiency and segmentation accuracy, all input images were uniformly adjusted to a resolution of 384×384 during training.

[0056] Through the above experimental environment and training settings, the performance of the method of the present invention in open vocabulary camouflaged target segmentation tasks and traditional camouflaged target segmentation tasks can be effectively verified while ensuring training stability.

[0057] 3. Evaluation index setting; In order to objectively evaluate the performance of the method of the present invention, this embodiment sets corresponding evaluation indexes for open vocabulary camouflage target segmentation task and traditional segmentation task respectively.

[0058] For open-vocabulary masquerading target segmentation tasks, six evaluation metrics are used for quantitative assessment, including: class structure measure... ), class-weighted F-measure, ), class mean absolute error ), class standard F-measure, ), class enhanced alignment measure ) and class intersection over union (Class intersection over union) ).in, , , , and The larger the value, the better the model performance; The smaller the value, the closer the prediction result is to the true label.

[0059] For traditional camouflaged target segmentation tasks, four evaluation metrics are used for performance assessment, including: structure measure, Enhanced alignment measure ), weighted F-measure, ) and mean absolute error (mARR) ).in, , and The larger the value, the stronger the model's ability to express the structure of camouflaged targets and predict their regions. The smaller the value, the lower the prediction error.

[0060] In this invention, the mask generation result and category recognition result are jointly constrained. During the training phase, this embodiment employs multiple loss functions to construct the total loss function, performing joint supervised optimization of the model. Specifically, for the mask generation branch, weighted binary cross-entropy loss, weighted intersection-union ratio loss, and... The loss is used to constrain the predicted mask from three aspects: pixel classification accuracy, region overlap consistency, and shape structure consistency, thereby improving the prediction quality of the disguised target region. For the mask recognition branch, a pairwise ranking loss based on interval is used for supervision to explicitly model the relative ranking relationship between candidate categories, thereby improving the ranking accuracy of the true category in the candidate category set.

[0061] In this embodiment, the total loss of the model is defined as: ,in, For weighted binary cross-entropy loss, For weighted average loss, for loss, The loss function is a pairwise ranking loss based on intervals. Through joint optimization of the above loss functions, the overall structural consistency and category discrimination ability of the target region can be improved while ensuring pixel-level segmentation accuracy, thereby enhancing the overall performance of the method of this invention in the task of camouflaged target segmentation.

[0062] Example 2 This embodiment illustrates the experimental results of the method of the present invention in the open vocabulary camouflage target segmentation task.

[0063] On an open-vocabulary masquerading target segmentation dataset, the method of this invention is compared with existing open-vocabulary semantic segmentation methods and open-vocabulary masquerading target segmentation methods. Specifically, the selected comparison methods include SimSeg, OVSeg, ODISE, SAN, CAT-Seg, OVCoser, and SuCLIP. Since open-vocabulary masquerading target segmentation is an emerging research task, the number of existing dedicated methods for this task is relatively limited. Therefore, in addition to selecting existing open-vocabulary masquerading target segmentation methods, this embodiment also introduces several representative open-vocabulary semantic segmentation methods for re-evaluation to more comprehensively and fairly verify the effectiveness and generalization ability of the method of this invention in this task. All comparison methods are tested under a unified experimental setting.

[0064] Table 1: Quantitative comparison results on the OVCamo dataset under different training settings. The best results are marked in bold.

[0065] As shown in Table 1, the method of this invention exhibits significant and stable performance advantages on the open vocabulary camouflage target segmentation dataset OVCamo. Specifically, compared to the baseline method OVCoser, the method of this invention... , , , , as well as The performance metrics show improvements of 14.4%, 17.3%, 12.5%, 17.3%, 14.6%, and 16.8%, respectively. Furthermore, compared to the current leading method, SuCLIP, the method of this invention still achieves consistent improvements of 5.6%, 6.9%, 3.1%, 6.0%, 4.0%, and 7.1% in the aforementioned six metrics. These results demonstrate that the method of this invention can effectively improve the accuracy and segmentation precision of camouflaged target category identification under open vocabulary conditions.

[0066] Further analysis reveals that existing methods typically treat category semantics as a one-time prior input to the segmentation model, lacking a deep interaction between category semantics and spatial structure. Therefore, in complex camouflage scenarios, problems such as misclassification, incomplete target regions, or blurred boundaries easily arise. Furthermore, quantitative results show that directly fine-tuning the pre-trained model often leads to performance degradation. This indicates that simple fine-tuning may destroy the cross-modal knowledge learned in the pre-trained model, thereby triggering a catastrophic forgetting problem.

[0067] In contrast, this invention addresses the characteristics of open-vocabulary camouflage target segmentation tasks by constructing an iterative optimization module based on a category rearranger and a segmentation modulator. Specifically, it first utilizes the spatial structure information of the mask region to filter and rearrange candidate categories, enhancing the model's ability to discriminate camouflage target categories. Then, the optimized category semantic information is fed back to the segmentation branch, dynamically modulating the segmentation process, thereby achieving iterative collaborative optimization between category semantics and spatial structure. Therefore, the method of this invention can more fully model the diverse semantic expressions of complex camouflage targets in open-vocabulary scenarios, exhibiting stronger robustness and generalization ability in the identification of unseen category camouflage targets and boundary refinement.

[0068] In addition, such as Figure 5 As shown, the method of this invention can achieve superior visualization segmentation results in various typical complex scenarios. The examples presented cover representative challenging scenarios such as blurred boundaries, severe background interference, severe target occlusion, and targets of different scales (large, medium, and small). In comparison, existing methods are prone to problems such as inaccurate target localization, incorrect category prediction, missing boundaries, or incomplete segmentation in these scenarios; while the method of this invention can more accurately locate camouflaged target regions and generate more complete, coherent, and clearly defined segmentation masks. The visualization results further verify the effectiveness and stability of the method of this invention.

[0069] Furthermore, combined Figures 6 to 11 , Figure 6 This is a schematic diagram of the image to be segmented, illustrating an open-vocabulary camouflage target segmentation method driven by semantics and space in an embodiment of this application. Figure 7 This is a pixel-level label illustration of an open-vocabulary camouflage target segmentation method driven by semantics and space in an embodiment of this application. Figures 8 to 10 The images show the segmentation results of existing open-vocabulary camouflaged target segmentation methods CAT-Seg, SAN, and OVCoser. Figure 11 This is a schematic diagram of the open-vocabulary camouflaged target segmentation result obtained by the method described in the embodiments of this application. Figures 6 to 11The visualization comparison shows that the method described in this application embodiment can obtain more accurate and complete segmentation results in scenarios with complex backgrounds, blurred boundaries, and strong target camouflage.

[0070] Example 3 This embodiment illustrates the experimental results of the method of the present invention in traditional camouflaged target segmentation tasks.

[0071] To verify the applicability and performance improvement of the method in closed-set camouflage target segmentation scenarios, this embodiment further extends the method to traditional camouflage target segmentation tasks and tests it on three public datasets: CAMO, COD10K, and NC4K. Simultaneously, it compares the method with existing mainstream camouflage target segmentation methods. Specifically, the comparison methods selected include SINet, FSPNet, ACUMEN, RISNet, VSCode, CamoFocus, ZoomNeXt, DSAM, UniCOS-I, RUN++, SAM2-UNet, CamoDiffusion, and CFF-KDNet-P2.

[0072] As shown in Table 2, the method of this invention achieves competitive performance on multiple traditional camouflaged target segmentation datasets and demonstrates a steady improvement in key evaluation metrics. Specifically, compared with the SAM2-UNet method, the method of this invention achieves better performance on three public datasets. and Significant improvements were achieved across all metrics: 0.9% and 1.3% on the CAMO dataset, 4.3% and 1.7% on the COD10K dataset, and 2.0% and 1.8% on the NC4K dataset.

[0073] Table 2: Quantitative comparison results of the proposed method with 13 state-of-the-art methods on the COS task. “–” indicates that the corresponding result is unavailable. The best results are marked in bold.

[0074]

[0075] Furthermore, compared to the CamoDiffusion method, the method of this invention performs better on three publicly available camouflaged target segmentation datasets. The metrics also showed steady improvement, increasing by 1.7%, 1.8%, and 2.4% respectively. Furthermore, compared to the recently proposed CFF-KDNet-P2 method, the method of this invention also exhibits significant advantages on the CAMO dataset. and The metrics improved by 3.7% and 2.5% respectively; on the COD10K dataset, improvements were 1.1% and 1.6% respectively; and on the NC4K dataset, improvements were 2.9% and 2.3% respectively. These results fully demonstrate that the method of this invention is not only applicable to open-vocabulary camouflaged target segmentation tasks, but also achieves excellent performance in traditional camouflaged target segmentation tasks. This is because, although this invention is proposed for open-vocabulary camouflaged target segmentation tasks, its core semantic-space inter-driving mechanism can also enhance the model's performance against camouflaged targets. This invention enhances the expressive power of the target region. By dynamically modulating multi-level visual features using category semantic information and inversely correcting category predictions by combining spatial structure information, the method of this invention can reduce background interference, improve the response intensity of the target region, and improve the recovery of boundary details even under closed-set conditions. Therefore, the method of this invention also possesses good versatility, transferability, and generalization ability in traditional camouflaged target segmentation tasks.

[0076] In summary, the experimental results of Example 3 show that the method proposed in this invention not only performs well in open-vocabulary camouflaged target segmentation scenarios, but also maintains stable and leading performance in conventional camouflaged target segmentation scenarios, thus verifying that the method of this invention has good cross-task adaptability and practical application value.

[0077] Example 4 This embodiment illustrates the ablation experiment results of the method of the present invention to verify the contribution of each component module to the overall performance. In this embodiment, the basic model consisting of CLIP visual-text encoder and Segment Anything Model 2 is used as the baseline model. On this basis, the component modules proposed in this invention are gradually introduced, including a learnable prompting module, a category reordering unit, a segmentation modulator, and an iterative refinement strategy. Comparative tests are conducted on an open vocabulary camouflaged target segmentation dataset to analyze the impact of each module on the overall performance.

[0078] Table 3: Ablation experimental results of the proposed method on the OVCamo dataset. The best results are marked in bold.

[0079]

[0080] As shown in Table 3, with the gradual introduction of each module, the model performance continuously and steadily improves. Specifically, after introducing a class rearranger into the basic model, the model... , , , , as well as The metrics improved by 2.0%, 1.9%, 1.2%, 2.5%, 1.9%, and 2.3%, respectively. These results demonstrate that the category reorderer can effectively alleviate the semantic domain discrepancy problem under open vocabulary settings, enhancing the model's ability to discriminate disguised target categories by filtering and reordering candidate categories.

[0081] Building upon this foundation, the introduction of a segmentation modulator further enhanced the model's performance, resulting in improvements of 1.9%, 0.8%, 1.4%, 1.5%, 1.3%, and 1.9% in the six evaluation metrics mentioned above, respectively. This demonstrates that the segmentation modulator can effectively utilize category semantic information to guide the segmentation process, thereby better aligning semantic representations with pixel-level prediction results and improving the localization accuracy and regional integrity of camouflaged targets.

[0082] Furthermore, after introducing the learnable cue module, the model achieved additional improvements across all evaluation metrics, with an average improvement of 8.6%. These results demonstrate that adaptive learnable cue can provide more discriminative semantic priors for category recognition, thereby enhancing the model's semantic adaptability and recognition stability in open-vocabulary scenarios where no category is known.

[0083] Finally, by further introducing an iterative refinement strategy into the complete model, the model achieves optimal performance on all evaluation metrics. This strategy, through collaborative updates of the class rearranger and segmentation modulator during multiple iterations, enables the model to progressively correct class prediction and segmentation results, thereby achieving more stable and accurate target localization and recognition in complex camouflage scenarios.

[0084] Furthermore, visualization results under different module configurations reveal that with the gradual introduction of the learnable hint module, category rearranger, segmentation modulator, and iterative refinement strategy, the model's prediction results gradually approach the actual labeled results in terms of target localization accuracy, category recognition accuracy, and segmentation region completeness. This demonstrates that the components proposed in this invention are not simply superimposed, but rather form a synergistic and mutually reinforcing relationship within a unified framework, collectively improving the overall performance of open-vocabulary camouflaged target segmentation tasks and traditional camouflaged target segmentation tasks.

[0085] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantic and spatial co-driving open-vocabulary camouflage target segmentation method, characterized in that, Includes the following steps: Obtain the input image to be segmented and the category text; The input image is fed into an open-vocabulary camouflage target segmentation model for prediction, and the corresponding target category prediction result and camouflage target segmentation result are output. The open-vocabulary camouflage target segmentation model includes a learnable text prompt module, an initial segmentation model, a category rearranger, a segmentation modulator, and an iterative refinement module. The open-vocabulary camouflage target segmentation model performs predictions and outputs corresponding target category prediction results and camouflage target segmentation results, specifically including the following operation steps: Obtain the category name of the category text and construct a learnable text prompt module for camouflage scenarios. The learnable text prompt module adaptively reconstructs the category text expression and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented to generate candidate category semantic information. The input image to be segmented is fed into the initial segmentation model to model the target region and generate an initial segmentation mask corresponding to the semantic information of the candidate category; The candidate category semantic information, the initial target segmentation mask, and the image visual features extracted by the visual language model are input into the spatial structure-aware category reorderer to reorder the candidate categories and obtain refined category semantics. The refined category semantics are injected as modulation information into the initial segmentation model to guide the segmentation of the target region and generate an updated segmentation mask. The updated segmentation mask is fed back to the iterative refinement module, and the iterative refinement module and segmentation modulator are repeatedly executed until the preset iteration termination condition is met. Finally, the target category prediction result and its corresponding camouflaged target segmentation result are output.

2. The semantically and spatially driven open-lexical camouflage target segmentation method according to claim 1, characterized in that, The learnable text prompting module adaptively reconstructs the category text representation and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented, generating candidate category semantic information. Specifically, it includes the following steps: Get the category name from the category text; A learnable text prompt module for camouflage scenarios is constructed, wherein the learnable text prompt module includes multiple learnable context vectors and category word embeddings corresponding to category names; The category text representations reconstructed by the learnable text prompting module are input into the text encoder of the visual language model to obtain the text semantic feature representations corresponding to each candidate category. The input image is fed into the image encoder of the visual language model to extract the visual feature representation corresponding to the input image; Based on the similarity between the textual semantic feature representations corresponding to each candidate category and the visual feature representations corresponding to the input image, open-vocabulary semantic matching is performed to obtain the semantic information of the candidate categories.

3. The semantically and spatially driven open-lexical camouflage target segmentation method according to claim 2, characterized in that, The initial segmentation model models the target region and generates an initial segmentation mask corresponding to the semantic information of the candidate categories. This includes the following steps: The input image is fed into the segmentation model to perform preliminary spatial modeling of the target region in the image and generate initial segmentation features. Based on the initial segmentation features, the segmentation model outputs the corresponding initial segmentation mask for the target. The segmentation model is Segment Anything Model 2, and the initial target segmentation mask is used to characterize the initial spatial distribution of the camouflaged target region in the input image. The initial segmentation mask of the target is associated with the candidate category semantic information generated in the semantic construction stage, and used as the spatial structure prior of the subsequent iterative refinement module.

4. The semantically and spatially driven open-lexical camouflage target segmentation method according to claim 3, characterized in that, The category reorderer re-sorts candidate categories to obtain refined category semantics, specifically including the following steps: Image encoders using visual language models extract normalized visual features from input images. and the normalized visual features Similarity is calculated between the text semantic features corresponding to the candidate categories to obtain category-level prediction scores. The calculation formula is as follows: ;in, This represents a category semantic matrix composed of the textual semantic features of candidate categories; Select the category-level predicted score with the highest confidence level. There are 10 candidate categories, forming a candidate category set. ; Visual features of a pre-defined intermediate layer are extracted from the image encoder of a visual language model and then pooled to obtain the intermediate layer visual representation. ; against Each candidate category in the candidate categories Based on its prediction confidence, relative ranking position and its correlation with Construct a ranking feature vector based on the score differences between candidate categories: ; The sorting feature vector Intermediate layer visual representation And the spatial mask representing the initial segmentation mask of the target generated by the initial segmentation model. The common input spatial structure-aware category reorderer yields the reordering score for the corresponding candidate categories, calculated using the following formula: ;in, This represents a category reorderer, based on the reordering scores corresponding to each candidate category. right The candidate categories are reordered to obtain refined category semantics.

5. The semantically and spatially driven open-lexical camouflage target segmentation method according to claim 4, characterized in that, The segmentation modulator injects the refined category semantics as modulation information into the initial segmentation model, guides segmentation of the target region, and generates an updated segmentation mask. The specific steps include the following: Extracting intermediate visual features from the image encoder output of the segmentation model and the textual semantic features corresponding to the candidate categories The gating visual features are obtained by adaptive filtering and alignment using visual gating units and semantic gating units, respectively. and gated semantic features It is represented as: , The gated visual features With gated semantic features The fusion process is performed to obtain the fusion features, calculated using the following formula: ; Based on a multi-head attention mechanism, semantic-spatial modulation is applied to the fused features to obtain attention-enhanced features, calculated using the following formula: Meanwhile, through gating visual features With gated semantic features Bilinear interaction yields cross-modal correlation graphs The attention-enhancing features are then semantically consistent and reweighted using the cross-modal correlation graph to obtain semantically enhanced features. The calculation formula is as follows: The refined category semantics output by the category reorderer are then encoded into category-level semantic vectors. And based on the category-level semantic vector The semantic enhancement features Class conditional modulation is performed to obtain class consistency enhancement features, calculated using the following formula: ; The attention enhancement feature Enhanced consistency with categories Weighted fusion is performed, where the fusion weights are determined by learnable semantic assignment parameters, and the calculation formula is as follows: and modulated features Input the mask decoder of the segmentation model to generate an updated segmentation mask.

6. The semantically and spatially driven open-lexical camouflage target segmentation method according to claim 5, characterized in that, The execution process of the iterative refinement module specifically includes the following steps: Based on the input image Initialize the semantic state of the candidate category set The target initial segmentation mask generated by the initial segmentation model is used as the initial spatial state. The initial semantic state is obtained by weighting and aggregating the text embeddings of candidate categories according to their corresponding confidence scores, and the calculation formula is as follows: ,in, Indicates candidate category Text embedding, This represents the confidence weight for the corresponding candidate category; In the t-th iteration, the current space state is... Semantic information of candidate categories and visual features of the input image The spatial structure-aware category rearranger in the common input iterative refinement module updates the semantic state, resulting in the updated semantic state. The calculation formula is: ,in, This represents the semantic update operator corresponding to the category reorderer. Represents the set of candidate categories; Updated semantic state The semantic context-based segmentation modulator in the input segmentation modulator updates the spatial state to obtain the updated segmentation mask. The calculation formula is: ,in, This represents the spatial update operator corresponding to the segmented modulator. The input image is represented; semantic state update and spatial state update are repeatedly performed until the preset iteration termination condition is met, and the target category prediction result and its corresponding camouflaged target segmentation result are output according to the final semantic state and the final spatial state.

7. A semantically and spatially driven open-lexical camouflage target segmentation system, characterized in that, include: A semantic construction module is used to obtain the category name of the category text and construct a learnable text prompt module for camouflage scenarios. The learnable text prompt module adaptively reconstructs the category text expression and combines it with a visual language model to perform open-vocabulary semantic matching on the input image to be segmented to generate candidate category semantic information. The initial segmentation module is used to feed the input image to be segmented into the initial segmentation model to model the target region and generate an initial segmentation mask corresponding to the semantic information of the candidate category; The category refinement module is used to construct a spatial structure-aware category reorderer. It uses the regional spatial structure information represented by the initial segmentation mask of the target and the image visual features to reorder the candidate categories and obtain refined category semantics. The segmentation modulation module is used to construct a segmentation modulator based on semantic context, inject the refined category semantics as modulation information into the segmentation model, guide the segmentation of the target region, and generate an updated segmentation mask; The iterative optimization module is used to feed back the updated segmentation mask to the category refinement module, and repeatedly perform category refinement and segmentation modulation until the preset iteration termination condition is met, and output the target category prediction result and its corresponding camouflaged target segmentation result.

8. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor being used to invoke and run the computer program to cause the electronic device to perform a semantically and spatially driven open-vocabulary camouflage target segmentation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a semantically and spatially driven open-vocabulary masquerading target segmentation method as described in any one of claims 1-6.

Images