Video non-modal segmentation apparatus, method, medium and device

Abstract

The application relates to a video non-modal segmentation device, method, medium and equipment, which comprises an image encoder, a prompt encoder and a lightweight mask decoder; the image encoder is integrated with a feature space completion adapter and a region feature clustering learning module; the feature space completion adapter and the region feature clustering learning module perform feature completion and clustering at a feature space level to obtain high-dimensional image features; the prompt encoder encodes input prompt information and converts the prompt information into a feature vector compatible with the high-dimensional image features; and the lightweight mask decoder fuses the high-dimensional image features and the feature vector to obtain a target segmentation result. By adding the feature space completion adapter with a small parameter amount, domain knowledge is transferred, the amount of training data required for adaptation to a downstream task is greatly reduced, good non-modal segmentation capability is achieved, and strong generalization capability is achieved.

Description

Technical Field

[0001] This invention relates to the field of visual image processing technology, and more specifically, to a video nonmodal segmentation apparatus, method, medium, and device. Background Technology

[0002] During the power production, operation, and construction phases, numerous high-risk operations are generated, such as electrical equipment and line maintenance, hoisting, and personnel working at heights. According to power plant safety regulations, these high-risk operations require safety supervision, typically through on-site personnel monitoring. With the widespread adoption of video surveillance technology and the rapid development of artificial intelligence, intelligent safety supervision models based on computer vision recognition technology are being demonstrated in some scenarios. In power production environments, complex settings such as turbine buildings and wind turbine nacelles exist. Due to the numerous pipelines and limited space, areas of interest in the monitoring video (such as personnel and equipment) are easily obscured by foreground elements of other personnel and equipment. Furthermore, the proportion and distribution of obscuration are highly random. Visual recognition algorithms need to accurately infer and separate the features of obscured personnel or equipment to ensure the accuracy of on-site risk identification, thereby improving the application effectiveness of visual recognition technology in the field of power safety production.

[0003] Instance segmentation is an important task in computer vision, used to identify and separate different instances of the same or different categories. Traditional instance segmentation methods mainly rely on the features of the visible parts of objects in an image. For the occlusion scenarios mentioned above, amodal instance segmentation methods with logical reasoning capabilities have emerged. This method can identify and separate the visible and occluded parts of an object. Specifically, it uses the visible features of the object combined with prior knowledge of its shape (such as human proportions, mechanical structures, etc.) to deduce the geometry behind the occluded object.

[0004] Existing nonmodal instance segmentation methods require large datasets for training and rely on the labeled categories learned during training. In industrial safety production scenarios, collecting real-world data on high-risk operations is difficult, and labeling large datasets requires expensive annotation labor and manpower. Furthermore, existing end-to-end methods train nonmodal segmentation models from scratch, resulting in high training costs and a lack of generalization ability for new or unlearned categories. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a video nonmodal segmentation device, method, medium and equipment, which addresses the problems existing in the prior art.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows: a video nonmodal segmentation device is constructed, comprising: an image encoder, a cue encoder, and a lightweight mask decoder; the image encoder integrates a feature space completion adapter and a region feature clustering learning module; the feature space completion adapter is used to perform feature completion at the feature space level, and the region feature clustering learning module is used to cluster the extracted features at the feature space level; The image encoder is used to extract and cluster features from the original input image to obtain high-dimensional image features; The prompt encoder is used to encode the input prompt information and convert it into a feature vector that is compatible with the high-dimensional image features output by the image encoder; The lightweight mask decoder fuses the high-dimensional image features output by the image encoder and the feature vector output by the cue encoder to obtain the target segmentation result.

[0007] In the video nonmodal segmentation device of the present invention, the feature space completion adapter includes: a query modulation branch module, a key-value interaction branch module, and a spatial bias attention module; The query modulation branch module is used to obtain a query vector corresponding to the occluded region by multiplying the original features and the spatial weights of the spatial guidance signal. The key-value interaction branch module is used to map the original features through two independent linear transformation layers to obtain key vectors and value vectors; The spatial bias attention module is used to calculate based on the query vector and the key vector, and to process the spatial position bias term generated by the spatial guidance signal and the value vector to complete the global reconstruction of the features of the occluded area.

[0008] In the video nonmodal segmentation device of the present invention, the spatial bias attention module is specifically used for: Calculate the dot product similarity between the query vector and the key vector, and add the spatial position bias term; The value vector is weighted and aggregated by the attention weights normalized by the normalized activation function to complete the global reconstruction of the features of the occluded region.

[0009] In the video nonmodal segmentation device of the present invention, the spatial bias attention module uses the following mathematical model for the global reconstruction of the features of the occluded region: ; In the formula, This is the final output result; For query vector; The key vector; It is a value vector; The input feature map; Spatial offset term; For space guidance signals; The dimension of the attention head; This is the normalized activation function.

[0010] In the video nonmodal segmentation device of the present invention, the region feature clustering learning module uses a momentum update-based learning clustering method to cluster the extracted features.

[0011] The present invention also provides a video nonmodal segmentation method, comprising: Step S1: The image encoder, which integrates the feature space completion adapter and the region feature clustering learning module, performs feature extraction and clustering on the original input image to obtain high-dimensional image features; Step S2: Encode the input prompt information using a prompt encoder and convert it into a feature vector compatible with the image features; Step S3: The lightweight mask decoder fuses the high-dimensional image features output by the image encoder and the feature vector output by the cue encoder to obtain the target segmentation result.

[0012] In the video nonmodal segmentation method of the present invention, step S1 includes: At the feature space level, feature completion is performed using a feature space completion adapter to complete the global reconstruction of the occluded region; At the feature space level, the region feature clustering learning module is used to dynamically cluster the features of the entire image using the region feature clustering learning method.

[0013] In the video nonmodal segmentation method of this invention, the step of performing feature completion at the feature space level using a feature space completion adapter to complete the global reconstruction of the occluded region includes: The query vector corresponding to the occluded region is obtained by multiplying the original features and the spatial weights of the spatial guidance signal. The original features are mapped through two independent linear transformation layers to obtain key vectors and value vectors; The global reconstruction of the features of the occluded area is completed by calculating based on the query vector and the key vector, and by combining the spatial position bias term generated by the spatial guidance signal and the value vector. The learning method for the regional feature clustering is a momentum-updated learning clustering method.

[0014] The present invention also provides a storage medium storing a computer program adapted for loading by a processor to perform the steps of the video nonmodal segmentation method described above.

[0015] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the video nonmodal segmentation method described above by calling the computer program stored in the memory.

[0016] The video nonmodal segmentation apparatus, method, medium, and device of the present invention have the following beneficial effects: They include an image encoder, a cue encoder, and a lightweight mask decoder; the image encoder integrates a feature space completion adapter and a region feature clustering learning module; the feature space completion adapter and the region feature clustering learning module perform feature completion and clustering at the feature space level to obtain high-dimensional image features; the cue encoder encodes the input cue information and transforms it into a feature vector compatible with the high-dimensional image features; the lightweight mask decoder fuses the high-dimensional image features and feature vectors to obtain the target segmentation result. The present invention, by adding a feature space completion adapter with a small number of parameters to transfer domain knowledge, significantly reduces the amount of training data required to adapt to downstream tasks, achieves better nonmodal segmentation capabilities, and has strong generalization ability. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is an overall block diagram of the video nonmodal segmentation device provided by the present invention; Figure 2 This is a schematic diagram of the encoder structure provided by the present invention; Figure 3 This is a schematic diagram of the feature space completion adapter provided by the present invention; Figure 4 This is a schematic diagram of the feature space completion adapter embedded in an image encoder provided by the present invention; Figure 5 This is a flowchart illustrating the video nonmodal segmentation method provided by the present invention; Figure 6 This is a visualization diagram of the segmentation effect of the model on different datasets provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To address the shortcomings of existing non-modal instance segmentation methods, this invention provides a video non-modal segmentation device and method, which is a non-modal instance segmentation method based on region feature clustering and feature space completion. The video non-modal segmentation device provided by this invention innovatively proposes a feature space completion adapter, adding a feature space completion adapter with a small number of parameters to the image feature encoder to transfer domain knowledge. Compared to training a complete model to adapt to the feature distribution of non-modal segmentation, this invention trains a small adapter, directly modeling and transferring the object shape prior learned during model training. This method can effectively segment occlusion cases. The video non-modal segmentation method of this invention proposes a feature completion learning method based on region feature clustering. By constructing cluster-based feature reconstruction, non-modal features are decoupled in the latent space. Based on the feature consistency principle, contrastive learning is introduced to cluster the features extracted from the image, aligning the feature distribution of the target object with the feature classes with the highest similarity.

[0020] By fully understanding the target, we can effectively solve the target recognition deviation caused by occlusion, ensure the complete perception of the target's true appearance and physical center of gravity, and thus achieve more accurate risk identification in complex and dynamic power production environments. This has great practical value in intelligent monitoring of power safety.

[0021] refer to Figure 1 In a preferred embodiment, the video nonmodal segmentation apparatus comprises three main parts: an image encoder 100, a cue encoder 200, and a lightweight mask decoder 300. The input of the model is an image and a cue, and the output is a predicted mask.

[0022] In this embodiment of the invention, the image encoder 100 is used to extract and cluster features from the original input image to obtain high-dimensional image features.

[0023] Specifically, such as Figure 2 As shown, the input is 1024. After preprocessing, the 1024-format RGB image is first subjected to high-resolution feature extraction using a Vision Transformer (ViT) architecture pre-trained based on MAE (Masked Autoencoder). Specifically, the initial input image is divided into 16 segments through a Patch Embedding layer. A 16-bit image patch is converted to a size of 64 with a stride of 16. The feature sequence of 64. During the feature encoding stage, these features are fed into the deep ViT backbone network. To balance computational efficiency with the sensory field (the area of ​​the original input image corresponding to a pixel on the feature map of a certain layer in a neural network; simply put, how large an area of ​​the original image each point on the feature map "can see") when processing high-resolution input, the image encoder 100 alternates between window-based self-attention and global self-attention mechanisms. Most layers utilize the window mechanism for local attention. Contextual details are extracted within a 14-block region, while a global attention map is introduced every few blocks to capture the overall structural features of the image. Subsequently, the features output by the Transformer Block (the smallest functional unit of a Transformer Encoder, multiple Transformer Blocks cascaded together form a Transformer Encoder) are fed into a lightweight Neck module for spatial headline news and feature refinement. This Neck module first utilizes two 3-step timers with a stride of 1. 3 convolutional layers combined with residual connections process features, then through 1 One convolution maps the number of channels to a uniform 256 dimensions. After the above processing, a final output is generated that is 1 / 16th the size of the original (i.e., 64). 64) Image encoding features with a depth of 256 dimensions (ImageEmbedding). The image features output by the Transformer block are high-dimensional semantic and geometric features that integrate local details and global structure, specifically including geometric shapes and contours, spatial relationships, and target surface texture features. The Neck module is a lightweight module loaded in the image encoder 100, connecting the Transformer block and subsequent feature output stages. Its core function is to perform spatial dimensionality reduction and feature refinement on the features output by the Transformer block, preparing for the subsequent generation of standard-dimensional image encoding features.

[0024] exist Figure 2In the Transformer encoder, the functions, interrelationships, and signal flow of each module are as follows: Embedded patches provide image input to the encoder; the normalization layer preprocesses high-dimensional feature vectors (such as human-object spatial relationships, human contour features, etc.) in the input image, allowing features from different patches to participate in attention calculation at the same scale; the multi-head attention layer extracts contextual detail features from local regions of the normalized patch, introduces global attention at intervals of several patches to capture the overall image structure, and outputs local / global contextual features; normalization is performed again; the multi-layer perceptron (MLP) performs nonlinear transformation and dimensionality augmentation / compression on the image feature vectors output by the multi-head attention layer, weights and filters features, weakens the weight of redundant features (such as the texture of factory walls), strengthens effective features related to non-modal segmentation (such as the visible limb features of occluded people), improves feature discriminability, and finally outputs image features as the result of the current Transformer encoding block. Figure 2 In this context, "Linear projection of flattened patches" represents the linear projection of flattened image patches; "Patch + position embedding" represents the image patch plus a position embedding layer.

[0025] In this embodiment of the invention, the prompt encoder 200 is used to encode the input prompt information and convert it into a feature vector that is compatible with the high-dimensional image features output by the image encoder 100.

[0026] Specifically, the prompt encoder 200 maps the input prompt to a 256-dimensional vector embedding. When using an anchor box as a prompt, the box is represented by an embedding pair: (1) the position encoding of its upper left corner is added to the learned embedding representing "upper left corner", and (2) the structure is the same, but the learned embedding is used to represent "lower right corner".

[0027] In this embodiment of the invention, the lightweight mask decoder 300 fuses the high-dimensional image features output by the image encoder 100 and the feature vector output by the cue encoder 200 to obtain the target segmentation result.

[0028] Specifically, the lightweight mask decoder 300 employs a lightweight Transformer decoder architecture to map image embeddings and cue features to the final segmentation mask in real time. The input image features and the Prompt sequence first enter a two-way cross-attention mechanism. In this stage, the model alternately performs "cue-to-image" and "image-to-cue" attention operations, enabling the cue information to accurately locate the target region in the image, while the image features can also correct semantic biases in the cue.

[0029] Power production operations are complex, often involving partial occlusion of personnel and equipment in the foreground. This necessitates visual recognition algorithms to accurately infer and separate the true full appearance of the personnel and equipment in question. Traditional non-modal segmentation methods require large datasets for training, but data collection for high-risk safety production operations is difficult, leading to low accuracy in traditional methods. To address this issue, this invention incorporates a feature space completion adapter 101 with a small number of parameters to transfer domain knowledge, significantly reducing the amount of training data required for adaptation to downstream tasks. This achieves better non-modal segmentation capabilities and strong generalization ability. That is, as... Figure 1 As shown, the image encoder 100 of the present invention integrates a feature space completion adapter 101 and a region feature clustering learning module 102. The feature space completion adapter 101 is used to perform feature completion at the feature space level, and the region feature clustering learning module 102 is used to cluster the extracted features at the feature space level. Both the feature space completion adapter 101 and the region feature clustering learning module 102 of the present invention operate at the feature space level. Specifically, feature space completion is performed first to achieve feature filling at the spatial level; then region feature clustering is performed to achieve feature alignment at the semantic level. The optimization of encoder features is completed jointly in a temporal sequence, rather than being processed simultaneously in parallel.

[0030] The feature spatial completion adapter 101 of this invention is a spatial attention completion adapter (SACA), which aims to utilize attention mechanisms to guide the spatial signal M in long-range modeling capabilities. spec With the assistance of [unclear], effective information is retrieved and extracted from global features to fill the occluded area. Simultaneously, this invention integrates the feature space completion adapter 101 into the image encoder 100, maintaining the original image encoder 100 + cue encoder 200 + mask decoder structure unchanged, freezing the original weights and training only the parameters of the feature space completion adapter 101, thus achieving high-performance non-modal segmentation.

[0031] Preferably, the feature space completion adapter 101 includes: a query modulation branch module, a key-value interaction branch module, and a spatial bias attention module; the query modulation branch module is used to obtain a query vector corresponding to the occluded region by multiplying the original features and the spatial weights of the spatial guidance signal; the key-value interaction branch module is used to map the original features through two independent linear transformation layers to obtain a key vector and a value vector; the spatial bias attention module is used to calculate based on the query vector and the key vector, and process them in conjunction with the spatial position bias term and value vector generated by the spatial guidance signal to complete the global reconstruction of the features of the occluded region. Specifically, the spatial bias attention module is used to: calculate the dot product similarity between the query vector and the key vector, and add a spatial position bias term; and perform weighted aggregation of the value vector using attention weights normalized by a normalized activation function to complete the global reconstruction of the features of the occluded region.

[0032] The SACA adapter will send the space guide signal M spec This is transformed into a spatial-aware query, thereby achieving accurate reconstruction of non-modal regions. Specifically, such as... Figure 3 As shown, the SACA module modulates the input feature map E with the spatially guided M. spec The features are fused to generate context-aware enhanced features O. The core logic of this module consists of a query modulation branch, a key-value interaction branch, and a spatial bias attention module.

[0033] Query modulation branch: Spatial guidance of input M spec First, the original features E are transformed into spatial weight mappings using a lightweight encoding layer. We then map the original features E and M... spec The corresponding spatial weights are multiplied to obtain a query vector Q specifically for the occluded region. This ensures that the attention mechanism can focus on M. spec Highlight the occluded parts to find missing semantic information.

[0034] Key-value interaction branch: The input feature map E is simultaneously passed through two independent linear transformation layers, mapped to a key vector K and a value vector V, respectively. At this stage, K and V represent the feature library of all visible regions and the background in the image.

[0035] Spatial-Biased Attention: The model calculates the dot product similarity between Q and K, and additionally introduces M. specThe resulting spatial bias strengthens the constraint on the boundary of the occluded region. Finally, V is weighted and aggregated using attention weights normalized by Softmax to achieve global reconstruction of the features of the occluded region.

[0036] The operation is shown in the following formula: ; ; in, , , These represent linear projection layers; This is the final output result; For query vector; The key vector; It is a value vector; The input feature map; Spatial offset term; For space guidance signals; The dimension of the attention head; This is the normalized activation function.

[0037] M spec It participates in the generation of Q and plays a guiding role. From The spatial location bias term obtained from the mapping is used to provide additional geometric constraints. The final output O is added to the original feature E through a residual connection, ensuring the integrity of the original feature.

[0038] This invention employs an attention-based completion method to significantly improve the model's non-modal feature reconstruction capabilities in scenarios involving complex geometries and long-distance semantic associations, generating more coherent representations of occluded regions than ordinary convolution. By integrating the feature space completion adapter 101 into the image encoder 100, the model completes the occluded regions of objects at the image feature level. After decoding by the mask decoder, it can then predict the complete shape of the object, i.e., non-modal segmentation.

[0039] in, Figure 3 In Chinese, "Image feature" refers to the characteristics or features of an image. , , These are the weight matrices for vectors K, V, and Q, respectively; Attention: attention layer; Mask Down Sample: mask downsampling; Spatial bias term; Enhanced Features: Image features provide the original image features; After being processed by Mask DownSample, it participates in the generation of the query vector Q and is also directly mapped to... Participate in attention calculation.

[0040] The query modulation branch and the key-value interaction branch are parallel branches at the same level. They both take the original image features E as input and complete the feature transformation synchronously before the spatial bias attention layer is calculated. The final output Q, K, and V dimensions are completely consistent, which satisfies the point calculation requirements of the attention layer.

[0041] Spatial biased attention layer (Attention) is Figure 3 The core integration and computation center, where intermediate results from all upstream modules are directly input, is the only feature fusion and completion computation unit. Enhanced features are the residual connection results between the output of the spatial bias attention layer and the original image features E. The two are related by feature fusion: the output of the spatial bias attention layer is the completed feature of the occluded region, and the original E is the visible feature of the target. The residual connection allows the completed feature to contain both the reconstructed occlusion information and the original visible information, avoiding information loss during feature transformation.

[0042] This invention embeds the specially designed feature space completion adapter 101 into the image encoder 100, thereby achieving the restoration of occluded areas at the image feature level, with the following effect: Figure 4 As shown. Figure 4 In the diagram, Mask Input: Mask input; Image: Image input; Mask Downsampling: Mask downsampling layer; Block: Transformer encoding block; Adapter: Spatial attention completion adapter (i.e., feature space completion adapter 101); Image Encoder: Image encoder 100.

[0043] In this embodiment of the invention, the regional feature clustering learning module 102 uses a momentum-update-based learning clustering method to cluster the extracted features.

[0044] Specifically, the input RGB image is processed by an image encoder 100 to extract high-dimensional feature maps. Where F represents the feature tensor, i.e., image feature E or augmentation feature O. ∈ indicates that it belongs to, meaning that F is a set of real numbers in the following three-dimensional real space, indicating that all elements in the feature tensor have real values. Height is the spatial height dimension of the feature map, Width is the spatial width dimension of the feature map, C is the spatial width dimension of the feature map, and Channels is the channel dimension of the feature map. Then, using the initial prediction or ground truth mask generated by the model, the features are divided into visible region features and occluded region features. The core logic of this method lies in the assumption that the underlying semantics of the occluded portion is highly correlated with certain visible regions in the image (such as other parts of the same object or similar objects). By dynamically clustering the features of the entire image, the model can automatically discover 5 or 7 representative semantic prototypes. Next, it calculates... The distribution distance between the model and each cluster center identifies the "positive sample class" that is semantically closest to it. Through feature completion learning, the model is guided to align the feature vectors of the occluded region to the target cluster center, thereby "completing" the semantic information lost due to occlusion and enhancing the decoder's ability to recover the complete shape of the object. Here, the visible region refers to the visual information of the visible part of the image, such as the body contours and textures of a person who is not occluded. The occluded region features refer to the features of the occluded areas of the image, such as the features of a person's occluded torso.

[0045] In this embodiment of the invention, the clustering method is designed as follows: To make the clustering process differentiable and integrated into end-to-end training, we employ learnable prototypes based on momentum updates: we initialize and define K learnable center vectors in the network. (K=5 or 7). In each training batch, instead of using traditional K-means iteration, the feature points are compared with each center. The cosine similarity is used, and the Sinkhorn-Knopp algorithm is employed for optimal transfer allocation to ensure that feature points are uniformly mapped to each cluster center, avoiding cluster collapse. The cluster centers are updated via backpropagation.

[0046] The loss function is designed as follows: To achieve the goal of "shortening the positive distance and lengthening the negative distance", the following loss function is used: Let The average feature vector of the occluded region. The cluster center is the one with the closest Euclidean distance. InfoNCE Loss is used. ; Where sim(u,v) represents the cosine similarity. This is a temperature parameter used to control the smoothness of the distribution. This represents the loss value. i = 1,..., K Indicates a total K One sample.

[0047] refer to Figure 5 The present invention also provides a video nonmodal segmentation method, wherein the video nonmodal segmentation method is implemented based on the video nonmodal segmentation apparatus disclosed in the embodiments of the present invention.

[0048] Specifically, such as Figure 5 As shown, the video nonmodal segmentation method includes: Step S1: The image encoder 100, which integrates the feature space completion adapter 101 and the region feature clustering learning module 102, performs feature extraction and clustering on the original input image to obtain high-dimensional image features.

[0049] Specifically, step S1 includes: performing feature completion at the feature space level through the feature space completion adapter 101 to complete the global reconstruction of the occluded area; and dynamically clustering the full image features at the feature space level through the region feature clustering learning module 102 using the region feature clustering learning method.

[0050] Specifically, at the feature space level, feature completion is performed using a feature space completion adapter 101 to complete the global reconstruction of the occluded region. This includes: multiplying the original features and the spatial weights of the spatial guidance signal to obtain a query vector corresponding to the occluded region; mapping the original features through two independent linear transformation layers to obtain a key vector and a value vector; calculating based on the query vector and key vector, and processing them in conjunction with the spatial position bias term and value vector generated by the spatial guidance signal to complete the global reconstruction of the features of the occluded region; the learning method for region feature clustering is a momentum-updated learning clustering method.

[0051] Step S2: The input prompt information is encoded by the prompt encoder 200 and converted into a feature vector compatible with image features; Step S3: The lightweight mask decoder 300 fuses the high-dimensional image features output by the image encoder 100 and the feature vector output by the cue encoder 200 to obtain the target segmentation result.

[0052] The non-modal segmentation method based on image feature space completion and region feature clustering provided by this invention can significantly improve the model's ability to segment occluded objects. Specific experimental data is presented below, as shown in Table 1.

[0053] Table 1. Comparison of the separation accuracy of the present invention with that of the state-of-the-art methods in the closed domain. Note: A closed region is a connected region in an image that contains a complete boundary.

[0054] As shown in Table 1, the present invention outperforms existing non-modal segmentation methods in segmentation accuracy across multiple datasets. For example, the visualization of the segmentation and recognition results of the present invention on the datasets in Table 1 is as follows: Figure 6 As shown.

[0055] Regarding the generalization ability test, in the comparison, the training data of this invention does not include the COCOA-cls training set, and the model is tested on the COCOA-cls test set. The model used as the comparison object is trained on the COCOA-cls training set and tested on the same test set. The results are shown in Table 2, demonstrating that the model of this invention achieves good results even on datasets that have not been trained on.

[0056] Table 2. Test results of the generalization ability of the present invention The non-modal segmentation method based on image feature space completion and region feature clustering provided by this invention significantly reduces the amount of training data required for adaptation to downstream tasks by incorporating a feature space completion adapter 101 with a small number of parameters for domain knowledge transfer. This achieves good non-modal segmentation capabilities and strong generalization ability. Furthermore, the method proposed in this invention is a plug-and-play module that can be directly used in video segmentation technology to achieve reliable and effective video non-modal segmentation. This can effectively improve the accuracy of identifying high-risk operational hazards in complex environments of power safety production, thereby achieving cost reduction and efficiency improvement in the industry while enhancing the level of power safety production.

[0057] Furthermore, an electronic device according to the present invention includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the video nonmodal segmentation method as described above. Specifically, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, when the computer program is downloaded, installed, and executed by an electronic device, it performs the functions defined in the methods of the embodiments of the present invention. The electronic device in the present invention can be a terminal such as a laptop, desktop computer, tablet computer, or smartphone, or it can be a server.

[0058] Furthermore, one type of storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the video nonmodal segmentation method described above. Specifically, it should be noted that the storage medium described above in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0059] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0060] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0061] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0062] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0063] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They do not limit the scope of protection of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should fall within the scope of the claims of the present invention.

Claims

1. A video nonmodal segmentation device, characterized in that, include: Image encoder, cue encoder, and lightweight mask decoder; The image encoder integrates a feature space completion adapter and a region feature clustering learning module; The feature space completion adapter is used to perform feature completion at the feature space level, and the region feature clustering learning module is used to cluster the extracted features at the feature space level. The image encoder is used to extract and cluster features from the original input image to obtain high-dimensional image features; The prompt encoder is used to encode the input prompt information and convert it into a feature vector that is compatible with the high-dimensional image features output by the image encoder; The lightweight mask decoder fuses the high-dimensional image features output by the image encoder and the feature vector output by the cue encoder to obtain the target segmentation result.

2. The video nonmodal segmentation device according to claim 1, characterized in that, The feature space completion adapter includes: a query modulation branch module, a key-value interaction branch module, and a spatial bias attention module; The query modulation branch module is used to obtain a query vector corresponding to the occluded region by multiplying the original features and the spatial weights of the spatial guidance signal. The key-value interaction branch module is used to map the original features through two independent linear transformation layers to obtain key vectors and value vectors; The spatial bias attention module is used to calculate based on the query vector and the key vector, and to process the spatial position bias term generated by the spatial guidance signal and the value vector to complete the global reconstruction of the features of the occluded area.

3. The video nonmodal segmentation device according to claim 2, characterized in that, The spatial bias attention module is specifically used for: Calculate the dot product similarity between the query vector and the key vector, and add the spatial position bias term; The value vector is weighted and aggregated by the attention weights normalized by the normalized activation function to complete the global reconstruction of the features of the occluded region.

4. The video nonmodal segmentation device according to claim 3, characterized in that, The spatial bias attention module uses the following mathematical model for the global reconstruction of features in the occluded region: ； In the formula, This is the final output result; For query vector; The key vector; It is a value vector; The input feature map; Spatial offset term; For space guidance signals; The dimension of the attention head; This is the normalized activation function.

5. The video nonmodal segmentation device according to claim 1, characterized in that, The regional feature clustering learning module uses a momentum-update-based learning clustering method to cluster the extracted features.

6. A video nonmodal segmentation method, characterized in that, include: Step S1: The image encoder, which integrates the feature space completion adapter and the region feature clustering learning module, performs feature extraction and clustering on the original input image to obtain high-dimensional image features; Step S2: Encode the input prompt information using a prompt encoder and convert it into a feature vector compatible with the image features; Step S3: The lightweight mask decoder fuses the high-dimensional image features output by the image encoder and the feature vector output by the cue encoder to obtain the target segmentation result.

7. The video nonmodal segmentation method according to claim 6, characterized in that, Step S1 includes: At the feature space level, feature completion is performed using a feature space completion adapter to complete the global reconstruction of the occluded region; At the feature space level, the region feature clustering learning module is used to dynamically cluster the features of the entire image using the region feature clustering learning method.

8. The video nonmodal segmentation method according to claim 7, characterized in that, The step of performing feature completion at the feature space level using a feature space completion adapter to complete the global reconstruction of the occluded region includes: The query vector corresponding to the occluded region is obtained by multiplying the original features and the spatial weights of the spatial guidance signal. The original features are mapped through two independent linear transformation layers to obtain key vectors and value vectors; The global reconstruction of the features of the occluded area is completed by calculating based on the query vector and the key vector, and by combining the spatial position bias term generated by the spatial guidance signal and the value vector. The learning method for the regional feature clustering is a momentum-updated learning clustering method.

9. A storage medium, characterized in that, The storage medium stores a computer program adapted for loading by a processor to perform the steps of the video nonmodal segmentation method as described in any one of claims 6 to 8.

10. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the video nonmodal segmentation method as described in any one of claims 6 to 8 by calling the computer program stored in the memory.

Images