Multi-dimensional enhanced open vocabulary video instance segmentation method

Abstract

The invention provides a multi-dimensional enhanced open vocabulary video instance segmentation method, which comprises the following steps of: 1, proposing a novel open vocabulary segmentation thought through mathematical modeling and analysis; step 2, preprocessing a video input by a user, carrying out frame sampling, and realizing category-related segmentation in a Transform architecture by respectively strengthening interaction of category texts, image features and query vectors; step 3, adopting an instance-driven TopK time sequence matching strategy to improve the stability and accuracy of cross-frame matching; and 4, improving the category recognition capability through multi-scale feature fusion, and optimizing target classification. The open vocabulary video instance segmentation method can efficiently perform open vocabulary video instance segmentation, is widely applied to the fields of video monitoring, automatic driving, video indexing and the like, and promotes the development of video understanding and reasoning technologies.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of video data processing, and particularly relates to a multi-dimensional enhanced open vocabulary video instance segmentation method. BACKGROUND

[0002] The open vocabulary video instance segmentation technology has wide application value in actual life, especially in scenes that need to accurately detect, semantically segment and cross-frame track open category objects. Traditional systems mostly rely on a preset category set, which is difficult to adapt to the changing target types in the real environment.

[0003] With the development of large-scale visual language models such as CLIP, cross-modal semantic understanding has become a research hotspot in artificial intelligence, which has promoted the progress of image classification, image-text retrieval and zero-shot learning tasks. With its text guiding ability, CLIP also provides a new technical path for open vocabulary video instance segmentation, enabling the system to identify any category target according to natural language description without relying on a fixed label set.

[0004] Existing methods often integrate CLIP into the Mask2Former architecture, combining query vectors and image features to interact, and achieving segmentation and classification through matching with text embeddings. However, CLIP outputs global semantic features, and directly performing mask pooling on the mask region can easily lose boundary details and local structures, limiting the system's performance in open category and fine-grained recognition scenarios.

[0005] On the other hand, there is a matching redundancy problem in time series modeling. Existing methods mostly use full query for frame-by-frame matching, lack object saliency filtering, bring additional computational burden, and are easily disturbed by noise, affecting cross-frame consistency and tracking stability.

[0006] Therefore, there is an urgent need for a new method that combines open vocabulary semantic modeling, fine structure alignment and efficient time series matching to improve the system's semantic generalization ability and practical performance in complex open scenarios. SUMMARY

[0007] The application provides a multi-dimensional enhanced open vocabulary video instance segmentation method, aiming to solve the three key problems existing in the existing system under open semantic conditions: first, the text semantics and segmentation process are disconnected, making it difficult to effectively guide unseen categories; second, the redundant operation on all queries in cross-frame matching causes computational waste and tracking instability; third, the visual features lack fine-grained structure expression, affecting instance recognition and mask quality.

[0008] The method of the application comprises the following steps: Step 1, mathematically modeling and analyzing the open vocabulary video instance segmentation task; Step 2, establish a class-related segmentation model, pre-process the video input by the user, and sample frames from the video; the class-related segmentation model enhances the image features obtained from each frame and the query used to represent objects during segmentation according to the user-input arbitrary class name; and the enhanced image features and query interact in the transformer decoder to achieve class-related segmentation. Step 3, execute the instance-driven TopK timing matching strategy. Step 4, improve the recognition ability of the class by multi-scale feature fusion, and optimize the target classification.

[0009] Step 1 includes: Given an input video as a series of frames I, the length of the video is , the t-th frame is , which is represented as: , Each frame contains two or more object instances, where each object instance is represented by a mask, denoted as , represents the x-th object instance in the t-th frame, and each frame contains N object instances, denoted as , which represents the mask of the N different objects in the current frame ; A user-defined text input model is introduced to represent the class target, and the given user input class name description is set as , is a natural language form of text; Participate in the generation of the mask during segmentation; Finally, the similarity between the mask and the class description is calculated to obtain the mask of the object that meets the user input requirements.

[0010] Step 2 includes: First, the user input class name description is a natural language string , which is mapped to a multi-dimensional semantic embedding vector by the text encoder of the pre-trained model CLIP (Contrastive Language-Image Pre-training): , where represents the text encoder function, is the text feature obtained according to the input natural language class name, and if multiple class names are input by the user, the text features is multi-dimensional, the initial acquisition dimension is wherein is the total number of frames of the video (the total number of frames is also obtained according to one frame per second), X is the number of input categories, C is the feature dimension channel number obtained by the text encoder, C is determined according to the selected text encoder, the dimension of the feature vector obtained by the initial input of each category is 1*C, the total text feature of each frame is obtained by splicing the feature vectors of X categories, and the dimension is X*C, and finally frames to obtain the final text feature , the dimension is *X*C, the text feature as a semantic guide source throughout the subsequent segmentation process; The input video is uniformly sampled at a rate of 1 frame / second (fps) to convert it into an image sequence , the t-th frame image is input into the image encoder of the pre-trained model CLIP to extract multi-scale image features: , wherein , represents the image encoder of the pre-trained model CLIP, which is used to obtain image features of different sizes, represents the image feature of the s-th layer, s represents the scale level of the feature, and S represents the total number of layers; these features are arranged from shallow to deep, and the spatial information represented changes layer by layer; The image encoder obtains the image features of a block area on each frame image, so the obtained image features are input into the pixel decoder to obtain pixel-level image features of different sizes: , wherein represents the pixel decoder, which is used to obtain pixel-level image features, represents the pixel-level image feature of the s-th layer obtained; the pixel-level image feature obtained here does not carry text category information, which will be used for feature fusion in the subsequent recognition stage; A visual-guided semantic enhancement mechanism is introduced, which is a cross-modal enhancement based on natural language semantic embedding to guide the pixel features by categories, which specifically includes the following steps: Perform a pixel-guided cross-attention Cross-Attention operation, taking the pixel feature as the query Q and the key K, and the text semantics as the value V, to perform cross-modal attention enhancement: , , where d represents the dimension of the query Q, divided by is used for normalization to avoid the dot product value being too large; softmax is a normalization function, used to convert the similarity score of into a probability distribution, so that the attention weight of each pixel position to the text semantics is between 0 and 1; represents the information that needs to be fused and enhanced obtained by cross attention; T represents transposition; respectively, the mapping matrix of the query Query, the mapping matrix of the key Key and the mapping matrix of the value Value; Each layer of pixel features is calculated separately , then is added to the original corresponding layer of pixel-level features and gate fusion is performed: , , , where is the intermediate feature fused with , used for subsequent gate fusion with the original pixel feature; represents the gate factor; sigmoid is a normalization function, used to control the gate factor between 0 and 1, and then control how much to keep or discard according to the gate factor; MLP is a multi-layer perceptron structure, used for channel mapping and nonlinear activation; is a mapping function; is the s-th layer of pixel-level image features enhanced by semantics; Initialize the learnable query vector , and fuse it with the text semantic vector, in the same way as the pixel-level image feature fusion, fuse and to obtain the fusion query vector with semantic orientation ; Send the pixel feature map enhanced by semantics and the fusion query vector together into the Transformer decoder: , where ; represents the feature embedding vector of the i-th instance in the t-th frame in the Transformer decoder, that is, the instance representation of the instance; represents the mask of the i-th object in the t-th frame; Intersection over Union score of the mask of the i-th instance in the t-th frame with the ground truth mask, used in training phase; object confidence score from the Transformer decoder, the Transformer decoder outputs , this score is the score of the i-th mask in the t-th frame outputted by the Transformer decoder, which is the score of the Transformer decoder judging the i-th mask as an object, as the object confidence score.

[0011] Step 3 comprises: Step 3-1, in the first frame of the video sequence, according to the initial prediction result of the class-related segmentation model, the object instance is obtained, whether it is an object is evaluated for all object instances, the object confidence score of each object instance is generated, and a set of instances with significant foreground features is selected through a dynamic confidence filtering mechanism , the object confidence score of the object instance with significant foreground features is registered into the time sequence memory storage unit Memory Bank as a reference template for subsequent frame matching; Step 3-2, in the subsequent frame processing process, first screen the object instances that meet the current frame through the dynamic confidence, and perform one-to-one matching between the high-confidence instance representations retained in the time sequence memory storage unit Memory Bank and the instance representations that meet the conditions screened in the current frame, and perform cross-frame matching operation through the minimum cost optimization algorithm to ensure the consistency of the instance index and the integrity of the instance track; Step 3-3, if there is an object instance in the current frame that is not successfully matched but the object confidence is still significant, the object instance is determined as a possible new instance through the object confidence score of the object instance, and is added to the time sequence memory storage unit Memory Bank in an incremental manner to support the dynamic expansion of object instances in the video; the object confidence is still significant means that the object confidence score is greater than the dynamic confidence threshold ; Step 3-4, when the storage capacity of the time sequence memory storage unit Memory Bank reaches the upper limit, a hybrid replacement strategy is executed, high-confidence instances are preferentially retained, and obsolete records that are not successfully matched for more than two consecutive frames are removed, thereby avoiding information redundancy and improving storage efficiency.

[0012] In step 3-1, the dynamic confidence filtering mechanism comprises: calculating the mean , variance and skewness of the object confidence scores of all object instances in the current frame: , ​​ Calculate the dynamic confidence threshold for the current frame based on the mean and variance. : , , Where exp represents the natural exponential function; Indicates intermediate parameters; The first frame is processed first, and then filtered out using the calculated dynamic confidence threshold. Score greater than object instances, will Register into the sequential memory storage unit Memory Bank, and construct the initial sequential memory storage unit Memory Bank through the first frame.

[0013] Step 3-2 includes:

[0014] Calculate the dynamic confidence of the current frame. Filter out Greater than For object instances, the Euclidean distance between the instance representation of each object instance selected in the current frame and the instance representation of each object instance in the Memory Bank is calculated, and then the Hungarian algorithm is used to perform cross-frame matching, assigning the same index to the same object instances.

[0015] Step 3-3 includes: When the number of object instances selected by dynamic confidence in the current frame exceeds the number of object instances in the Memory Bank, it indicates that a new object instance has appeared in the current frame that does not exist in the Memory Bank. After cross-frame matching, the remaining unmatched objects are scored. Greater than The instance representation of the object instance is added to the Memory Bank as a newly appearing object instance in the current frame, that is, Add it to the sequential memory storage unit Memory Bank and assign it a new sequence number for matching subsequent frames.

[0016] Steps 3-4 include: When the number of object instances stored in the Memory Bank reaches a threshold (set to 30, because users typically don't need to look up more than 10 object instances at a time in a video, so 30 is sufficient), a hybrid replacement strategy is implemented. If the storage limit of 30 is reached, but new object instances still need to be added, then calculation is performed. Choose the product The smallest object instance is replaced, and a new sequence number is given after replacement, and the sequence number of the replaced object instance is not used, where K represents the time that is not matched.

[0017] Step 4 includes: Step 4-1, introducing an image feature and pixel-level image feature fusion mechanism, integrating each scale pixel feature channel by channel and the original last layer image feature representation to generate a fused feature map for subsequent classification and matching judgment; Step 4-2, performing a region aggregation operation based on instance segmentation mask to extract a region feature vector corresponding to each object instance and a class embedding vector generated by a text encoder to perform similarity evaluation to obtain a semantic similarity score for further target classification and identification; Step 4-3, adopting a multi-factor confidence fusion strategy, comprehensively considering semantic similarity , object confidence score and model IoU score on mask boundary , calculating a weighted aggregation representation by calculating and fusing each factor.

[0018] Step 4-1 includes: the maximum layer feature of the image feature of each frame obtained by the image encoder of CLIP and the obtained multi-scale pixel-level image feature are fused layer by layer (here is not semantically enhanced), and the layer-by-layer fusion method is: , , , , wherein is a learnable weight matrix of the Sth layer, used to learn the dimension change after feature splicing and the fusion mechanism, and learned during training; is the result of intermediate fusion, and the fused feature obtained each time is fused with the pixel-level image feature of the next layer, and the final fused feature is obtained after each layer is fused. Step 4-2 includes: after obtaining the final fused feature , selecting the frame where each object instance first appears to obtain the fused feature of the frame, and according to the mask of the object instance Calculate the region feature vector of the object instance , the formula is: , Wherein, the specific content of the mask is that the pixel point belonging to the object instance is 1, and the pixel point not belonging to the object instance is 0; is the number of pixel points in the mask region; p represents each pixel position in the mask region; represents the feature value of the position p; Calculate the similarity score between the region feature vector and the text feature vector , which is used for classifying the target, and the text feature vector comes from the category name input by the user, and the initial acquisition dimension is *X*C, and then the text feature of each category is obtained, and the text feature of the i-th category is represented as The similarity score between the i-th mask and the j-th category is calculated by the following formula : , The similarity score is used as the semantic similarity, because the continuity association is performed on the time dimension, so only a given semantic of a frame is needed, and the semantics of the object instance of the remaining frames are consistent, so frame t is discarded; Step 4-3 includes: obtaining the similarity score , the object confidence score and the intersection over union score of the mask boundary After that, the multi-factor confidence fusion strategy is fused to obtain the weighted aggregation representation : , Wherein, represents the final score of the object instance i for the category j; the fusion weight is automatically learned through the training process, and is used to adjust the influence of semantic similarity, target existence and mask quality on the matching confidence in the final matching process. The final score of a certain object instance i for the category j is the highest, that is, the category name of the object instance can be obtained.

[0019] The application can support dynamic recognition and instance-level modeling of any category target by introducing a natural language driven mechanism and an open vocabulary semantic understanding framework, and has good class expansion and generalization ability.

[0020] The present technology can be widely applied in the following fields: Video monitoring analysis: Supports automatic segmentation and continuous tracking of multiple types of targets such as pedestrians, vehicles, and luggage. The system can detect open-class objects such as "abandoned backpacks" and "suspicious persons" through natural language instructions, adapting to the changing monitoring needs of complex environments such as airports and shopping malls.

[0021] Automatic driving perception system: The system can identify and segment objects such as vehicles, pedestrians, and bicycles in the front scene in real time, and track them across frames based on a timing memory mechanism. For unseen or rare classes such as "freight trailers" and "construction robots", target recognition and response can also be completed through semantic guidance, improving safety and robustness.

[0022] Video editing and content indexing: Supports users to flexibly specify the object class to be extracted in text form, realizing fast segment positioning and material extraction. For example, inputting "all people riding electric bikes" can automatically edit the corresponding video segment, meeting the semantic-level creation needs.

[0023] Augmented reality and human-computer interaction system: The system can segment and highlight any object in the real environment according to user voice or text instructions, support open-class identification, and be suitable for industrial training, maintenance and assembly, smart education, etc. such as "please mark the hydraulic valve" or "show the operator's right hand tool".

[0024] Video question answering and auxiliary understanding system: With open vocabulary perception and multi-modal matching capability, it supports natural language driven question analysis and instance-level visual feedback, such as "how many printers are there in the video?" and "where did the person in green go?" Complex semantic question and answer scenarios are widely used in education, justice, and medical fields.

[0025] The present application has the following advantages: (1) The natural language semantic guidance mechanism is introduced to support mask learning driven by any text description, without pre-setting class labels, enhancing segmentation accuracy and semantic generalization ability; (2) Object perception filtering and dynamic updating are performed in the memory module, reducing redundant calculation, improving cross-frame matching efficiency and instance consistency; (3) Multi-scale visual details are fused and combined with semantic similarity discrimination to improve instance-level classification accuracy, supporting the identification and matching of unseen classes; (4) During training and evaluation, datasets with non-overlapping class spaces are combined to test open target identification based on natural language instructions, effectively verifying the zero-shot capability of the system; (5) The module structure is loosely coupled and pluggable, suitable for multi-modal video understanding and target tracking tasks in various open scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a method flowchart of the present application.

[0027] Figure 2It is a schematic diagram of the display result of the video instance segmentation part frame of the application. DETAILED DESCRIPTION

[0028] The above and / or other aspects of the application will become more apparent by describing in detail the preferred embodiments thereof with reference to the attached drawings, in which:

[0029] As shown in the drawings, the embodiment of the application provides a multi-dimensional enhanced open vocabulary video instance segmentation method, comprising the following steps: Figure 1 Step 1, mathematical modeling and analysis of the open vocabulary video instance segmentation task, including: Set the given input video as a series of frames, the length of the video is , each frame is , denoted as: , Each frame contains multiple object instances, where each instance is represented by a mask; The mask is denoted as , representing the xth object instance in the tth frame, and each frame contains N object instances; Denote the mask of the current frame containing N different objects as ; In order to realize "open vocabulary", the application introduces a user-defined text input model to represent the category target. Assuming that the category description input by the user is , which is a natural language form of text. The text will participate in the generation of the mask during the segmentation process.

[0030] Finally, the similarity between the mask and the category description is calculated to obtain the mask of the object that meets the user input requirements.

[0031] Step 2, establish a class-related segmentation model, pre-process the video input by the user, and sample the frames of the video; the class-related segmentation model enhances the image features obtained from each frame according to the arbitrary category name input by the user and the query used to represent the object during the segmentation process; the enhanced image features and the query interact in the transformer decoder to realize the class-related segmentation, including: First, the user input category name is a natural language string , which is mapped to a multi-dimensional semantic embedding vector by the text encoder of CLIP: , wherein,​ denotes a text encoder function, is the text feature obtained according to the input natural language category name, if the user inputs multiple category names, the obtained is multi-dimensional, the initial obtained dimension is , wherein is the total number of frames of the video (the total number of frames is obtained according to one frame per second, and the total number of frames is also obtained according to one frame per second), wherein X is the number of input categories, C is the feature dimension channel number obtained by the text encoder, C is determined according to the selected text encoder, the dimension of the feature vector obtained for each initial input category is 1*C, after obtaining the feature vector of X categories, the total text feature of each frame is obtained by splicing them, the dimension is X*C, and finally frames are copied to obtain the final text feature , the dimension is *X*C. The feature vector is used as a semantic guide source throughout the subsequent segmentation process.

[0032] The input video is uniformly sampled at a rate of 1 frame / second (fps) to convert it into an image sequence . The t-th frame image is input into the image encoder of CLIP to extract multi-scale image features: , wherein , denotes that the image encoder of CLIP is used to obtain image features of different sizes, denotes image features of different layers, s denotes the scale level of the features, and there are S layers in total. These features are arranged from shallow to deep in structure, and the spatial information represented changes layer by layer.

[0033] The image encoder obtains image features of a block area on each frame image, so the obtained multi-layer image features are input into the pixel decoder to obtain pixel-level image features of different sizes: , wherein, denotes that the pixel decoder is used to obtain pixel-level image features, denotes pixel-level image features of different layers obtained, and s denotes the scale level of the features, and there are S layers in total. The pixel-level image features obtained here do not carry text category information, and will be used for feature fusion in the subsequent recognition stage.

[0034] After obtaining the pixel-level image features at different scales After that, the application introduces a visual dominant semantic enhancement mechanism, which is a cross-modal enhancement based on natural language semantic embedding to guide the pixel features by categories, including the following steps: Perform a pixel dominant cross-attention Cross-Attention operation, taking the pixel features as the query (Query) and the key (Key), and the text semantics as the value (Value) to perform cross-modal attention enhancement: , , wherein are mapping matrices, each layer of pixel features is calculated separately The calculated result is added to the original corresponding layer of pixel-level features and is gate fused: , , , wherein MLP is a multi-layer perceptron structure, performing channel mapping and nonlinear activation, is a mapping function. The obtained is a semantic enhanced multi-layer pixel-level image feature.

[0035] Initialize the learnable query vector and fuse it with the text semantic vector, using the same fusion method as the pixel-level image feature fusion method above, fuse and to obtain , forming a fusion query vector with semantic orientation.

[0036] Send the semantic enhanced pixel feature map and the fusion query to the Transformer decoder together: , wherein , represents the feature embedding vector of the i-th instance in the t-th frame in the Transformer decoder, that is, the instance representation of the instance; represents the mask of the i-th object in the t-th frame; represents the intersection over union score of the mask of the i-th instance in the t-th frame and the real target mask, which is used in the training stage; from the Transformer decoder, the Transformer decoder outputs This score is the score given by the Transformer decoder for determining that the object instance in the i-th mask of the t-th frame is an object, i.e., the object confidence score.

[0037] Step 3, execute the instance-driven TopK time-series matching strategy, including: According to step 2, in the first frame of the video sequence, instances are obtained based on the initial prediction results of the model. The objectivity scores of all instances are evaluated, and an object confidence score is generated for each object instance. And a set of instances with significant foreground features is selected through a dynamic confidence filtering mechanism. The dynamic confidence filtering mechanism calculates the confidence level of all object instances in the current frame. Mean, variance, and skewness of the scores: , , Calculate the dynamic confidence threshold for the current frame based on the mean and variance: , , The first frame is processed first, and then filtered out using the calculated dynamic confidence threshold. Greater than An object instance, its Register into the sequential memory storage unit Memory Bank, construct a preliminary sequential memory storage unit Memory Bank through the first frame, and use it as a reference template for matching subsequent frames; In the subsequent frame processing, the object instances that match the current frame are first filtered out using dynamic confidence, and the dynamic confidence of the current frame is calculated. Filter out Greater than For each object instance selected in the current frame, the Euclidean distance is calculated between its instance representation and the high-confidence instance representations stored in the Memory Bank. The Euclidean distance is... Where X1 represents the instance representation of the object instance, and X2 represents the identifier of each object instance in the Memory Bank; Then, the Hungarian algorithm is used to perform cross-frame matching. The Hungarian cross-frame matching uses the Euclidean distance calculated between the instance representation of the object instance that meets the dynamic confidence in the current frame and the object instance representation in the Memory Bank as the cost to obtain the corresponding matching method with the minimum total cost. Objects with the same instance are labeled with the same number, thereby ensuring the consistency of the instance index and the integrity of the instance trajectory. When the number of object instances stored in the Memory Bank reaches a certain requirement (set to 30, because users typically don't need to look up more than 10 objects at a time in a video, so 30 is sufficient), a hybrid replacement strategy is executed, prioritizing the retention of high-confidence instances. If the storage limit of 30 is reached, but new object instances still need to be added, then the replacement strategy is implemented by calculating the confidence level of each object instance. If a product has not been matched for a certain period of time, the object instance with the smallest product is selected as the replacement. After replacement, a new sequence number is assigned, and the sequence number of the replaced object instance is not used. This avoids information redundancy and improves storage efficiency. Step 4: Improve category recognition capabilities and optimize target classification through multi-scale feature fusion, including: An image feature and pixel-level image feature fusion mechanism is introduced to integrate pixel features at each scale channel by channel. Compared with the original last layer image feature representation Generate fused feature maps This is used for subsequent classification and matching; the image features are the highest-level features of each frame obtained by the CLIP image encoder. , and the obtained multi-scale pixel-level image features Perform layer-by-layer fusion (the method used here) (This is without semantic content enhancement), the layer-by-layer fusion method is: , , , , in , … For each layer, there is a learnable weight matrix used to learn the dimensional changes and fusion mechanism after feature concatenation, which is learned during the training process; As the intermediate fusion result, the fused features obtained from each fusion are fused with the pixel-level image features of the next layer. After each layer is fused, the final fusion result is obtained. .

[0038] After calculating the fusion features, a region aggregation operation is performed based on the instance segmentation mask. The first frame in which each instance appears is selected to obtain the fusion features of this frame, and the mask of the object is obtained based on this frame. The region feature vector of the object is calculated. The formula is: , in Mask of the object in this frame, the specific content of the mask is 1 for the pixel points belonging to the object, and 0 for the pixel points not belonging to the object; The number of pixel points in the mask region; p represents each pixel position in the mask region; The characteristic value of the position. The region feature vector of the object is obtained through the formula .

[0039] Extract the region feature vector corresponding to each instance object , and perform similarity evaluation with the category embedding vector generated by the text encoder to obtain the semantic similarity score , which is used for further target classification and recognition, so as to classify the target, and the category embedding vector comes from the category name input by the user at the beginning The dimension is *X*C, and the text features of each category are obtained The formula for calculating the semantic similarity score is: , Among them represents the similarity score of the i-th mask and the j-th category, that is, its semantic similarity. Because of the continuity correlation in the time dimension, only a given semantic of a frame is needed, and the semantics of the object in the remaining frames are consistent, so frame t is discarded.

[0040] After obtaining the semantic similarity score corresponding to each mask, a multi-factor confidence fusion strategy is adopted, which comprehensively considers the semantic similarity , object perception score and self-evaluation index of the model on the mask boundary , and the weighted aggregation representation is calculated by calculating and fusing various factors, that is , the formula is: , Among them, represents the final score of the object i for the category j; the fusion weight of the multi-factor confidence fusion strategy is automatically learned through the training process, which is used to adjust the influence of semantic similarity, target existence and mask quality on matching confidence in the final matching process.

[0041] After obtaining the final , the mask object with the highest score for each input category can be found, and the object in each frame of the video frame is segmented to obtain the mask, and the mask region is displayed, which is the segmentation result of the object corresponding to the category required by the user.

[0042] Step 5, result evaluation; Dataset: The COCO dataset is used in the training stage, which is a picture dataset covering 80 categories and mainly used for training segmentation and recognition modules; the lvvis dataset is used in the final evaluation stage, which is composed of video data. The use of the dataset also reflects the open vocabulary capability. The category types in the two datasets are not completely consistent, with some overlap but also some new categories.

[0043] Evaluation indicators include: Average precision AP: It measures the average precision of the model at different intersection over union threshold values. The proportion of mask overlap between the segmentation mask obtained by the above method and the correct mask ground truth mask is calculated, and the proportion is divided into 0.5-0.95, every 0.05 is divided once, the correct precision Precision and recall Recall at each proportion are calculated, then the Precision-Recall curve is integrated, the area is obtained, and finally the average value of the area at each proportion is calculated to obtain the average precision AP. As shown in Figure 2 The user input segmentation category name is person and dog, Figure 2 Four frames in the video frame are selected to show the segmentation results, and the objects segmented by the pink transparent mask are person, and the objects segmented by the orange transparent mask are dog.

[0044] Open vocabulary average precision APn: This indicator is specifically used to measure the segmentation performance of the system on the training unseen categories, and is the core indicator for evaluating the open vocabulary adaptation capability. APn statistics only involves categories in the test set that do not appear in the training stage. Through matching with the semantic embedding corresponding to the natural language description, the semantic perception and mask generation capability of the system on completely unknown categories are measured.

[0045] Experimental results: The Baseline method is trained on the COCO dataset, and the evaluation result on the LVVIS dataset is AP = 9.4, APn = 13.7; The result of the method of the application under the same condition is AP = 15.4, APn = 21.1.

[0046] Result analysis and advantages: Compared with Baseline, the method of the application improves the AP indicator by 6.0 percentage points and the APn indicator by 7.4 percentage points; It is shown that the application significantly improves the segmentation performance on known categories and unseen categories, especially in the open vocabulary scenario.

[0047] The experimental results fully embody the advantages and realizability of the method in a complex video scene, and avoid the deficiencies of the prior art in cross-frame consistency and unknown category recognition.

[0048] The application provides a multi-dimensional enhanced open vocabulary video instance segmentation method, and there are many methods and approaches to specifically implement the technical solution, and the above description is only the preferred embodiment of the application, and it should be pointed out that, for ordinary skilled persons in the technical field, some improvements and refinements can be made without departing from the principle of the application, and these improvements and refinements should also be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be realized by using the prior art.

Claims

1. A multi-dimensional augmented open vocabulary video instance segmentation method, characterized in that, Comprising the following steps: Step 1, mathematical modeling and analysis of open vocabulary video instance segmentation task; Step 2, establish a class-related segmentation model, pre-process the video input by the user, and sample the frames of the video; the class-related segmentation model enhances the image features obtained by each frame according to the user input of any class name and the query used to represent the object in the segmentation process; the enhanced image features and the query are interacted in the transformer decoder to realize the class-related segmentation; Step 3, execute instance-driven TopK timing matching strategy; Step 4, improve the recognition ability of the class by multi-scale feature fusion, and optimize the target classification.

2. The method of claim 1, wherein, Step 1 includes: Given an input video as a sequence of frames I, the length of the video is , the t-th frame is , denoted as: ， Each frame contains two or more object instances, where each object instance is represented by a mask denoted as , is denoted as the xth object instance in the tth frame, each frame contains N object instances, denoted as is denoted as the xth object instance in the tth frame, each frame contains N object instances, denoted as contains the masks of N different objects in the current frame A user-defined text input model is introduced to represent the category target, setting the category name description of the given user input as , is a text in natural language form; Participate in the generation of the mask in the segmentation process; Final masks are computed by calculating the similarity of the masks and the category description to get the final mask of the object that meets the user's input requirements.

3. The method of claim 2, wherein, Step 2 includes: First the user inputs a category name description as a natural language string which is mapped by the text encoder of the pre-trained model CLIP into a multi-dimensional semantic embedding vector: ， wherein, represents a text encoder function, is a text feature obtained according to an input natural language category name, the initial input natural language category name is obtained from a pre-trained text classification model, and the text feature is obtained from a pre-trained text encoder model. dimension is wherein, is the total number of frames of the video, X is the input category number, C is the feature dimension channel number obtained by the text encoder, the dimension of the feature vector obtained by the initial input of each category is 1*C, the total text feature of each frame is obtained by splicing the feature vectors of X categories, and the dimension is X*C. After obtaining the feature vectors of X categories, the final text feature is obtained by copying frames, and the final text feature has a dimension of *X*C. uniformly sampling the input video at a rate of 1 frame per second into an image sequence the t-th frame image inputting the image encoder of the pre-trained model CLIP to extract multi-scale image features: ， wherein , denotes an image encoder of the pre-trained model CLIP, denotes the image features of the s-th layer, s denotes the scale level of the features, and S is the total number of layers. Send the obtained image features to the pixel decoder to obtain pixel-level image features of images of different sizes: ， wherein, denotes a pixel decoder for obtaining pixel-level image features, denotes the obtained pixel-level image features of the s-th layer; Introduce a visual dominant semantic enhancement mechanism, which is a cross-modal enhancement mechanism that guides the pixel features based on natural language semantic embedding, including the following steps: Perform pixel dominant cross-attention Cross-Attention operation, taking pixel features as query Q and key K, and text semantics as value V, to perform cross-modal attention enhancement: ， ， where d represents the dimension of the query Q; softmax is a normalization function used to convert the similarity score of each pixel position into a probability distribution, so that the attention weight of each pixel position to the text semantics is between 0 and 1; represents the information that needs to be fused and enhanced obtained by cross attention; T represents transposition; represents the information that needs to be fused and enhanced obtained by cross attention; T represents transposition; These are the mapping matrices for the query, key, and value, respectively; each layer of pixel features is calculated separately. Then Pixel-level features of the original corresponding layer Perform addition and gating fusion: ， ， ， wherein is an intermediate feature that fuses ; represents a gating factor; sigmoid is a normalization function; MLP is a multi-layer perceptron structure; is a mapping function; is a semantic-enhanced s-th layer pixel-level image feature; Initializing a learnable query vector , and fusing the text semantic vector to obtain a fusion query vector with semantic direction ;​​ semantically enhanced pixel feature map with the fused query vector into the Transformer decoder: ， wherein ; represents the feature embedding vector for the i-th instance in the t-th frame in the Transformer decoder; represents the mask for the i-th object in the t-th frame; represents the intersection over union score of the mask for the i-th instance in the t-th frame with the ground truth mask; object confidence score from the Transformer decoder, which outputs the score as the object confidence score.

4. The method of claim 3, wherein, Step 3 includes: Step 3-1, in the first frame of the video sequence, according to the initial prediction result of the class correlation segmentation model, the object instances are obtained, whether each object instance is an object is evaluated, the object confidence score of each object instance is generated, and a group of instance representations with significant foreground features are screened out through a dynamic confidence filtering mechanism The object instances with significant foreground features are registered into the timing memory storage unit Memory Bank as the reference template for subsequent frame matching The object instances with significant foreground features are registered into the timing memory storage unit Memory Bank as the reference template for subsequent frame matching Step 3-2, in the subsequent frame processing process, first filter the object instances that meet the current frame through dynamic confidence, and perform one-to-one matching between the high-confidence instance representations retained in the timing memory storage unit Memory Bank and the filtered instance representations that meet the conditions of the current frame, and perform cross-frame matching operation through the minimum cost optimization algorithm, to ensure the consistency of instance index and the integrity of instance trajectory; Step 3-3, if there is an object instance in the current frame which is not successfully matched but the object confidence is still significant, the object instance is determined as a possible new instance by the object confidence score of the object instance, and an incremental manner is added to the time memory storage unit Memory Bank to support the dynamic expansion of the object instance in the video; the object confidence is still significant refers to the object confidence score greater than the dynamic confidence threshold ; Step 3-4, when the storage capacity of the timing memory storage unit Memory Bank reaches the upper limit, execute the hybrid replacement strategy, preferentially retain high-confidence instances, and remove obsolete records that have not been successfully matched for more than two consecutive frames.

5. The method of claim 4, wherein, In step 3-1, the dynamic confidence filtering mechanism comprises: calculating the mean , variance , and skewness of the scores of all object instances in the current frame : ， ， calculating a dynamic confidence threshold for the current frame from the mean and the variance : ， ， where exp denotes the natural exponential function; denotes an intermediate parameter; The first frame is processed first, and then filtered out using the calculated dynamic confidence threshold. Score greater than object instances, will Register into the sequential memory storage unit Memory Bank, and construct the initial sequential memory storage unit Memory Bank through the first frame.

6. The method of claim 5, wherein, Step 3-2 includes: The dynamic confidence of the current frame is calculated The object instance with a dynamic confidence greater than is screened out The Euclidean distance between the instance representation of each object instance screened out by calculation of the current frame and the instance representation of each object instance in the time sequence memory storage unit Memory Bank is calculated, and then the Hungarian algorithm is used to perform cross-frame matching to label the same object instance with the same serial number.

7. The method of claim 6, wherein, Step 3-3 includes: When the number of object instances in the current frame filtered out by dynamic confidence screening is greater than the number of object instances in the temporal memory storage unit Memory Bank, it indicates that there are new object instances in the current frame and they do not exist in the temporal memory storage unit Memory Bank. After cross-frame matching is completed, the remaining unmatched object instances with object confidence scores greater than are added to the temporal memory storage unit Memory Bank as new object instances appearing in the current frame, i.e., the object instances with object confidence scores greater than are added to the temporal memory storage unit Memory Bank and are given new sequence numbers for matching in subsequent frames.

8. The method of claim 7, wherein, Step 3-4 includes: When the number of object instances stored in the Memory Bank reaches a threshold, a hybrid replacement strategy is employed. If the storage limit is reached, but new object instances still need to be added, then calculation is performed. Choose the product The smallest object instance is replaced, and a new sequence number is assigned after the replacement, where K represents the time when no match was found.

9. The method of claim 8, wherein, Step 4 includes: Step 4-1, introduce image feature and pixel-level image feature fusion mechanism, integrate each scale pixel feature channel by channel with the original last layer image feature representation , generate fusion feature map ; Step 4-2, perform a region aggregation operation based on the instance segmentation mask to extract a region feature vector corresponding to each object instance , and the category embedding vector generated by the text encoder Perform similarity evaluation to obtain a semantic similarity score ; Step 4-3, a multi-factor confidence fusion strategy is adopted to comprehensively consider semantic similarity , object confidence score and model intersection over union score on mask boundary , the weighted aggregation representation is calculated by calculating the fusion of each factor .

10. The method of claim 9, wherein, Step 4-1 includes: the image feature of the image encoder of CLIP obtains the maximum layer of the image feature of each frame of image , the multi-scale pixel-level image feature obtained is fused layer by layer, and the fusion mode is ， ， ， ， wherein is a learnable weight matrix of the Sth layer; is the intermediate fusion result, and each time the fusion feature is fused with the pixel-level image feature of the next layer, and after each layer is fused, the final fusion feature is obtained ; Step 4-2 includes: obtaining the final fusion feature Afterwards, the frame in which each object instance first appears is selected, the fusion feature of the frame is obtained, and the region feature vector of the object instance is calculated according to the mask of the object instance The region feature vector of the object instance is calculated The formula is: ， Wherein, the specific content of the mask is that the pixel point belonging to the object instance is 1, and the pixel point not belonging to the object instance is 0; is the number of pixel points in the mask region; p represents each pixel position in the mask region; represents the feature value of the position p; Computing region feature vectors Similarity scores between the text feature vectors from the user-input category names and the initial obtained Dimension *X*C, and then obtain the text features of each category, the text features of the i-th category are represented as The similarity score between the i-th mask and the j-th category is calculated using the following formula :​ ， Take the similarity score as the semantic similarity; Step 4-3 includes obtaining a similarity score , an object confidence score , and an intersection over union score of the model against the mask boundary After that, the weighted aggregated representation is obtained by a multi-factor confidence fusion strategy : ， wherein, represents the final score of the object instance i for the class j; fusion weight are automatically learned through the training process.

Images