A video concept interpretation method based on SAM 2

Abstract

The invention discloses a video concept interpretation method based on SAM 2, which mainly comprises the following steps: defining a plurality of semantic categories for a target task to be performed, obtaining a semantic segmentation data set and a prompt corresponding to each semantic category, and training a prompt generation model by using the prompt corresponding to each semantic category and the semantic segmentation data set; using the trained prompt generation model to generate a corresponding prompt for each frame of an image in a video to be interpreted, extracting features of the video to be interpreted frame by frame, and inputting the prompt and features of each frame of the image into a SAM 2 model to obtain a spatiotemporal concept set; considering each spatiotemporal concept as a participant in a game, classifying the video to be interpreted by using a video classification model, and calculating the Shapley value of each spatiotemporal concept based on the predicted category of the model to be interpreted, thereby generating an optimal spatiotemporal concept interpretation of the video data, and performing high-quality concept interpretation on the predicted result of a 3D neural network model.

Description

Technical Field

[0001] The present invention relates to the field of explainable artificial intelligence, and in particular to a video concept interpretation method based on SAM 2. Background Art

[0002] In recent years, due to the rapid development of large-scale data sets and computer computing power, deep neural networks have been widely used in various computer vision applications, such as semantic segmentation, image classification, object detection, etc. However, due to the large number of parameters and highly nonlinear characteristics, neural networks are still a black box for users. This feature limits the understanding of the model's decision-making process, especially in scenarios that require transparency and explainability (such as medical, finance, etc.). To solve this problem, explainable artificial intelligence (Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access, 6:52138–52160, 2018.) has become a popular solution, which aims to explain the results of neural network predictions.

[0003] Among the interpretable methods for two-dimensional image classification neural networks, the most common one is to provide pixel-level or super-pixel-level explanations for image classification results. This includes using the back propagation of the neural network to evaluate the impact of input feature pixels on the model classification results, and also includes perturbing the input pixels to measure the impact of pixel perturbations on the image classification results, generating pixel-level explanations. However, the pixel-level explanation results are often difficult for humans to understand and lack corresponding semantic explanations. In recent years, the method of using high-level concepts to explain neural networks has attracted great attention. The most representative work is ACE (Ghorbani A, Wexler J, Zou J, et al. Towards Automatic Concept-based Explanations [J]. 2019. DOI: 10.48550 / arXiv.1902.03129.), which uses feature vectors to cluster superpixels in the dataset into a set of concepts, and automatically identifies and uses concepts that can be understood by humans to generate global explanations, making the output of the model easier to explain and understand. However, the conceptual explanations generated by ACE are still difficult to understand in most cases, lacking clear outlines and clear semantic definitions. In order to more accurately define clear and easy-to-understand conceptual information, EAC (A. Sun, P. Ma, Y. Yuan, and S. Wang, “Explain anyconcept: Segment anything meets concept-based explanation,” Advances in Neural Information Processing Systems, vol. 36, 2024.) uses the Segment Anything Model (SAM) to achieve high-quality semantic segmentation and automatically extract relevant concept sets from a given image. At the same time, EAC combines the resulting concept set obtained by SAM segmentation with explainable artificial intelligence methods, thereby promoting XAI tasks with high credibility and understandability.

[0004] Although EAC has shown excellent interpretability in image classification, in the field of 3D videos, since video data contains a lot of spatiotemporal information, simply applying SAM to frame-by-frame segmentation and performing conceptual explanations is not only time-consuming, but the segmentation results often fail to meet expectations. Another method for conceptual explanation in the field of 3D videos is STCE (YingJi, Yu Wang, and Jien Kato. Spatial-temporal concept based explanation of 3dconvnets. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 15444–15453, 2023.), which extends ACE from 2D to 3D video scenes. It clusters spatiotemporal segments in video datasets, generates a global concept set, and scores each concept according to its contribution to network prediction, ultimately providing a global explanation for 3D convolutional neural networks. However, in the video field, the appearance of entities may change significantly due to motion, deformation, occlusion, illumination changes and other factors, and the concept set generated by STCE cannot accurately define the segmentation semantic information in spatiotemporal scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a video concept interpretation method based on SAM 2 to address the above-mentioned deficiencies of the existing conceptual interpretation algorithms in the video field.

[0006] According to a first aspect of an embodiment of the present application, a video concept interpretation method based on SAM 2 is provided, comprising:

[0007] According to the target task to be carried out, several semantic categories are defined, and the semantic segmentation dataset and prompts corresponding to each semantic category are obtained. The prompt generation model is trained using the prompt and semantic segmentation dataset corresponding to each semantic category.

[0008] Generate corresponding prompts for each frame of the video to be interpreted using the trained prompt generation model, extract features of the video to be interpreted frame by frame, input the prompts and features of each frame into the SAM 2 model, and obtain a spatiotemporal concept set;

[0009] Each spatiotemporal concept is regarded as a participant in the game, the video to be interpreted is classified using a video classification model, and the Shapley value of each spatiotemporal concept is calculated based on the predicted category of the model to be interpreted, so as to generate the optimal spatiotemporal concept interpretation of the video data.

[0010] Furthermore, for the target task to be performed, several semantic categories are defined, and the semantic segmentation dataset and prompt corresponding to each semantic category are obtained. The prompt generation model is trained using the prompt and semantic segmentation dataset corresponding to each semantic category, including:

[0011] Define the target tasks to be carried out Semantic categories, and obtain corresponding semantic segmentation datasets based on each semantic category;

[0012] For each semantic category defined, introduce a corresponding prompt , where each prompt Corresponding to semantic categories, and use the prompts corresponding to each semantic category and the semantic segmentation dataset to train the prompt generation model.

[0013] Furthermore, the trained prompt generation model is used to generate corresponding prompts for each frame of the video to be interpreted, and the features of the video to be interpreted are extracted frame by frame. The prompts and features of each frame are input into the SAM 2 model to obtain a spatiotemporal concept set, including:

[0014] Obtaining a video to be interpreted, extracting images frame by frame from the video to be interpreted, and inputting each frame of the image into an encoder to obtain its corresponding image features;

[0015] Generate corresponding prompts for each frame of the video to be explained using the trained prompt generation model;

[0016] The image features of each frame of the video to be interpreted and the corresponding prompt are input into the SAM 2 model to obtain The video mask of spatiotemporal information, that is, the spatiotemporal concept set, is recorded as:

[0017]

[0018] in is the mask set output by the SAM 2 model, Indicates A spatiotemporal mask is used to capture the spatiotemporal correlation information of different semantic categories in the video to be interpreted.

[0019] Furthermore, each spatiotemporal concept is regarded as a participant in the game, the video to be explained is classified using a video classification model, and the Shapley value of each spatiotemporal concept is calculated based on the predicted category of the model to be explained, so as to generate the optimal spatiotemporal concept explanation of the video data, including:

[0020] For including A set of space-time concepts with space-time masks , each spatiotemporal concept is regarded as a participant in the game, the video classification model is used to classify the video to be explained, and the predicted category of the model to be explained is regarded as the utility function. Then for the spatiotemporal concept , whose marginal contribution is the video classification model in the concept set and collection The difference between the predicted results of

[0021] Based on the concept of space and time The marginal contribution of computing the space-time concept set Each concept The Shapley value of

[0022] Based on each space-time concept The Shapley value of the video to be interpreted is used to obtain the optimal spatiotemporal concept interpretation given for the predicted category of the video to be interpreted, wherein the optimal spatiotemporal concept interpretation is a subset of spatiotemporal concepts that maximizes the Shapley value.

[0023] Furthermore, the concept of space and time The marginal contribution is

[0024]

[0025] Among them It is in the video to be explained The corresponding model prediction results when the represented spatiotemporal mask is blocked.

[0026] Furthermore, the space-time concept set Each concept The Shapley value By the following calculation:

[0027]

[0028] Furthermore, the Shapley value is estimated by Monte Carlo method.

[0029]

[0030] in, It is times randomly sampled alliances.

[0031] According to a second aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0032] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0033] one or more processors;

[0034] A memory for storing one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0036] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0037] The technical solution of the present invention is summarized as follows:

[0038] 1. Propose a prompt generation model that automatically generates prompts. Class semantics, collect corresponding data sets, train the model to learn the association between semantic categories and prompts, and generate trainable prompts to replace manual input in subsequent tasks.

[0039] 2. A method for automatically acquiring spatiotemporal concepts in videos. The target video sequence is encoded frame by frame, and combined with the automatically generated prompt input, the corresponding spatiotemporal mask set is generated through the SAM 2 model.

[0040] 3. A key spatiotemporal concept interpretation method based on Shapley values. Based on the previously obtained spatiotemporal mask set, the Shapley value method is used to quantify the contribution of each spatiotemporal concept to the model prediction results. The marginal contribution of each concept is defined as the change in utility of the model prediction after the concept is added or removed. Due to the high complexity of accurately calculating the Shapley value, Monte Carlo sampling is used to approximate the Shapley value. Finally, the concept subset that maximizes the Shapley value is selected as the optimal explanation, and it is marked as a mask to generate the optimal spatiotemporal concept interpretation of the target video.

[0041] The beneficial effect of the present invention is that by applying the SAM 2 model to generate a concept set with spatiotemporal information, the concept interpretation generated based on this can greatly promote human understanding, improve the interpretability of the neural network, and ensure the computational efficiency and reliability of the interpretation. The present invention proposes to apply the prompt generation model to the automatic generation of prompts for video data, thereby effectively avoiding the target loss problem that may occur when using SAM 2 to directly generate a video mask set, and improving the accuracy and integrity of video data processing.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] Figure 1 It is a principle diagram showing a video concept interpretation method based on SAM 2 according to an exemplary embodiment.

[0045] Figure 2 The figure is a block diagram showing a video concept interpretation device based on SAM 2 according to an exemplary embodiment.

[0046] Figure 3 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0048] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] The core technology of this invention is to build a concept hint generation model related to the prediction results and combine it with SAM2 (Segment Anything Model 2) to generate a spatiotemporal concept set. Furthermore, by integrating the Shapley value method, high-quality concept explanations are performed on the prediction results of the 3D neural network model, thereby improving the model interpretability.

[0051] The present invention proposes a video concept interpretation method based on SAM 2, such as Figure 1As shown, the following steps are included:

[0052] (1) Training the cue generation model: Define the cue generation model of interest in the target task. The corresponding semantic categories are obtained, and the semantic segmentation dataset and prompts of each semantic category are obtained. The prompt generation model is trained using the prompt and semantic segmentation dataset corresponding to each semantic category, so that it can learn the mapping relationship between each semantic category and the prompt, so as to generate a set of The prompts that match the semantic categories can be included in the following sub-steps:

[0053] (1.1) Define the target tasks to be carried out Semantic categories, such as wheels, windows, front of the car and other specific object concepts, are used to obtain corresponding semantic segmentation data sets based on each semantic category. In one embodiment, data can be collected from the Kinetics 700 data set.

[0054] (1.2) For each semantic category defined in step (1.1), introduce the corresponding prompt , where each prompt Corresponding to In this embodiment, a prompt box is used as a specific application prompt, and a prompt generation model is trained using the prompt corresponding to each semantic category and a semantic segmentation dataset.

[0055] The prompt generation model in this application is based on ViT (Vision Transformer). After extracting image features using ViT, a prompt suitable for the target object of the corresponding semantic category is generated through an additional feedforward layer or a multi-head attention mechanism. In one embodiment, Prompt Head can be used, with the prompt as the query Query, the features extracted by ViT as the key Key and the value Value, and the features extracted by ViT are weighted and aggregated through multi-head attention. The trained prompt generation model can generate a set of prompts to replace the manually input prompts in subsequent tasks, thereby automatically performing guided segmentation tasks.

[0056] (2) Obtaining a spatiotemporal concept set: Use the trained prompt generation model to generate corresponding prompts for each frame of the video to be interpreted, extract the features of the video to be interpreted frame by frame, and input the prompts and features of each frame into the SAM 2 model to obtain a spatiotemporal concept set. The specific steps include the following:

[0057] (2.1) Obtain the video to be interpreted, extract images frame by frame from the video to be interpreted, and convert each frame into Input to the encoder to get its corresponding image features ,in ( indicates the number of frames).

[0058] (2.2) Use the prompt generation model trained in step (1) to generate corresponding prompts for each frame of the video to be interpreted .

[0059] (2.3) The features of each frame image The prompt corresponding to each frame of the image Input them into the SAM 2 model together to obtain The video mask of spatiotemporal information, that is, the spatiotemporal concept set, is recorded as:

[0060]

[0061] in is the mask set output by the SAM 2 model, Indicates spatiotemporal mask, i.e., the corresponding The semantic mask of the semantic category prompt can accurately capture the spatiotemporal correlation information of different semantic categories in multiple frames of video. is the semantic category appearing in the video to be explained, .

[0062] It should be noted that there is no strict order between step (2.1) and step (2.2).

[0063] (3) Calculate the optimal explanation based on the Shapley value: Consider each spatiotemporal concept as a participant in the game, use the video classification model to classify the video to be explained, and calculate the Shapley value of each spatiotemporal concept based on the predicted category of the model to be explained, so as to generate the optimal spatiotemporal concept explanation for the video data. Specifically, it includes the following sub-steps:

[0064] (3.1) For the set of spatiotemporal concepts generated in (2) , the goal is to explain the model output of the video to be explained, and the explanation can be expressed as A subset of Based on the definition of Shapley value, each spatiotemporal concept is regarded as a participant in the game, and the model's predicted category of the video to be interpreted is regarded as a utility function. , its marginal contribution It can be defined as: the model is in the concept set and collection The difference between the predicted results of ,Right now:

[0065]

[0066] Among them It is in the video to be explained The model's prediction results when the represented spatiotemporal mask is blocked.

[0067] In a specific implementation, the model may be a model of the 3D convolutional neural network category, and in one embodiment may be a C3D (Convolutional 3D) model, and the model output is a category label of the video.

[0068] (3.2) Furthermore, the space-time concept set Each concept The Shapley value By the following calculation:

[0069]

[0070] However, since the computational complexity of the Shapley value grows exponentially with the size of the set, the Monte Carlo method is used to approximate the Shapley value. Specifically, for each spatiotemporal concept Random Sampling coalitions, and approximate the Shapley value as:

[0071]

[0072] in, It is Sub-sampling alliance.

[0073] (3.3) According to the Shapley value of each space-time concept calculated in (3.2), the optimal interpretation is defined as the subset of space-time concepts that maximizes the Shapley value. The specific form is as follows:

[0074]

[0075] in, , represents the optimal concept subset The Shapley value of is the sum of the Shapley values ​​of all the space-time concepts it contains. Finally, the selected subset The annotations are used as masks to generate the optimal spatiotemporal concept explanations of the target explanation video.

[0076] Corresponding to the aforementioned embodiment of the video concept interpretation method based on SAM 2, the present application also provides an embodiment of a video concept interpretation device based on SAM 2.

[0077] Figure 2 FIG. 1 is a block diagram of a video concept interpretation device based on SAM 2 according to an exemplary embodiment. Figure 2 , the device may include:

[0078] The prompt generation model module 21 is used to define a number of semantic categories according to the target task to be performed, and obtain the semantic segmentation data set and prompt corresponding to each semantic category, and train the prompt generation model using the prompt and semantic segmentation data set corresponding to each semantic category;

[0079] The spatiotemporal concept set acquisition module 22 is used to generate corresponding prompts for each frame of the video to be interpreted using the trained prompt generation model, extract the features of the video to be interpreted frame by frame, and input the prompts and features of each frame of the image into the SAM 2 model to obtain the spatiotemporal concept set;

[0080] The optimal explanation calculation module 23 is used to regard each spatiotemporal concept as a participant in the game, classify the video to be explained using the video classification model, and calculate the Shapley value of each spatiotemporal concept based on the predicted category of the model to be explained, so as to generate the optimal spatiotemporal concept explanation for the video data.

[0081] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0082] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0083] Accordingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the video concept interpretation method based on SAM 2 as described above.

[0084] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the video concept interpretation method based on SAM 2 as described above. Figure 3 As shown in FIG. 1 , a hardware structure diagram of a video concept interpretation device based on SAM 2 provided by an embodiment of the present invention is provided for any device with data processing capability, except Figure 3In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0085] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the video concept interpretation method based on SAM 2 as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0086] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. A video concept interpretation method based on SAM 2, characterized in that: The steps include: According to the target task to be carried out, several semantic categories are defined, and the semantic segmentation dataset and prompts corresponding to each semantic category are obtained. The prompt generation model is trained using the prompt and semantic segmentation dataset corresponding to each semantic category. Generate corresponding prompts for each frame of the video to be interpreted using the trained prompt generation model, extract features of the video to be interpreted frame by frame, input the prompts and features of each frame into the SAM 2 model, and obtain a spatiotemporal concept set; Considering each spatiotemporal concept as a participant in the game, classifying the video to be interpreted using a video classification model, and calculating the Shapley value of each spatiotemporal concept based on the predicted category of the video to be interpreted, thereby generating an optimal spatiotemporal concept interpretation of the video data; Each spatiotemporal concept is regarded as a participant in the game, the video to be explained is classified using a video classification model, and the Shapley value of each spatiotemporal concept is calculated based on the predicted category of the video to be explained, so as to generate the optimal spatiotemporal concept explanation of the video data, including: For including A set of space-time concepts with space-time masks , regard each spatiotemporal concept as a participant in the game, use the video classification model to classify the video to be explained, and regard the predicted category of the video to be explained as a utility function. Then for the spatiotemporal concept , whose marginal contribution is the video classification model in the concept set and collection The difference between the predicted results of Based on the concept of space and time The marginal contribution of computing the space-time concept set Each concept The Shapley value of Based on each space-time concept The Shapley value of the video to be interpreted is used to obtain the optimal spatiotemporal concept explanation given to the predicted category of the video to be interpreted, wherein the optimal spatiotemporal concept explanation is a subset of spatiotemporal concepts that maximizes the Shapley value; Concept of space and time The marginal contribution is ， Among them It is in the video to be explained The corresponding model prediction results when the represented spatiotemporal mask is blocked.

2. The method according to claim 1, characterized in that According to the target task to be carried out, several semantic categories are defined, and the semantic segmentation datasets and prompts corresponding to each semantic category are obtained. The prompt generation model is trained using the prompts and semantic segmentation datasets corresponding to each semantic category, including: Define the target tasks to be carried out Semantic categories, and obtain corresponding semantic segmentation datasets based on each semantic category; For each semantic category defined, introduce a corresponding prompt , where each prompt Corresponding to semantic categories, and use the prompts corresponding to each semantic category and the semantic segmentation dataset to train the prompt generation model.

3. The method according to claim 1, characterized in that The trained prompt generation model is used to generate corresponding prompts for each frame of the video to be interpreted, and the features of the video to be interpreted are extracted frame by frame. The prompts and features of each frame are input into the SAM 2 model to obtain a set of spatiotemporal concepts, including: Obtaining a video to be interpreted, extracting images frame by frame from the video to be interpreted, inputting each frame of the image into an encoder, and obtaining its corresponding image features; Generate corresponding prompts for each frame of the video to be explained using the trained prompt generation model; The image features of each frame of the video to be interpreted and the corresponding prompt are input into the SAM 2 model to obtain The video mask of spatiotemporal information, that is, the spatiotemporal concept set, is recorded as: ， in is the mask set output by the SAM 2 model, Indicates A spatiotemporal mask is used to capture the spatiotemporal correlation information of different semantic categories in the video to be interpreted.

4. The method according to claim 1, characterized in that: Space and time concept collection Each concept The Shapley value By the following calculation: 。 5. The method according to claim 1, characterized in that The Shapley value is obtained by Monte Carlo method ， in, It is times randomly sampled alliances.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images