Video recognition and segmentation method based on mamba architecture, storage medium and device

Abstract

The application discloses a video recognition and segmentation method based on Mamba architecture, a storage medium and equipment. First, the video to be recognized and segmented is preprocessed, the resolution of the video is reduced, and the video is converted into a standardized format suitable for further analysis. Then, an efficient video backbone network is used to extract the visual spatiotemporal features of the video, and the extracted features are input into an improved time action detector. The time action detector significantly improves the recognition ability of short-time fast-changing steps and long-time stable steps by effectively fusing action features of different durations. Finally, the category and corresponding time segmentation result of each experimental step are output. The method can accurately capture the changes and action features of the experimental steps, significantly improve the precision and robustness of the detection model, and is efficient and accurate. Meanwhile, the method effectively solves the problems of large consumption of human resources, unstable detection effect and low efficiency, and has high application value and wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent education of artificial intelligence, and also belongs to the technical field of deep learning, and relates to a technology for recognizing and segmenting experimental videos in middle schools, and mainly relates to a video recognition and segmentation method based on a Mamba architecture, a storage medium and equipment. BACKGROUND

[0002] Experimental teaching, as a key link for cultivating students' scientific literacy, practical ability and innovation ability, occupies an irreplaceable position in the modern education system. It not only deepens students' understanding of subject knowledge, but also promotes the development of students' critical thinking, problem-solving ability and team cooperation spirit through hands-on operation. With the increasing emphasis on STEM (Science, Technology, Engineering and Mathematics) education in various countries, experimental teaching has become an important part of national curriculum programs and curriculum standards.

[0003] However, in the current experimental teaching practice, especially in the middle school stage, there are many challenges affecting the quality and effectiveness of teaching. First, due to large class size, limited teacher resources and other factors, traditional manual evaluation methods are difficult to achieve comprehensive and timely evaluation of students' experimental skills. Second, the subjective bias in the process of manual scoring may lead to inconsistent scores given by different teachers or the same teacher at different time points, thereby affecting the fairness and objectivity of the evaluation results. In addition, with the popularization of online education resources, more and more teaching activities begin to adopt video recording forms, and how to effectively utilize these video materials to assist experimental teaching has become one of the problems to be solved.

[0004] In view of the above problems, in recent years, researchers have begun to explore the use of artificial intelligence technology to improve the evaluation mechanism in the experimental teaching process. Artificial intelligence technology can provide instant feedback through automatic analysis of students' experimental operation videos, and help establish a standardized evaluation system, thereby significantly improving the quality and efficiency of experimental teaching. However, some existing solutions still face some limitations when applied to middle school experimental videos. For example, many existing action recognition algorithms perform poorly when dealing with experimental videos containing complex semantic information and continuous long sequence actions, making it difficult to accurately extract key step information; at the same time, these algorithms are often computationally inefficient, unable to meet the real-time requirements in teaching scenarios.

[0005] In recent years, artificial intelligence technology, especially deep learning models, has made significant progress in image recognition, natural language processing, and other fields, and has gradually been applied to the education field. Among them, the Mamba architecture, as an efficient data processing platform, stands out with its powerful parallel computing capabilities and flexible modular design. The Mamba architecture was initially developed for high-performance computing scenarios, such as large-scale data processing and complex algorithm acceleration, and its features include support for multiple programming languages, ease of extension, and optimized memory management, making it very suitable for application scenarios involving large amounts of video data. With the development of the Mamba architecture, researchers have begun to explore its application in video content analysis tasks, especially in situations that require high precision and real-time performance. In practical applications, especially in teaching scenarios, real-time performance is crucial, and previous methods have difficulty meeting this demand, limiting their effectiveness in actual teaching. SUMMARY

[0006] The present application is aimed at the problems of accuracy, timeliness, and operability in the identification and segmentation process of existing video, providing a video identification and segmentation method based on Mamba architecture, storage medium, and equipment. First, the video to be identified and segmented is preprocessed to reduce the video resolution and convert it into a standardized format suitable for further analysis. Then, an efficient video backbone network is used to extract the visual spatiotemporal features of the video, and the extracted features are input into an improved temporal action detector. The temporal action detector significantly improves the recognition ability of short-term rapid change steps and long-term stable steps by effectively fusing action features of different durations. Finally, the class and corresponding time segmentation result of each experimental step are output. The method can accurately capture the changes and motion features of experimental steps, significantly improving the precision and robustness of the detection model, and is efficient and accurate. It effectively solves the problems of high consumption of human resources, unstable detection effect, and low efficiency, and has high application value and wide application prospect.

[0007] To achieve the above purpose, the technical solution adopted by the present application is as follows: a video identification and segmentation method based on Mamba architecture, comprising the following steps:

[0008] S1, data preprocessing: pre-processing the input experimental video to be identified and segmented to reduce the video resolution;

[0009] S2, feature extraction: extracting spatiotemporal features from the pre-processed video in step S1 through a large video backbone network VideoMAE;

[0010] S3, nonlinear mapping: inputting the spatiotemporal features extracted in step S2 into a temporal action detector based on a hollow Mamba, and performing nonlinear mapping on the features through an embedding module;

[0011] S4, spatio-temporal information fusion: after the spatio-temporal features embedded in step S3 module, different scale spatio-temporal information is extracted through intra-block parallel components or inter-block parallel components, and different scale spatio-temporal information is fused through summation; the intra-block parallel component is based on multi-convolution information fusion inside the hollow Mamba, and the inter-block parallel component directly fuses the output features of multiple hollow Mambas;

[0012] S5, identification and segmentation: the multi-scale features fused in step S4 are down-sampled in a feature pyramid manner, and the classification head and the regression head are used to output the category and time segmentation result of each experimental step respectively, so as to realize the identification and segmentation of the experimental step.

[0013] As an improvement of the present application, the resolution reduction in the step S1 data preprocessing mainly comprises: adjusting the short side of the video to 480 pixels, keeping the aspect ratio of the video unchanged, automatically calculating and adjusting the corresponding long side resolution, and ensuring the consistency of the video ratio.

[0014] As another improvement of the present application, in step S2, before feature extraction, the input video frame is cropped to a resolution of 224x224; in feature extraction, the video is divided into video segments of 16 frames each, the video frame rate is 30 frames per second, and the step size is 4 frames, so that a feature vector is extracted every 4 / 30 seconds.

[0015] As another improvement of the present application, the workflow of the embedding module in step S3 is: first, the input spatio-temporal features are processed through a 1D convolution layer to extract time sequence features; if the input data needs to be masked, the data is masked before and after the convolution operation, and the masking function is to shield or weight the specific data position; after the convolution calculation is completed, the normalization operation is performed, and the normalization layer will adjust the dimensions of the input data according to its type; finally, an activation function is applied to introduce nonlinear characteristics.

[0016] As another improvement of the present application, in step S4, the inter-block parallel component captures different scale features by parallelizing multiple hollow Mamba blocks and using summation or splicing fusion strategy, wherein the number of convolution operations of the hollow Mamba is 1.

[0017] The intra-block parallel component captures different scale features by serially stacking multiple hollow Mamba blocks, each hollow Mamba block contains N convolution operations, and different scale features are learned by parallelizing multiple convolution operations with different expansion rates inside the hollow Mamba block, and then the convolution outputs of different expansion rates are fused by summation; the multiple convolution operations share weights.

[0018] As a further improvement of the present application, the hollow Mamba block is defined as: let the input tensor be X ∈ R B×D×L , where B is the batch size, D is the feature dimension, L is the sequence length, X represents the output after the video backbone network processing and mapping through the embedding module;

[0019] The input X is mapped to a higher dimension through a linear layer, and then divided into four parts on the feature dimension D: X f , Z f , X b , Z b :

[0020]

[0021]

[0022] Where X b and Z b are the inputs for the reverse sequence;

[0023] The features X f and the input X b of the reverse sequence are processed by a hollow causal convolution, where X b is reversed along the feature dimension L:

[0024]

[0025] Where Conv i represents the hollow causal convolution, the dilation rate is d i , and F is the flip function for reversing the sequence;

[0026] The outputs D f and D b of the hollow causal convolution are input into the selective state space model after the activation function:

[0027] S f , S b = σ (SSM (D f , D b ))

[0028] Where σ is the SiLU activation function;

[0029] S f , S b , Z f , Z b are spliced, the reverse sequence S b and Z b are flipped back to the original order and spliced with the forward sequence, and finally the output is mapped back to the original D dimension through a linear layer:

[0030] R = Linear(Concat(S f , sigma(Z f ), F(S b ), sigma(Z b )))

[0031] where R represents the output result, and Concat represents a concatenation operation in feature dimension D.

[0032] As a further improvement of the present application, the dilated causal convolution separates the input time steps according to a dilation factor, and its output is represented as:

[0033]

[0034] where d is the dilation factor, k is the size of the convolution kernel, w[k] is the weight of the kth convolution kernel element, x(t) is the input at time t, and p is the padding factor.

[0035] As a further improvement of the present application, the padding factor p is calculated as:

[0036]

[0037] To achieve the above-mentioned purposes, the technical scheme adopted by the present application is: a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the video recognition and segmentation method based on the Mamba architecture as described above.

[0038] To achieve the above-mentioned purposes, the technical scheme adopted by the present application is: an apparatus comprising

[0039] a memory for storing instructions;

[0040] a processor for executing the instructions, so that the apparatus performs the operations of the video recognition and segmentation method based on the Mamba architecture as described above when executed.

[0041] Compared with the prior art, the present application has the beneficial effects of:

[0042] (1) The present application adopts a time action detection method based on the Mamba architecture, which can more accurately capture the changes and action features of experimental steps, significantly improving the precision and robustness of the classification model.

[0043] (2) The present application introduces dilated causal convolution and multi-scale information fusion method, effectively enhances the model's recognition ability for experimental steps of different duration, further improves the accuracy and stability of detection.

[0044] (3) The application can reduce the consumption of human resources in the artificial detection process, solve the problem that the existing method does not fully utilize the time sequence relationship when processing long time sequence videos, improve the analysis and processing efficiency, and has high application value and broad application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a step flow chart of the video recognition and segmentation method based on the Mamba architecture of the application;

[0046] Figure 2 is a comparison chart of the hollow causal convolution and the standard convolution for processing time sequence features in the method of the application. DETAILED DESCRIPTION

[0047] The application will be further illustrated in combination with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the application and not to limit the scope of the application.

[0048] Example 1

[0049] The video recognition and segmentation method based on the Mamba architecture is mainly used for segmentation and recognition of experimental steps in offline middle school experimental videos, such as Figure 1 As shown, it includes the following steps:

[0050] Step S1: Preprocessing the input middle school experimental video, referring to the common time action positioning method, slightly reducing the video resolution to reduce memory occupation and speed up the processing speed, while maintaining enough visual information for subsequent analysis.

[0051] In the preprocessing step, at least by scaling by selecting the short side of the video, the short side is adjusted to 480 pixels, while keeping the aspect ratio of the video unchanged, automatically calculating and adjusting the corresponding long side resolution to ensure the consistency of the video content proportion.

[0052] Step S2: Extracting the spatio-temporal features of the preprocessed video using a large video backbone network VideoMAE, and in the extraction process, extracting feature vectors in units of video clips to ensure efficient and fine-grained spatio-temporal feature representation.

[0053] The video is cut into video clips of 16 frames each when extracting spatio-temporal features, and the video frame rate is about 30 frames per second, with a step of 4 frames, so that a feature vector is extracted every 4 / 30 seconds (about 0.1333 seconds). The input video frame is cropped and adjusted to 224x224 resolution before feature extraction to ensure standardized input size and ensure efficient and fine-grained representation of spatio-temporal features.

[0054] Step S3: as Figure 1As shown on the left, the spatio-temporal features extracted in step S2 are input into the cavity Mamba-based temporal action detector, and the features are nonlinearly mapped by an embedding module to adapt to subsequent multi-scale information processing.

[0055] The workflow of the embedding module is as follows: first, the input spatio-temporal features are processed by a 1D convolutional layer to extract time sequence features; if the input data needs to be masked, the data is masked before and after the convolution operation, and the masking function is to shield or weight specific data positions; after the convolution calculation is completed, a normalization operation is performed, and the normalization layer will adjust the dimensions of the input data according to its type; finally, an activation function is applied to introduce nonlinear characteristics.

[0056] Step S4: The spatio-temporal features passing through the embedding module extract different scales of spatio-temporal information through intra-block parallel components or inter-block parallel components, and then fuse different scales of spatio-temporal information through a simple summation method. The intra-block parallel component and the inter-block parallel component represent two different combination methods, the former relies on the multi-convolution information fusion inside the cavity Mamba, and the latter directly fuses the output features of multiple cavity Mambas.

[0057] As shown on the left, Figure 1 As shown on the left, the two components are implemented through different combination methods, which are divided into internal multi-convolution parallel and external multi-cavity Mamba parallel. The intra-block parallel component is more suitable for detecting scenes with low overlap, providing higher recall rate and reducing the possibility of missed detection. The inter-block parallel component is more suitable for detecting complexity and provides more accurate positioning.

[0058] Inter-block parallel component: This component parallelly processes multiple cavity Mamba blocks and uses a simple fusion strategy (such as summation or concatenation) to capture features of different scales, which is called a feature fusion layer. In this process, the concatenation method needs to introduce an additional linear layer to reduce the dimension, making it complex, while direct summation can provide higher efficiency. In order to further improve the action positioning capability, the fused features are passed to a multi-layer feature pyramid. The number of convolutions of the cavity Mamba in this component is 1.

[0059] Intra-block parallel component: Unlike the inter-block parallel component, the intra-block parallel component is implemented by stacking multiple hollow Mamba blocks in series, which makes its structure more concise. Unlike the inter-block parallel component, each hollow Mamba block in the intra-block parallel component contains N convolution operations inside. This component learns diverse information by parallelizing multiple convolution operations with different dilation rates inside the hollow Mamba block, thereby enhancing the representation ability of the module. Subsequently, a simple summation method is used to effectively fuse the convolution outputs with different dilation rates, thereby achieving a better balance between the receptive fields. The multiple convolutions in this method share weights, so the parameter amount is not increased while the performance is improved.

[0060] As Figure 1 The core module of the time action detector in step S4 is the hollow Mamba, as shown on the right. The formal definition of the hollow Mamba is as follows: Let the input tensor be X ∈ R B×D×L , where B is the batch size, D is the feature dimension, and L is the sequence length. X represents the output after processing by the video backbone network and mapping by the embedding module.

[0061] First, the input X is mapped to a higher dimension by a linear layer, and then divided into four parts in the feature dimension D: X f , Z f , X b , and Z b , as shown below:

[0062]

[0063]

[0064] where X b and Z b are the inputs for the reversed sequence.

[0065] Next, the features X f and the input X b of the reversed sequence are processed by the hollow causal convolution, where X b is reversed along the feature dimension L. This operation aims to capture the temporal features from the forward and reversed sequences, and is represented as follows:

[0066]

[0067] where Conv i represents the hollow causal convolution with dilation rate d i , and F is the flip function for the reversed sequence. Each convolution operation has its own dilation rate d i , allowing the model to capture varying temporal dependencies at different time scales.

[0068] The output D of dilated causal convolution f and D b After being activated by a function to enhance nonlinearity, the input is fed into a selective state-space model (SSM). The selective state-space model follows the original SSM in Mamba. This process can be represented as:

[0069] S f ,S b =σ(SSM(D) f D b ))

[0070] Where σ is the SiLU activation function.

[0071] Finally, S f S b Z f Z b The results (after activation function) are concatenated. The reverse sequence S b and Z b The sequence will be flipped back to its original order and concatenated with the forward sequence. The final output is then mapped back to the original d-dimensional sequence through a linear layer. This process can be represented as:

[0072] R = Linear(Concat(S) f ,σ(Z f ),F(S b ),σ(Z b )))

[0073] Where R represents our output, and Concat represents the concatenation operation on the feature dimension D.

[0074] like Figure 2 As shown, the difference between dilated causal convolution in dilated Mamba and standard convolution in step S4 is that dilated causal convolution separates the input time steps according to the dilation factor, allowing the model to cover a wider time range while maintaining causality. Formally, the output of dilated causal convolution can be expressed as:

[0075]

[0076] Where d is the dilation factor, k is the kernel size, w[k] is the weight of the k-th kernel element, and x(t) is the input at time t. The constant p is called the fill factor, and its calculation formula is:

[0077]

[0078] The padding factor p is used to pad the input x before the convolution operation, so as to ensure the alignment correctness in the convolution operation. In contrast, the standard causal convolution processes continuous time steps, so its receptive field is significantly smaller. This feature makes the dilated causal convolution particularly suitable for processing long-lasting actions.

[0079] Step S5: The fused multi-scale features are down-sampled in a feature pyramid manner, and the classification head and the regression head are used to output the category and the time segmentation result of each experimental step respectively, so as to realize accurate identification and segmentation of the experimental steps.

[0080] In the embodiment, the pre-processing step follows the common time action positioning method, and the Adam method is also used for stochastic gradient descent, the batch size is configured as 2, the learning rate is set as 0.0002, and the weight decay is set as 0.05. In the embodiment, end-to-end training can be directly performed, or a separate method can be used to extract features and then train the time action detector.

[0081] Test example

[0082] The method of the present application is applied to a carbon dioxide experimental data set for experiments, which contains 222 groups of videos of limestone preparation of carbon dioxide and 11 steps. Due to the complexity of the experimental content, and the duration of each action is not the same, which poses a challenge to the accuracy and sensitivity of identification and segmentation. In order to measure the gap between the segmentation result and the actual result, the average precision (mAP) is used as the main evaluation index for evaluating the experimental task in the test example. mAP is a comprehensive evaluation method, which measures the overall performance of the model in action classification and positioning by weighted average of the average precision (AP) of each action category. The higher the mAP value, the better the model performs in identifying action categories and accurately positioning action time periods.

[0083] In the test example, the method of the present application is compared with the most popular time action detector. The main comparison indicators include: the average mAP under the condition of freezing the video backbone network, the parameter amount of the time action detector, and the average training time of the network. The comparison results are shown in Table 1:

[0084] Table 1

[0085] Method Average mAP Parameter quantity Average training time Actionformer 76.8 27.6M 10.5S The invention 79.3 19.8M 7.5S

[0086] As can be seen from the above table, by comparing the average mAP, the application has more advantages in action recognition accuracy; at the same time, by comparing the parameter amount and training time of the model, the advantages of the application in training efficiency and inference efficiency are highlighted. From the table, it can also be seen that the detection accuracy of the application on the carbon dioxide dataset is better than the popular method, and significant optimization can be achieved in parameter efficiency and training time, which is more suitable for practical application.

[0087] In summary, the video recognition and segmentation method based on the Mamba architecture disclosed in the application uses an efficient video backbone network to extract the visual spatiotemporal features of the video, and inputs the extracted features into an improved temporal action detector, thereby significantly improving the recognition ability of short-time fast-changing steps and long-time stable steps through effective fusion of action features of different duration, and finally realizing accurate classification and time segmentation of each experimental step in the experimental video. The method of the application can significantly improve the recognition accuracy and robustness of experimental steps when processing complex experimental video data, effectively solve the problems of large consumption of human resources, unstable detection effect and low efficiency, and greatly improve the automation level and evaluation accuracy of experimental teaching. This method has high application value and wide application prospect, and is more practical in the field of middle school experimental teaching and education evaluation.

[0088] In the description of the present specification, the description referring to the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0089] It should be noted that the above content only illustrates the technical idea of the application, and cannot limit the protection scope of the application. For ordinary skilled persons in the art, without departing from the principles of the application, a number of improvements and refinements can be made, which fall within the protection scope of the claims of the application.

Claims

1. A video recognition and segmentation method based on the Mamba architecture, characterized in that, Includes the following steps: S1. Data preprocessing: Preprocess the input experimental video to be identified and segmented to reduce the video resolution; S2. Feature extraction: Spatiotemporal features are extracted from the video preprocessed in step S1 using the large-scale video backbone network VideoMAE. S3. Nonlinear mapping: Input the spatiotemporal features extracted in step S2 into the time action detector based on holed Mamba, and perform nonlinear mapping on the features through the embedding module; S4. Spatiotemporal Information Fusion: After the spatiotemporal features of the embedding module in step S3 are obtained, spatiotemporal information at different scales is extracted through intra-block parallel components or inter-block parallel components, and spatiotemporal information at different scales is fused by summation. The intra-block parallel components are based on the fusion of multiple convolutional information within dilated Mamba, and the inter-block parallel components directly fuse the output features of multiple dilated Mamba. The inter-block parallel components capture features at different scales by parallelizing multiple dilated Mamba blocks and using a fusion strategy of summation or concatenation. The number of convolutions in dilated Mamba is 1. The intra-block parallel component captures features at different scales by serially stacking multiple dilated Mamba blocks. Each dilated Mamba block contains N convolutional operations. Diverse information is learned by parallelizing multiple convolutional operations with different dilation rates within the dilated Mamba block. Then, the convolutional outputs with different dilation rates are summed and fused. The multiple convolutions share weights. A holed Mamba block is defined as follows: Let the input tensor be... ,in It refers to the batch size. It is the feature dimension. It is the sequence length. This represents the output after processing by the video backbone network and mapping through the embedding module. enter Mapped to a higher dimension through a linear layer, and then along the feature dimension. It is divided into four parts: , , , : ； ； in, and It is the input used for reverse sequences; feature and the input of the reverse sequence Processed using dilated causal convolution, where Along the feature dimension Reverse: ； in, This represents dilated causal convolution with an inflation rate of 1:

1. , It is a flip function used to reverse sequences. ; Output of dilated causal convolution and After passing through the activation function, the input is fed into the selective state-space model: ； in, It is the SiLU activation function; Will , , , Perform concatenation, reverse sequence and The sequence will be flipped back to its original order and concatenated with the forward sequence. The final output will be mapped back to the original sequence through a linear layer. Dimensions: ； in, Indicates the output result. Indicating in the feature dimension splicing operations on; S5. Identification and Segmentation: The multi-scale features fused in step S4 are downsampled using a feature pyramid. The classification head and regression head are used to output the category and time segmentation results for each experimental step, thereby realizing the identification and segmentation of the experimental steps.

2. The video recognition and segmentation method based on the Mamba architecture as described in claim 1, characterized in that: The video resolution reduction in step S1 of the data preprocessing involves adjusting the shorter side of the video to 480 pixels while maintaining the aspect ratio of the video. The corresponding longer side resolution is automatically calculated and adjusted to ensure that the video proportions are consistent.

3. The video recognition and segmentation method based on the Mamba architecture as described in claim 1, characterized in that: In step S2, before feature extraction, the input video frames are cropped to a resolution of 224×224; during feature extraction, the video is divided into video segments of 16 frames each, with a video frame rate of 30 frames per second and a stride of 4 frames, so that a feature vector is extracted every 4 / 30 seconds.

4. The video recognition and segmentation method based on the Mamba architecture as described in claim 1, characterized in that: In the embedding module of step S3, the input spatiotemporal features are used to extract temporal features through a 1D convolutional layer; After the convolution calculation is completed, a normalization operation is performed, and finally, an activation function is applied to introduce nonlinear characteristics.

5. The video recognition and segmentation method based on the Mamba architecture as described in claim 4, characterized in that: The dilated causal convolution separates the input time steps according to the dilation factor, and its output is represented as: ； in, It is the expansion factor. It is the size of the convolution kernel. It is the first The weights of each convolutional kernel element, It is time Input, It is the fill factor.

6. The video recognition and segmentation method based on the Mamba architecture as described in claim 5, characterized in that: The fill factor The calculation method is as follows: 。 7. A computer-readable storage medium, characterized in that: It stores a computer program that, when executed by a processor, implements the video recognition and segmentation method based on the Mamba architecture as described in any one of claims 1-6.

8. A device, characterized in that: include Memory, used to store instructions; A processor for executing the instructions, causing the device to perform the operation of the video recognition and segmentation method based on the Mamba architecture as described in any one of claims 1-6.

Images