A Multi-View Video Action Recognition Method Based on Adaptive Modeling

Abstract

A multi-view video action recognition method based on adaptive modeling, belonging to the field of video recognition technology, is proposed. This method utilizes a pre-trained 3D convolutional visual model to extract temporal features from multi-view videos, constructs cross-view shared prior knowledge representations, and generates prototype weights through a prototype routing mechanism. These weighted combinations of a learnable prototype knowledge base yield a knowledge-guided prototype hybrid representation. Furthermore, a prototype-guided view-adaptive expert enhancement module is constructed to perform differentiated enhancement modeling on features from different views, achieving cross-view common information modeling and view difference information expression. Finally, the pre-trained 3D convolutional visual model and subsequent enhancement modules are jointly fine-tuned in conjunction with downstream action recognition tasks, thereby improving the accuracy and robustness of multi-view video action recognition.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and more specifically to a multi-view video action recognition method based on adaptive modeling. Background Technology

[0002] With the development of video sensing devices and intelligent analysis technologies, video-based human motion recognition is finding increasingly widespread applications in intelligent monitoring, behavior analysis, sports training, medical rehabilitation, and human-computer interaction. Compared to image recognition tasks, video motion recognition not only needs to depict human appearance information but also needs to model the dynamic changes of actions over time, thus placing higher demands on feature representation capabilities and spatiotemporal modeling capabilities.

[0003] In practical applications, human movements are often captured simultaneously by multiple cameras from different angles, forming multi-view video data. Compared to single-view videos, multi-view videos provide richer action cues, helping to mitigate information loss caused by factors such as occlusion, changing viewpoints, and background interference. Therefore, multi-view video action recognition has become an important research direction in the field of video understanding.

[0004] However, multi-view video action recognition still faces significant modeling challenges. On the one hand, videos from different viewpoints typically exhibit significant differences in appearance, motion trajectory, and visible area, leading to inconsistent representations of the same action across different viewpoints. On the other hand, while different viewpoints share action semantics, extracting a unified and discriminative shared representation from the differentiated viewpoint information, while avoiding the loss of viewpoint-specific complementary cues, remains a challenge for current methods. Most existing methods achieve recognition through multi-view feature extraction and fusion, but some methods provide a coarse characterization of shared and differentiated information from multiple viewpoints, making it difficult to fully utilize the complementary relationships between different viewpoints. Furthermore, some methods overemphasize viewpoint-independent representations, which can weaken viewpoint-specific cues relevant to action discrimination, thus affecting recognition performance.

[0005] Furthermore, in real-world multi-camera acquisition environments, the amount of effective information in videos from different perspectives is often uneven. Some perspectives may clearly reflect the overall human movement process, while others may provide only weak or even noisy visual cues due to factors such as occlusion, limited shooting angle, cluttered background, or changes in target scale. If the model lacks an adaptive processing mechanism for perspective differences during the modeling process, it is easy to introduce redundant information from lower-quality perspectives along with effective information from high-value perspectives, thereby affecting the overall recognition performance.

[0006] Therefore, existing multi-view video action recognition technologies still have shortcomings in areas such as shared action semantic extraction, utilization of viewpoint difference information, and adaptive enhancement of features from different viewpoints, and need further improvement. Summary of the Invention

[0007] In order to overcome the shortcomings of the above technologies, this invention provides a method to improve the accuracy of multi-view video action recognition.

[0008] The technical solution adopted by this invention to overcome its technical problems is: A multi-view video action recognition method based on adaptive modeling includes: S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , ; S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. ; S3. Transfer the preprocessed video clips The input is fed into a shared pre-trained 3D convolutional vision model, and the output is the first... Temporal feature sequences corresponding to each perspective Using time-series feature sequences Get the first View-level feature representation for each viewpoint ; S4. Utilize all View-level feature representation for each viewpoint Get the first A multi-perspective example Shared prior knowledge representation ; S5. The first A multi-perspective example Shared prior knowledge representation Inputting it into the prototype routing module yields the result related to the first... A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation ; S6. The first View-level feature representation for each viewpoint , No. A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation , No. Temporal feature sequences corresponding to each perspective The input is fed into the viewpoint adaptive expert enhancement module to obtain the enhanced viewpoint features. ; S7. Enhanced viewpoint features Input into the shared classifier to obtain the first... A multi-perspective example The Middle The prediction score for each action category corresponding to each viewpoint.

[0009] Furthermore, the dataset in step S1 is either the NTU RGB+D 120 dataset or the PKU-MMD dataset.

[0010] Furthermore, the annotation information corresponding to the video segments in step S1 includes configuration identifiers, camera identifiers, performer identifiers, repetition count identifiers, and action category identifiers, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

[0011] Furthermore, step S2 includes the following steps: S2-1. Use the linspace() function from the NumPy library to process the first... A multi-perspective example The Middle Video clips from various perspectives Perform equal-interval sampling to extract 16 frames to generate a fixed-length video clip. ; S2-2. Use the cv2.cvtColor() function in the OpenCV library to convert a fixed-length video segment. Each frame of the image is converted from the BGR color space to the RGB color space to obtain a video clip. ; S2-3. Use the Resize() function in torchvision.transforms to resize the video clip. Adjust the resolution to 270×480 pixels to obtain the video clip. ; S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. ; S2-5. Use the transforms.Normalize() function from the PyTorchVideo library to transform video clips. Each channel is standardized to obtain preprocessed video segments. .

[0012] Furthermore, step S3 includes the following steps: S3-1. The shared pre-trained 3D convolutional visual model consists of an R3D-18 model, which processes pre-processed video segments. Inputting into the R3D-18 model, the output is the first... Temporal feature sequences corresponding to each perspective ; S3-2. Using the mean() function in the PyTorch library to analyze time-series feature sequences along the time dimension. Perform average polymerization to obtain the first View-level feature representation for each viewpoint .

[0013] Furthermore, step S4 includes the following steps: S4-1. Use the mean() function from the PyTorch library to perform a cross-section along the view dimension. A multi-perspective example of The i-th viewpoint-level feature representations are averaged and aggregated to obtain the i-th viewpoint-level feature representation. A multi-perspective example Cross-perspective aggregation features ; S4-2. Construct a shared prior knowledge encoding module, which consists of a first linear mapping layer, a GELU activation function, a Dropout layer, and a second linear mapping layer. The shared prior knowledge encoding module is constructed by sequentially adding a first linear mapping layer, a GELU activation function, a Dropout layer, and a second linear mapping layer. A multi-perspective example Cross-perspective aggregation features The input is fed into the shared prior knowledge encoding module, and the output is obtained. A multi-perspective example Shared prior knowledge representation .

[0014] Furthermore, step S5 includes the following steps: S5-1. Construct a prototype routing module consisting of a prototype weighted routing unit, a learnable prototype knowledge base unit, a normalization processing unit, and a prototype weighted combination unit; S5-2. The prototype weighted routing unit of the prototype routing module consists of a first linear mapping layer, a GELU activation function, a second linear mapping layer, and a Softmax function, in sequence. The output dimension of the second linear mapping layer is... , will the A multi-perspective example Shared prior knowledge representation The input is fed into the prototype weight routing unit, and the output is the prototype weight vector. , ,in, For the first A multi-perspective example The The weight of each prototype, ; S5-3. Learnable prototype knowledge base units generated using Gaussian random initialization. The learnable prototype vector, the first The learnable prototype vectors are , The feature dimension of each learnable prototype vector is the same as the first... A multi-perspective example Shared prior knowledge representation The feature dimensions are consistent, all A learnable prototype vector constitutes a learnable prototype knowledge base. , ; S5-4. In the normalization processing unit, the first... A learnable prototype vector Normalization is performed to obtain the normalized learnable prototype vector. The normalized learnable prototype knowledge base is , ; S5-5. In the prototype weighted combination unit, through the formula... Calculated and obtained with the first A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation .

[0015] Furthermore, step S6 includes the following steps: S6-1. Construct a view-adaptive expert enhancement module consisting of view gating units, isomorphic expert units, weighted fusion units, and residual enhancement units; S6-2. Hybrid representation of knowledge-guided prototypes After expanding along the visual dimension, and with the first View-level feature representation for each viewpoint By concatenating the data, the gated input features are obtained. ; S6-3. The viewpoint gating unit of the viewpoint adaptive expert enhancement module consists of a first linear mapping layer, a GELU activation function, a second linear mapping layer, and a Softmax function. The output dimension of the second linear mapping layer is... Gating input features The input is fed into the view gating unit, and the output is the expert weight vector. , ,in, For the first A multi-perspective example The The first perspective Each expert's weight, ; S6-4. The isomorphic expert unit of the viewpoint adaptive expert enhancement module is composed of a first one-dimensional convolutional layer, a first GroupNorm layer, a first GELU activation function, a second one-dimensional convolutional layer, a second GroupNorm layer, a second GELU activation function, and a Dropout layer. Temporal feature sequences corresponding to each perspective Enter them separately In a group of isomorphic expert units, we obtain Expert output characteristics ,in, For the first A multi-perspective example The The characteristics of expert output from multiple perspectives ; S6-5. The first A multi-perspective example The Expert output characteristics from multiple perspectives With the A multi-perspective example The The first perspective Expert weights The input is fed into the weighted fusion unit of the viewpoint adaptive expert enhancement module, and processed by formula. Calculation of fusion features ; S6-6. Merging Features With the View-level feature representation for each viewpoint The residuals are added in the residual enhancement unit of the viewpoint adaptive expert enhancement module to obtain the enhanced viewpoint features. .

[0016] Furthermore, the shared classifier in step S7 is a fully connected layer.

[0017] The beneficial effects of this invention are as follows: By aggregating the temporal features of multi-view videos across different perspectives, a shared prior knowledge representation is constructed. Furthermore, a prototype routing mechanism is used to weight and combine the learnable prototype knowledge base to generate a knowledge-guided prototype hybrid representation corresponding to the sample. This is further combined with a prototype-guided perspective-adaptive expert enhancement module to achieve differentiated enhancement processing for features from different perspectives. Compared to existing multi-view action recognition methods, this invention not only extracts shared action semantic information between different perspectives but also retains effective discrimination cues specific to each perspective, thus balancing cross-view common semantic modeling with perspective difference expression. Simultaneously, this invention can adaptively enhance features based on different perspectives, suppressing the interference of unfavorable perspective information on the recognition results and improving the accuracy and robustness of multi-view video action recognition in complex scenes. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] The following is in conjunction with the appendix Figure 1 The present invention will be further described below.

[0020] A multi-view video action recognition method based on adaptive modeling includes: S1. Obtain from the dataset Each video segment and its corresponding annotation information will be... Each video clip is organized into a multi-view instance set according to the camera ID. , , For the first A multi-perspective example , For the number of multi-view instances, , For the first A multi-perspective example The Middle Video clips from various perspectives , .

[0021] S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. .

[0022] S3. Transfer the preprocessed video clips The input is fed into a shared pre-trained 3D convolutional vision model, and the output is the first... Temporal feature sequences corresponding to each perspective Using time-series feature sequences Get the first View-level feature representation for each viewpoint .

[0023] S4. Utilize all View-level feature representation for each viewpoint Get the first A multi-perspective example Shared prior knowledge representation .

[0024] S5. The first A multi-perspective example Shared prior knowledge representation Inputting it into the prototype routing module yields the result related to the first... A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation .

[0025] S6. The first View-level feature representation for each viewpoint , No. A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation , No. Temporal feature sequences corresponding to each perspective The input is fed into the viewpoint adaptive expert enhancement module to obtain the enhanced viewpoint features. .

[0026] S7. Enhanced viewpoint features Input into the shared classifier to obtain the first... A multi-perspective example The Middle The prediction score for each action category corresponding to each viewpoint.

[0027] This paper utilizes a pre-trained 3D convolutional vision model to extract temporal features from multi-view videos, constructs a cross-view shared prior knowledge representation, and generates prototype weights through a prototype routing mechanism. These weighted weights are then combined with a learnable prototype knowledge base to obtain a knowledge-guided prototype hybrid representation. Furthermore, a prototype-guided view-adaptive expert enhancement module is constructed to perform differentiated enhancement modeling of features from different viewpoints. This achieves cross-view common information modeling and viewpoint difference information expression, thereby improving the accuracy and robustness of multi-view video action recognition.

[0028] In one embodiment of the present invention, the dataset in step S1 is an NTU RGB+D 120 dataset or a PKU-MMD dataset.

[0029] In one embodiment of the present invention, the annotation information corresponding to the video segment in step S1 includes a configuration identifier, a camera identifier, a performer identifier, a repetition count identifier, and an action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

[0030] In one embodiment of the present invention, step S2 includes the following steps: S2-1. Use the linspace() function from the NumPy library to process the first... A multi-perspective example The Middle Video clips from various perspectives Perform equal-interval sampling to extract 16 frames to generate a fixed-length video clip. .

[0031] S2-2. Use the cv2.cvtColor() function in the OpenCV library to convert a fixed-length video segment. Each frame of the image is converted from the BGR color space to the RGB color space to obtain a video clip. .

[0032] S2-3. Use the Resize() function in torchvision.transforms to resize the video clip. Adjust the resolution to 270×480 pixels to obtain the video clip. .

[0033] S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. .

[0034] S2-5. Use the transforms.Normalize() function from the PyTorchVideo library to transform video clips. Each channel is standardized to obtain preprocessed video segments. .

[0035] In one embodiment of the present invention, step S3 includes the following steps: S3-1. The shared pre-trained 3D convolutional visual model consists of an R3D-18 model, which processes pre-processed video segments. Inputting into the R3D-18 model, the output is the first... Temporal feature sequences corresponding to each perspective .

[0036] S3-2. Using the mean() function in the PyTorch library to analyze time-series feature sequences along the time dimension. Perform average polymerization to obtain the first View-level feature representation for each viewpoint .

[0037] In one embodiment of the present invention, step S4 includes the following steps: S4-1. Use the mean() function from the PyTorch library to perform a cross-section along the view dimension. A multi-perspective example of The i-th viewpoint-level feature representations are averaged and aggregated to obtain the i-th viewpoint-level feature representation. A multi-perspective example Cross-perspective aggregation features .

[0038] S4-2. Construct a shared prior knowledge encoding module, which consists of a first linear mapping layer, a GELU activation function, a Dropout layer, and a second linear mapping layer. The shared prior knowledge encoding module is constructed by sequentially adding a first linear mapping layer, a GELU activation function, a Dropout layer, and a second linear mapping layer. A multi-perspective example Cross-perspective aggregation features The input is fed into the shared prior knowledge encoding module for nonlinear mapping, and the output is obtained. A multi-perspective example Shared prior knowledge representation .

[0039] In one embodiment of the present invention, step S5 includes the following steps: S5-1. Construct a prototype routing module consisting of a prototype weighted routing unit, a learnable prototype knowledge base unit, a normalization processing unit, and a prototype weighted combination unit.

[0040] S5-2. The prototype weighted routing unit of the prototype routing module consists of a first linear mapping layer, a GELU activation function, a second linear mapping layer, and a Softmax function, in sequence. The output dimension of the second linear mapping layer is... , will the A multi-perspective example Shared prior knowledge representation The input is fed into the prototype weight routing unit, and the output is the prototype weight vector. , ,in, For the first A multi-perspective example The The weight of each prototype, .

[0041] S5-3. Learnable prototype knowledge base units generated using Gaussian random initialization. The learnable prototype vector, the first The learnable prototype vectors are , The feature dimension of each learnable prototype vector is the same as the first... A multi-perspective example Shared prior knowledge representation The feature dimensions are consistent, all A learnable prototype vector constitutes a learnable prototype knowledge base. , .

[0042] S5-4. In the normalization processing unit, the first... A learnable prototype vector Normalization is performed to obtain the normalized learnable prototype vector. The normalized learnable prototype knowledge base is , .

[0043] S5-5. In the prototype weighted combination unit, through the formula... Calculated and obtained with the first A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation .

[0044] In one embodiment of the present invention, step S6 includes the following steps: S6-1. Construct a view-adaptive expert enhancement module consisting of view gating units, isomorphic expert units, weighted fusion units, and residual enhancement units.

[0045] S6-2. Hybrid representation of knowledge-guided prototypes After expanding along the visual dimension, and with the first View-level feature representation for each viewpoint By concatenating the data, the gated input features are obtained. .

[0046] S6-3. The viewpoint gating unit of the viewpoint adaptive expert enhancement module consists of a first linear mapping layer, a GELU activation function, a second linear mapping layer, and a Softmax function. The output dimension of the second linear mapping layer is... Gating input features The input is fed into the view gating unit, and the output is the expert weight vector. , ,in, For the first A multi-perspective example The The first perspective Each expert's weight, .

[0047] S6-4. The isomorphic expert unit of the viewpoint adaptive expert enhancement module is composed of a first one-dimensional convolutional layer, a first GroupNorm layer, a first GELU activation function, a second one-dimensional convolutional layer, a second GroupNorm layer, a second GELU activation function, and a Dropout layer. Temporal feature sequences corresponding to each perspective Enter them separately In a group of isomorphic expert units, we obtain Expert output characteristics ,in, For the first A multi-perspective example The The characteristics of expert output from multiple perspectives .

[0048] S6-5. The first A multi-perspective example The Expert output characteristics from multiple perspectives With the A multi-perspective example The The first perspective Expert weights The input is fed into the weighted fusion unit of the viewpoint adaptive expert enhancement module, and processed by formula. Calculation of fusion features .

[0049] S6-6. Merging Features With the View-level feature representation for each viewpoint The residuals are added in the residual enhancement unit of the viewpoint adaptive expert enhancement module to obtain the enhanced viewpoint features. .

[0050] In one embodiment of the present invention, the shared classifier in step S7 is a fully connected layer.

[0051] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-view video action recognition method based on adaptive modeling, characterized in that, include: S1. Obtain a plurality of video clips and corresponding label information of each video clip from a dataset, ;​​​​​​​​​​​​​​ S2. The first A multi-perspective example The Middle Video clips from various perspectives Preprocessing is performed to obtain preprocessed video clips. ; S3. Transfer the preprocessed video clips The input is fed into a shared pre-trained 3D convolutional vision model, and the output is the first... Temporal feature sequences corresponding to each perspective Using time-series feature sequences Get the first View-level feature representation for each viewpoint ; S4. Utilize all View-level feature representation for each viewpoint Get the first A multi-perspective example Shared prior knowledge representation ; S5. The first A multi-perspective example Shared prior knowledge representation Inputting it into the prototype routing module yields the result related to the first... A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation ; S6. The first View-level feature representation for each viewpoint , No. A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation , No. Temporal feature sequences corresponding to each perspective The input is fed into the viewpoint adaptive expert enhancement module to obtain the enhanced viewpoint features. ; S7. Enhanced viewpoint features The input is fed into the shared classifier to obtain the first... A multi-perspective example The Middle The prediction score for each action category corresponding to each viewpoint.

2. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that: The dataset in step S1 is either the NTU RGB+D 120 dataset or the PKU-MMD dataset.

3. The multi-view video action recognition method based on adaptive modeling according to claim 2, characterized in that: The annotation information corresponding to the video segment in step S1 includes configuration identifier, camera identifier, performer identifier, repetition count identifier, and action category identifier, which will be obtained from the dataset. Based on the annotation information, the video clips were captured using the same capture configuration, the same performer, the same number of repetitions, the same action category, and were filmed by different cameras. The video clip was inserted into the first... A multi-perspective example middle.

4. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that, Step S2 includes the following steps: S2-1. Use the linspace() function from the NumPy library to process the first... A multi-perspective example The Middle Video clips from various perspectives Perform equal-interval sampling to extract 16 frames to generate a fixed-length video clip. ; S2-2. Use the cv2.cvtColor() function in the OpenCV library to convert a fixed-length video segment. Each frame of the image is converted from the BGR color space to the RGB color space to obtain a video clip. ; S2-3. Use the Resize() function in torchvision.transforms to resize the video clip. Adjust the resolution to 270×480 pixels to obtain the video clip. ; S2-4. Use the RandomCrop() function in the PyTorch library to crop the video clip. Perform a random cropping operation to obtain a video clip with a resolution of 224×224 pixels. ; S2-5. Use the transforms.Normalize() function from the PyTorchVideo library to transform video clips. Each channel is standardized to obtain preprocessed video segments. .

5. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that, Step S3 includes the following steps: S3-1. The shared pre-trained 3D convolutional visual model consists of an R3D-18 model, which processes pre-processed video segments. Inputting into the R3D-18 model, the output is the first... Temporal feature sequences corresponding to each perspective ; S3-2. Using the mean() function in the PyTorch library to analyze time-series feature sequences along the time dimension. Perform average polymerization to obtain the first View-level feature representation for each viewpoint .

6. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that, Step S4 includes the following steps: S4-1. Use the mean() function from the PyTorch library to perform a cross-section along the view dimension. A multi-perspective example of The i-th viewpoint-level feature representations are averaged and aggregated to obtain the i-th viewpoint-level feature representation. A multi-perspective example Cross-perspective aggregation features ; S4-2. Construct a shared prior knowledge encoding module, which consists of a first linear mapping layer, a GELU activation function, a Dropout layer, and a second linear mapping layer. The shared prior knowledge encoding module is constructed by sequentially adding a first linear mapping layer, a GELU activation function, a Dropout layer, and a second linear mapping layer. A multi-perspective example Cross-perspective aggregation features The input is fed into the shared prior knowledge encoding module, and the output is obtained. A multi-perspective example Shared prior knowledge representation .

7. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that, Step S5 includes the following steps: S5-1. Construct a prototype routing module consisting of a prototype weighted routing unit, a learnable prototype knowledge base unit, a normalization processing unit, and a prototype weighted combination unit; S5-2. The prototype weighted routing unit of the prototype routing module consists of a first linear mapping layer, a GELU activation function, a second linear mapping layer, and a Softmax function, in sequence. The output dimension of the second linear mapping layer is... , will the A multi-perspective example Shared prior knowledge representation The input is fed into the prototype weight routing unit, and the output is the prototype weight vector. , ,in, For the first A multi-perspective example The The weight of each prototype, ; S5-3. Learnable prototype knowledge base units generated using Gaussian random initialization. The learnable prototype vector, the first The learnable prototype vectors are , The feature dimension of each learnable prototype vector is the same as the first... A multi-perspective example Shared prior knowledge representation The feature dimensions are consistent, all A learnable prototype vector constitutes a learnable prototype knowledge base. , ; S5-4. In the normalization processing unit, the first... A learnable prototype vector Normalization is performed to obtain the normalized learnable prototype vector. The normalized learnable prototype knowledge base is , ; S5-5. In the prototype weighted combination unit, through the formula... Calculated and obtained with the first A multi-perspective example Shared prior knowledge representation Corresponding knowledge-guided prototype hybrid representation .

8. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that, Step S6 includes the following steps: S6-1. Construct a view-adaptive expert enhancement module consisting of view gating units, isomorphic expert units, weighted fusion units, and residual enhancement units; S6-2. Hybrid representation of knowledge-guided prototypes After expanding along the visual dimension, and with the first View-level feature representation for each viewpoint By concatenating the data, the gated input features are obtained. ; S6-3. The viewpoint gating unit of the viewpoint adaptive expert enhancement module consists of a first linear mapping layer, a GELU activation function, a second linear mapping layer, and a Softmax function. The output dimension of the second linear mapping layer is... Gating input features The input is fed into the view gating unit, and the output is the expert weight vector. , ,in, For the first A multi-perspective example The The first perspective Each expert's weight, ; S6-4. The isomorphic expert unit of the viewpoint adaptive expert enhancement module is composed of a first one-dimensional convolutional layer, a first GroupNorm layer, a first GELU activation function, a second one-dimensional convolutional layer, a second GroupNorm layer, a second GELU activation function, and a Dropout layer. Temporal feature sequences corresponding to each perspective Enter them separately In a group of isomorphic expert units, we obtain Expert output characteristics ,in, For the first A multi-perspective example The The characteristics of expert output from multiple perspectives ; S6-5. The first A multi-perspective example The Expert output characteristics from multiple perspectives With the A multi-perspective example The The first perspective Expert weights The input is fed into the weighted fusion unit of the viewpoint adaptive expert enhancement module, and processed by formula. Calculation of fusion features ; S6-6. Merging Features With the View-level feature representation for each viewpoint The residuals are added in the residual enhancement unit of the viewpoint adaptive expert enhancement module to obtain the enhanced viewpoint features. .

9. The multi-view video action recognition method based on adaptive modeling according to claim 1, characterized in that: The shared classifier in step S7 is a fully connected layer.

Images