Multi-view action recognition-oriented cross-view fusion method based on RGB (Red, Green, Blue) mode

Abstract

The invention discloses a multi-view action recognition-oriented cross-view fusion method based on an RGB (Red, Green and Blue) mode. The method comprises the following steps: collecting a multi-view video; and inputting the multi-view video into a trained cross-view fusion network to obtain an action recognition result. The cross-view fusion network comprises a shallow feature extraction module, a time feature extraction module, a space feature extraction module, a time-space cross attention mechanism module and a multi-view feature fusion module, and the shallow feature extraction module is used for extracting shallow features from a multi-view video; the time feature extraction module is used for obtaining time features in the superficial layer features, the spatial feature extraction module is used for obtaining spatial features in the superficial layer features, and the time-space cross attention mechanism module is used for carrying out cross-view time-space fusion to obtain global features. And the multi-view fusion feature module is used for predicting an action category in the video based on the global feature. According to the invention, the accuracy and robustness of action recognition are improved.

Description

Technical Field

[0001] The present invention relates to electronic information technology, and more specifically, to a cross-view fusion method based on RGB modality for multi-view action recognition. Background Art

[0002] With the popularity of multi-camera monitoring systems in real life, multi-view action recognition has gradually become a research focus in the field of computer vision. Multi-view action recognition technology recognizes and understands human actions and behaviors by analyzing image information from different cameras. This technology is widely used in security monitoring, intelligent transportation, and behavior analysis.

[0003] In multi-view action recognition, traditional skeleton data methods are limited in practical applications because they rely on the hardware support of depth sensors. Such methods usually recognize actions through depth information or three-dimensional position information of the human skeleton. For example, a dedicated depth camera is used to capture the skeleton structure of a person. However, the hardware used is often expensive and difficult to promote in large-scale monitoring applications. In contrast, methods based on RGB video data are more practical and economical. Such methods do not rely on depth information and are only based on two-dimensional images captured by traditional cameras. Action recognition is achieved by analyzing RGB videos of different views, but the complementarity between multiple views is not considered.

[0004] At present, multi-view action recognition methods mainly include two categories: end-to-end methods based on deep learning and methods based on feature extraction and fusion. In deep learning methods, architectures based on 3D convolutional neural networks and Transformers are widely used to extract spatiotemporal features. For example, the I3d model can effectively extract spatiotemporal features by extending two-dimensional convolutions to three-dimensional convolutions in the time dimension. For another example, a self-supervised view generator is introduced to generate potential viewpoint representations, so that the video encoder can adapt to unseen views. Or by passing information about specific representations of different views, the classification scores of each view are combined using a view classifier. However, none of these methods fully exploits the complementarity between multiple views.

[0005] In summary, existing multi-view action recognition methods often rely on skeleton data or depth sensors, which limits their flexibility and applicability in practical applications that do not have these hardware conditions. At the same time, in actual operations, existing technologies fail to effectively and fully exploit the complementarity between multiple views, resulting in insufficient robustness of feature extraction. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a cross-view fusion method based on RGB modality for multi-view action recognition. The method comprises the following steps:

[0007] Collect multi-view videos for the target;

[0008] Inputting the multi-view video into a trained cross-view fusion network to obtain an action recognition result;

[0009] Among them, the cross-view fusion network includes a shallow feature extraction module, a temporal feature extraction module, a spatial feature extraction module, a spatiotemporal cross-attention mechanism module and a multi-view feature fusion module. The shallow feature extraction module is used to extract shallow features from multi-view videos, the temporal feature extraction module is used to obtain the temporal flow features in the shallow features, the spatial feature extraction module is used to obtain the spatial flow features in the shallow features, the spatiotemporal cross-attention mechanism module is used to perform spatiotemporal fusion across views to obtain global features, and the multi-view fusion feature module is used to predict the action category in the video based on the global features.

[0010] Compared with the prior art, the advantage of the present invention is that, unlike the traditional method that relies on skeleton data or depth information, the present invention makes full use of the characteristics of RGB video data and proposes a multi-view fusion network that can effectively extract key spatial and temporal features from multi-view video data without relying on dedicated hardware, significantly improving the efficiency, accuracy and robustness of action recognition. Experimental results show that the present invention has achieved excellent performance on multiple multi-view action datasets, demonstrating its practicality and effectiveness in complex multi-view scenes.

[0011] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 is a flowchart of a cross-view fusion method based on RGB modality for multi-view action recognition according to an embodiment of the present invention;

[0014] Figure 2 is an architecture diagram of a cross-view fusion network based on RGB modality for multi-view action recognition according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of a gated adaptive fusion model according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of a spatiotemporal cross attention mechanism according to an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of action classification accuracy in the NTU-RGB+D 60 dataset according to an embodiment of the present invention;

[0018] Figure 6 t-SNE visualization diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0022] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0023] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0024] See also Figure 1 As shown, the cross-view fusion method based on RGB modality for multi-view action recognition provided includes the following steps:

[0025] Step S110, constructing a cross-view fusion network, which uses a gated adaptive fusion model to aggregate spatial features from different views and highlights the feature areas of key views through dynamic weighting.

[0026] See also Figure 2As shown in the figure, the constructed cross-view fusion network mainly includes a shallow feature extraction module, a temporal feature extraction module, a spatial feature extraction module, a temporal-spatial cross attention mechanism module and a multi-view feature fusion module. The shallow feature extraction module is used to extract shallow features from multi-view videos, and then obtain temporal features and spatial features from shallow features through, for example, global pooling. The temporal feature extraction module is used to obtain temporal features. The spatial feature extraction module is used to obtain spatial features, for example, the spatial feature fusion module uses a gated adaptive fusion model to extract spatial features. The temporal-spatial cross attention mechanism module is used to perform spatiotemporal fusion across views to obtain global features. The multi-view fusion feature module is used to predict the action category in the video through global features.

[0027] In the following, the spatiotemporal feature processing process and the spatiotemporal cross-attention mechanism of the present invention will be explained in detail.

[0028] 1) Spatiotemporal feature processing

[0029] In one embodiment, a novel method is proposed for spatiotemporal feature processing. Considering that the differences and complementarities between multi-view data mainly exist in spatial features, the fusion strategy first emphasizes spatial information and then integrates temporal information for enhancement. The spatial stream S and temporal stream T are obtained from shallow features through global average pooling. Then gated adaptive fusion is applied to the spatial stream S to promote competition between channels.

[0030] Specifically, in order to determine the feature priority in the spliced ​​full view representation, a gated adaptive fusion model is proposed, such as Figure 3 As shown in the figure, H represents height and W represents width. The model mainly includes convolutional layers, Softmax function processing layers, channel normalization layers, and tanh function processing layers. First, global attention pooling is used to aggregate global context information in each channel. Unlike global average pooling, global attention pooling dynamically weights the input content to highlight important feature areas, expressed as:

[0031]

[0032] Where N is the area of ​​each frame, which can be understood as dividing a frame into blocks numbered 1 to N, and n and m are the indexes of the blocks. k is the weight matrix of 1*1 convolution, C is the number of channels, Z S represents the original spatial flow before pooling, z s Represents the spatial stream after pooling by the global attention mechanism.

[0033] Considering that channel normalization can achieve a consistent standard between channels, it is used to promote competition between features from different views. At the same time, weight α and bias β are used to adjust the weight of each channel. Finally, a residual connection is introduced to adjust the original channel according to the corresponding gating weight. The relevant formula is as follows:

[0034]

[0035] Among them, the size of weight α and bias β is [C, 1, 1], represents the original spatial stream before pooling of each channel, c represents the channel index, represents the normalized spatial stream after the global attention mechanism pooling. Represents the spatial stream after residual connection adjustment.

[0036] 2) Spatial-temporal cross attention mechanism

[0037] Different from the traditional use of cross-attention mechanism to emphasize the complementary relationship between different modalities, the present invention utilizes this mechanism to focus on the spatiotemporal variation information in different views in a global manner. Figure 4 The spatiotemporal cross attention mechanism module is illustrated, and the specific details are as follows. is considered as the main information and converted into a vector query Q s , and the time flow Z T is considered as context information and mapped into a set of independent vector values ​​V t and K t Key. Use the scaled multiplication attention mechanism to extract the vector Q s and K t Retrieve the correlation scores from , which represent the similarity between different frames and locations. Then, output Z C By combining the correlation matrix with the vector V t This operation obtains more detailed action features by utilizing the similarity between spatiotemporal features, which is specifically expressed as follows:

[0038]

[0039] Figure 4 In, Z C represents the cross attention output, T represents the frame, Represents the cross-attention output after FFN (feedforward network), and n represents the number of heads of the multi-head attention mechanism.

[0040] Therefore, the spatial features of each channel are able to capture critical temporal context information to obtain spatiotemporally enhanced action features.

[0041] In addition, a multi-head cross-attention mechanism is adopted to jointly learn the correlations between spatiotemporal features from different representation subspaces. Finally, Z C Get the final output features.

[0042] Step S120: training the cross-view fusion network using the set loss function.

[0043] The cross-view fusion network can be trained using an open source action recognition dataset or a self-built dataset until the set loss function is minimized. For example, the action recognition dataset uses NTU RGB-D 60. The loss function can use the cross entropy loss function.

[0044] Step S130: performing action recognition on the multi-view video actually captured using the trained cross-view fusion network.

[0045] After the cross-view fusion network training is completed, the optimized network parameters, such as weights, biases, etc., can be obtained. The trained cross-view fusion network can be used for actual action recognition. For example, the network application process includes: collecting multi-view videos for the target; inputting the multi-view videos into the trained cross-view fusion network to obtain action recognition results. The network application process is basically similar to the training process and will not be repeated here.

[0046] It should be noted that the training process of the cross-view fusion network involved in the present invention can be performed offline on a server or in the cloud, and real-time action recognition can be achieved by embedding the trained model into an electronic device. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a car computer, an intelligent wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be an independent server or a cluster server or a cloud server, etc.

[0047] In order to further verify the effect of the present invention, experiments were conducted.

[0048] 1) Comparative experiment.

[0049] The present invention is evaluated on three widely used multi-view action recognition datasets. For a comprehensive comparison, the present invention is compared with a variety of alternative methods, including skeleton-based, RGB+skeleton, and RGB-based methods, and the comparison results are shown in Table 1. Although the method combining RGB data and skeleton data achieves the best recognition performance, its application in practical multi-view environments is limited by the acquisition of skeleton information.

[0050] Table 1: Comparison with state-of-the-art methods on the NTU-RGB+D 60 dataset

[0051]

[0052] As shown in Table 1, in the test of NTU-RGB+D 60 dataset, the performance of the present invention under CS (cross-subject) protocol is only inferior to the best method, and the accuracy is only 0.1% lower than the best method, showing its effectiveness. In CV (cross-view) setting, the present invention can achieve a significant performance of 94.3%, exceeding most existing algorithms. This evidence shows that the present invention is highly robust to view changes.

[0053] 2) Action category accuracy.

[0054] Figure 5 The performance of each category on the NTU-RGB+D 60 dataset is shown, including drinking, eating, brushing teeth, sneezing, coughing and other categories (not shown). It can be observed through experiments that, except for the "sneeze / cough" category, the present invention achieves an accuracy of more than 80% in all categories, and half of the categories achieve an accuracy of about 95% or more.

[0055] 3) Visualization experiment.

[0056] Figure 6 The feature visualization of the top ten categories in the NTU-RGB+D 60 dataset is shown. The output feature distribution of ResNet3D and the network proposed in the present invention is compared through the t-SNE embedding diagram. For the two different models, each color represents an action category. The left figure corresponds to ResNet3D, and the right figure corresponds to the present invention. It can be observed that the network model proposed in the present invention generates better clustering effect than ResNet3D.

[0057] In summary, compared with the prior art, the present invention has the following advantages:

[0058] 1) This invention aggregates contextual information of different channels through global attention pooling, and uses a gating mechanism to weight features, highlighting the feature areas of key views, thereby dynamically adjusting the importance of each channel and enhancing the robustness of multi-view features. Subsequently, using temporal features as contextual information, the similarity of spatiotemporal positions is extracted through a self-attention mechanism, further achieving the complementarity of different views and capturing more comprehensive action features. This invention improves the accuracy and robustness of action recognition by separating and aggregating spatiotemporal features on a global scale.

[0059] 2) The cross-view fusion network designed in the present invention is based on the RGB modality. It is only based on RGB video data and does not rely on depth sensors or skeleton data, which effectively reduces the hardware cost, making it more flexible in more practical application scenarios and easy to deploy in actual monitoring scenarios.

[0060] 3) The gated adaptive fusion mechanism designed by the present invention can enhance the effective spatial features under multiple views, and proposes a spatiotemporal cross-attention mechanism to achieve efficient fusion of spatiotemporal features under different views. For example, through the gated adaptive fusion module, the spatial features from different views can be effectively aggregated, and the feature areas of the key views can be highlighted through dynamic weighting. Compared with simple feature splicing or global average pooling, it can better capture the complementary information between multiple views, thereby improving the accuracy of action recognition. At the same time, the spatiotemporal cross-attention mechanism adopted by the present invention takes spatial features as the main information source and combines temporal features for contextual association. This mechanism can fuse multi-view data in the spatiotemporal dimension and extract the similarities of different spatiotemporal positions, thereby more comprehensively capturing action features and improving recognition robustness. In short, through gated adaptive fusion and spatiotemporal cross-attention mechanisms, the feature information of different views can be fully integrated to improve the robustness of action recognition.

[0061] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0062] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0064] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0065] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0066] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0067] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0068] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0069] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A cross-view fusion method based on RGB modality for multi-view action recognition, comprising the following steps: Collect multi-view videos for the target; Inputting the multi-view video into a trained cross-view fusion network to obtain an action recognition result; Among them, the cross-view fusion network includes a shallow feature extraction module, a temporal feature extraction module, a spatial feature extraction module, a spatiotemporal cross-attention mechanism module and a multi-view feature fusion module. The shallow feature extraction module is used to extract shallow features from multi-view videos, the temporal feature extraction module is used to obtain the temporal flow features in the shallow features, the spatial feature extraction module is used to obtain the spatial flow features in the shallow features, the spatiotemporal cross-attention mechanism module is used to perform spatiotemporal fusion across views to obtain global features, and the multi-view fusion feature module is used to predict the action category in the video based on the global features.

2. The method according to claim 1, characterized in that The spatial feature extraction module uses a gated adaptive fusion model to extract spatial features, which is expressed as; in: in, represents the original spatial stream before pooling of each channel, c represents the channel index, C represents the number of channels, represents the normalized spatial stream after the global attention mechanism pooling. represents the spatial stream after residual connection adjustment, α is the weight, β is the bias, N is the area of ​​each frame, which means dividing a frame into blocks numbered 1 to N, n and m are the indexes of the blocks, and W k It is the weight matrix of 1*1 convolution.

3. The method according to claim 2, characterized in that For the spatiotemporal cross attention mechanism module, the spatial flow Convert to query vector Q s , time stream A T As context information, and mapped into a set of independent value vectors V t and the key vector K t , the spatiotemporal cross attention mechanism module uses the similarity between spatiotemporal features to extract action features, which is expressed as: Among them, Z c represents the cross attention output.

4. The method according to claim 3, characterized in that The spatiotemporal crisscross attention mechanism module outputs z from the crisscross attention through a residual connection and a feedforward network. c The final output features are obtained.

5. The method according to claim 1, characterized in that The cross-view fusion network is trained on a local server or in the cloud.

6. The method according to claim 1, characterized in that The multi-view video is used for security monitoring, intelligent transportation or behavior analysis.

7. The method according to claim 1, characterized in that The cross-view fusion network is trained based on the NTU RGB-D 60 action recognition dataset.

8. The method according to claim 1, characterized in that The loss function for training the cross-view fusion network adopts a cross entropy loss function.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images