Multi-view motion recognition method and device based on deep learning

Abstract

The invention discloses a multi-view action recognition method and device based on deep learning, and the method comprises the steps: obtaining a multi-view image containing a human body, extracting a human body position region in the image based on a pre-trained YOLOv5 model, and carrying out the preprocessing of the human body position region; constructing a multi-view action recognition model, wherein the structure of the multi-view action recognition model comprises a multi-view perception layer, a Backbone network layer and an output layer; using the preprocessed data to train the multi-view action recognition model; deploying the pre-trained YOLOv5 model and the trained multi-view action recognition model, and carrying out the deployment of the pre-trained YOLOv5 model and the trained multi-view action recognition model; and taking a to-be-processed image as input to realize human body multi-view action recognition. According to the method, the influence of shielding and view angle change on action recognition can be effectively overcome, so that the accuracy and robustness of action recognition are improved; the method is suitable for various complex scenes.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, image processing, and computer vision technology, and more specifically to a multi-view action recognition method and device based on deep learning. Background Art

[0002] Currently, among the existing motion detection and recognition technologies, most methods rely on single-view video or image data. However, single-view data often cannot provide sufficient spatial information, especially in some complex scenes, such as multi-person interaction, occlusion, etc., the accuracy of motion detection and recognition decreases.

[0003] Therefore, how to effectively utilize multi-view data to improve the accuracy of action detection and recognition has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present invention provides a multi-view action recognition method and device based on deep learning. By fusing video or image data from multiple viewpoints, the method can more comprehensively capture the spatial characteristics and morphological change characteristics of the action, improve the accuracy of action detection and recognition, and effectively solve the problem of reduced action recognition accuracy in complex scenarios, such as multi-person interaction, occlusion, and viewpoint changes; it can be applied to fields such as intelligent monitoring of safe production.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, an embodiment of the present invention provides a multi-view action recognition method based on deep learning, comprising the following steps:

[0007] Obtain multi-view images containing human bodies, extract the human body location area in the image based on the pre-trained YOLOv5 model and perform preprocessing;

[0008] Constructing a multi-view action recognition model, wherein the multi-view action recognition model structure includes: a multi-view perception layer, a backbone network layer and an output layer;

[0009] Using the preprocessed data to train the multi-view action recognition model;

[0010] The pre-trained YOLOv5 model and the trained multi-view action recognition model are deployed; the image to be processed is used as input to realize multi-view action recognition of human body.

[0011] Furthermore, a multi-view image containing a human body is obtained, and the human body position area in the image is extracted based on a pre-trained YOLOv5 model and pre-processed; including:

[0012] Deploy and set up multiple cameras at preset heights and angles to capture images containing human movements in the same scene;

[0013] Obtaining annotation data of human target frames and action behaviors in the image, and dividing the annotated image into a training set and a test set according to a preset ratio;

[0014] Use the pre-trained YOLOv5 model to identify and locate the human body area in the image;

[0015] The human body position area is expanded and cropped to serve as an action recognition image area.

[0016] Furthermore, the multi-view perception layer, as an input layer of the multi-view action recognition model, is used to perceive the spatial feature information of each view and extract shallow semantic features of the image;

[0017] The multi-view perception layer is composed of 4 Branch branches with the same structure; each Branch branch adopts a three-layer convolution structure; each convolution structure is composed of a convolution layer, a batch normalization layer, and an activation function layer connected in sequence; the 4 Branch branches each extract the human motion features under the corresponding perspective and splice them through concat.

[0018] Furthermore, the Backbone network layer uses ResNet18 as the basic network structure to extract deep semantic features of the image; the ResNet18 uses the BasicBlock module to build the network, introduces residual connections, and adds a channel attention module to assign different importance weights to different channels.

[0019] Furthermore, the channel attention module includes:

[0020] a) A global pooling layer, which is used to perform a global pooling operation on the feature map of each channel to obtain a one-dimensional vector; the vector contains the feature information of all positions of the channel;

[0021] b) Fully connected layer, used to fully connect the vector output by the global pooling layer and learn the weight of each channel through nonlinear transformation;

[0022] c) Normalization layer, which is used to normalize the weights of the output of the fully connected layer so that the sum of the weights of all channels is 1; the weight of each channel is used as the attention coefficient of each channel;

[0023] The channel attention module multiplies the obtained channel attention coefficient vector by the original feature map element by channel, and weights the features of each channel.

[0024] Furthermore, the output layer is composed of a fully connected layer and is used to predict the action category.

[0025] Further, the multi-view action recognition model is trained using the preprocessed data; including:

[0026] The pre-processed data is resized to a uniform size and used as the input of the multi-view action recognition model;

[0027] Set batch parameters, learning rate parameters, and number of iterations, and use the Adam optimizer and cross entropy loss function for training;

[0028] During the training process, after each iteration, the accuracy of the corresponding model in the test set is calculated;

[0029] When the accuracy meets the preset conditions, the corresponding model weight is saved as the optimal recognition model.

[0030] In a second aspect, an embodiment of the present invention further provides a multi-view action recognition device based on deep learning, comprising:

[0031] The acquisition and preprocessing module is used to acquire multi-view images containing human bodies, extract the human body location area in the image based on the pre-trained YOLOv5 model, and perform preprocessing;

[0032] A construction module is used to construct a multi-view action recognition model, wherein the multi-view action recognition model structure includes: a multi-view perception layer, a Backbone network layer and an output layer;

[0033] A training module, used to train the multi-view action recognition model using the preprocessed data;

[0034] The inference module is deployed to deploy the pre-trained YOLOv5 model and the trained multi-view action recognition model; the image to be processed is used as input to realize multi-view action recognition of human body.

[0035] In a third aspect, an embodiment of the present invention further provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0036] Memory, used to store computer programs;

[0037] The processor, when used to execute the program stored in the memory, can implement a multi-view action recognition method based on deep learning as described in any one of the first aspects.

[0038] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes a multi-view action recognition method based on deep learning as described in any one of the first aspects.

[0039] The descriptions of the second to fourth aspects of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second to fourth aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0040] It can be seen from the above technical solution that compared with the prior art, the present invention has the following advantages:

[0041] 1) A MARM model multi-view action recognition method is proposed, which can effectively overcome the influence of occlusion and perspective change on action recognition, thereby improving the accuracy and robustness of action recognition.

[0042] 2) A multi-view perception layer is proposed to perceive and fuse the spatial feature information from multiple perspectives, which can more comprehensively capture the spatial features and morphological change characteristics of the action;

[0043] 3) Improve and optimize the BasicBlock structure of ResNet18 and add a channel attention module. By assigning different importance weights to different channels, the model can focus on features that are more critical to the current task, thereby improving the overall feature representation capability of the model.

[0044] 4) It supports multiple action recognition and is suitable for various complex scenes. Even in complex scenes where a certain perspective is blocked, it can accurately recognize actions and has strong robustness and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 Flowchart of the multi-view action recognition method based on deep learning provided by the present invention.

[0047] Figure 2 This is a diagram of the implementation process of Example 1 provided by the present invention.

[0048] Figure 3 This is a schematic diagram of the site deployment provided by the present invention.

[0049] Figure 4 This is a schematic diagram of the human body frame processing provided by the present invention.

[0050] Figure 5 A schematic diagram of the structure of a multi-view action recognition model is provided for the present invention.

[0051] Figure 6 A schematic diagram of the convolution structure of the Branch is provided for the present invention.

[0052] Figure 7 It is a schematic diagram of the BasicBlock module structure in the prior art.

[0053] Figure 8 This is a schematic diagram of the improved BasicBlock module structure provided by the present invention.

[0054] Fig. 9 A block diagram of a multi-view action recognition device based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Embodiment 1:

[0057] The embodiment of the present invention discloses a multi-view action recognition method based on deep learning, referring to Figure 1 As shown, the following steps are included:

[0058] S10, obtaining a multi-view image containing a human body, extracting a human body position area in the image based on a pre-trained YOLOv5 model, and performing pre-processing;

[0059] S20, constructing a multi-view action recognition model, wherein the multi-view action recognition model structure includes: a multi-view perception layer, a Backbone network layer and an output layer;

[0060] S30, training the multi-view action recognition model using the preprocessed data;

[0061] S40, deploying the pre-trained YOLOv5 model and the trained multi-view action recognition model; taking the image to be processed as input to realize multi-view action recognition of human body.

[0062] In this embodiment, step S10 uses the pre-trained YOLOv5 model to extract the human body position area in the image to ensure the efficiency and accuracy of human body detection. The YOLOv5 model is pre-trained and can quickly identify and locate the human body, thereby providing accurate input data for subsequent action recognition.

[0063] In step S20, a multi-view action recognition model is constructed, and the multi-view features of human actions are obtained through the multi-view perception layer, which helps to improve the accuracy and robustness of action recognition. In addition, the multi-view perception layer can integrate image information from different perspectives, overcoming the information loss and misjudgment problems that may be caused by a single perspective. The Backbone backbone network layer can be used for deep feature extraction to fully explore the action features in the image and improve the expression ability of the model. In addition, the Backbone backbone network layer uses a deep learning algorithm, which can extract richer and more abstract features and improve the performance of action recognition.

[0064] In step S30, through model optimization and efficient training, model training is performed based on the preprocessed data to ensure the consistency and quality of the input data, thereby improving the training efficiency and the generalization ability of the model.

[0065] In addition, during the model training process, efficient action recognition model training is achieved through the joint optimization of the multi-view perception layer, Backbone network layer and output layer.

[0066] Finally, in step S40, real-time deployment and application:

[0067] The pre-trained YOLOv5 model and the trained multi-view action recognition model can be deployed, which can quickly process the images to be recognized and realize real-time multi-view action recognition of the human body.

[0068] The method is applicable to practical application scenarios and can be deployed in different devices and environments to meet the requirements of real-time and accuracy. The method of the present invention has significant improvements in the accuracy, efficiency and robustness of action recognition and is applicable to action recognition tasks in a variety of complex scenarios.

[0069] The technical solution of the present invention is described in detail from the following four stages:

[0070] 1. The first stage is site deployment, which mainly involves deploying and setting up multiple cameras at appropriate heights / angles to capture more complete and comprehensive multi-view action images;

[0071] 2. In the second stage, the dataset is constructed to collect action recognition data (such as smoking, making phone calls, etc., but not limited to these actions) and manually annotate them for model training and testing;

[0072] 3. The third stage is human body detection, which uses the pre-trained YOLOv5 model to detect and locate the area of ​​the human body in the image;

[0073] 4. The fourth stage is action recognition, building a multi-view action recognition model to achieve multi-view action recognition.

[0074] The specific implementation process is as follows Figure 2 shown.

[0075] The first stage is site deployment, which mainly involves deploying and setting up multiple cameras at appropriate heights and angles to capture more complete and comprehensive multi-view action images. This solution uses wide-angle surveillance cameras for site deployment, which have a horizontal field of view of 120° and a vertical field of view of 93°. In order to cover the entire action as much as possible, the cameras need to be deployed with the site as the center, at a height of 2 meters, with one camera deployed around the site, one camera each in front, back, left, and right, for a total of 4 cameras. Figure 3 shown.

[0076] The second stage is to build a data set: use 4 cameras to collect video or image data containing actions in the same scene, and use annotation tools to manually annotate the human target box and action behaviors in the image (such as smoking, making phone calls, etc., but not limited to these actions). The collected data is divided into a training set and a test set in a ratio of 7:3, where the training set is used to train the model and the test set is used to evaluate the model performance. These data are only used in the model training process. After the model training is completed, these data are not needed in the subsequent detection, recognition and testing processes.

[0077] The third stage is human detection: As an important application in the field of computer vision, human detection aims to accurately identify and locate the human body in an image or video. This technology is not only crucial for security monitoring, intelligent security, human-computer interaction and other fields, but also the basis for modern intelligent systems to achieve efficient action recognition and behavior analysis. To achieve this goal, this solution directly uses the pre-trained model YOLOv5, which has been fully trained on the large-scale COCO (Common Objects in Context) dataset, covering various complex scenes and multiple object categories, including human bodies. By using the YOLOv5 model, human bodies in images can be quickly and accurately identified and their location areas can be precisely located. This provides a solid foundation for subsequent action recognition, behavior analysis and other work.

[0078] The fourth stage is action recognition: using the multi-view perception layer as the input layer and ResNet18 as the backbone network, a multi-view action recognition model (Multi-view Action Recognition Model, MARM) is proposed:

[0079] 1) Data processing: The data collected in the second stage is processed using the human body detection method, that is, using YOLOv5 to predict the human body frame; although YOLOv5 has excellent performance, in practical applications, due to various factors such as complex background, illumination changes, occlusion, etc. in the image, the human body frame predicted by YOLOv5 may not be perfect. Specifically, the predicted human body frame may not fully include all parts of the human body, especially key parts such as hands, feet or head, which play an important role in action recognition.

[0080] To solve this problem, it is necessary to adopt an expansion and cropping strategy when processing data. Specifically, after YOLOv5 predicts the human body frame, it does not directly use it as the image area for action recognition, but expands it by 1.2 times. The purpose of this step is to ensure that even if the predicted human body frame is slightly deviated, it can be fully included through the expansion operation. At the same time, it is also necessary to pay attention to the scale of the expansion. Excessive expansion may lead to the introduction of too much background noise, affecting the subsequent action recognition effect.

[0081] Therefore, after multiple experimental verifications, this embodiment selects 1.2 times as a more appropriate expansion ratio. After expanding the human body frame, a cropping operation is required to obtain the final action recognition image area. The purpose of cropping is to remove redundant background information and make the image more focused on the human body area. Through this step, the accuracy of action recognition can be further improved. Therefore, after using YOLOv5 to predict the human body frame, it is necessary to expand it 1.2 times and crop the area as the action recognition image area, which can effectively solve the problem that the predicted human body frame may be inaccurate and ensure the accuracy of subsequent action recognition. The processing flow can be referred to Figure 4 As shown, the red box is the human body area predicted by YOLOv5, and the green box is the area expanded 1.2 times; the rightmost side of the figure is the action recognition image area obtained by cropping this area.

[0082] 2) Construct a multi-view action recognition model: The currently commonly used deep learning CNN networks (such as MobileNet, VGG16, ResNet) all use single-view single-frame images as model input, do not support multi-view multi-frame image input, and lack multi-view spatial information perception and learning. In order to solve this problem, the method of the present invention proposes a multi-view action recognition model (Multi-view Action Recognition Model, MARM) with a multi-view perception layer as the input layer and ResNet18 as the backbone network. The model supports multi-view input and multi-view perception, and can more comprehensively capture the spatial characteristics and morphological change characteristics of the action. Figure 5As shown in the figure, the MARM model structure can be divided into: multi-view perception layer, Backbone network layer, and output layer. The specific descriptions are as follows:

[0083] a. Multi-view perception layer: As the input layer of the MARM model, it is used to perceive the spatial feature information of each view and extract the shallow semantic features of the image. It consists of 4 branches, and each branch is consistent in structure to ensure that the same processing logic and parameter settings can be maintained when processing images of different viewpoints. Figure 6 As shown in the figure, each branch adopts a three-layer convolution structure: Conv-BN-Relu6-Conv-BN-Relu6-Conv-BN-Relu6. This structure can not only effectively extract the key features in the image, but also enable the model to learn more complex mapping relationships by introducing the nonlinear activation function Relu6. Figure 6 As shown in the figure, in the three-layer convolution structure, each layer of convolution is set with a different stride. The stride of the first layer of convolution is set to stride = 2, which means that during the convolution process, the filter moves with a step of 2 pixels. This setting helps to quickly reduce the size of the image, reduce the amount of calculation, and retain the key information in the image. The stride of the second layer of convolution is set to stride = 1, that is, the filter only moves 1 pixel each time it moves, which helps to further extract the detailed features of the image. The stride of the third layer of convolution is set to stride = 2 again to further reduce the image size and prepare to pass the feature map to the subsequent network layer. When a perspective image is input, the Branch module uses the above three-layer convolution structure to perform a convolution operation on the image to extract the human motion features under that perspective. These features contain spatial feature information and shallow semantic information, which are then spliced ​​by concat to achieve feature information fusion of different network layers and different perspectives.

[0084] b. Backbone network layer: ResNet18 is used as the basic network structure to extract deep semantic features of images. ResNet18 uses BasicBlock modules to build the network, such as Figure 7 As shown in , this module introduces residual connections, which allow input data to bypass some layers directly and pass to a deeper layer and be summed with the output of that layer. In this way, even when the network depth increases, the gradient can be effectively back-propagated, thus solving the gradient vanishing and gradient exploding problems in deep neural network training. However, the BasicBlock module still lacks the design of model attention, such as Figure 8As shown in the figure, the solution of the present invention improves and optimizes the BasicBlock structure of ResNet18, adds a channel attention module, and assigns different importance weights to different channels, so that the model can focus on those features that are more critical to the current task, thereby improving the overall feature representation ability of the model. The channel attention module is mainly composed of structures such as Global Pooling-FC-sigmoid, and is specifically implemented as follows:

[0085] a) Global Pooling: It is responsible for performing global pooling operations on the feature maps of each channel to obtain a one-dimensional vector. This vector contains the feature information of all positions in the channel and is crucial for the subsequent learning process.

[0086] b) Fully connected layer (FC), which performs a fully connected operation on the vector output by the Global Pooling layer and learns the weight of each channel through nonlinear transformation. These weights reflect the importance of different channels to the current task and are the core of the attention mechanism.

[0087] c) Normalization layer: It normalizes the weights output by the FC layer (such as the sigmoid function) so that the sum of the weights of all channels is 1. In this way, the weight of each channel becomes a probability value between 0 and 1, that is, the attention coefficient of each channel is obtained, which is convenient for the subsequent network layer to perform weighted operations.

[0088] d) Attention application: The obtained channel attention coefficient vector is multiplied by the original feature map by channel element-wise (element-wise multiplication), so as to weight the features of each channel. The weighted feature map retains the original spatial structure, but the influence of each channel has been adaptively adjusted.

[0089] c. Output layer: It consists of a fully connected layer and is used to predict action categories. In order to train and optimize this output layer, a loss function (such as the cross entropy loss function) is usually defined to measure the difference between the model prediction results and the true labels, where the true labels of the data can be obtained by manual labeling using labeling tools. During the training process, the model will continuously adjust the model's weights and bias terms through the back-propagation algorithm and the gradient descent optimizer to minimize the loss function, thereby improving the prediction accuracy of the action category.

[0090] 3) Training action recognition model: Use the above training set to train and optimize the MARM model; use the above test set to test the MARM model and evaluate the performance of the model indicators. During model training, the input data is uniformly scaled to 224×224, the batch size batch_size=32, the optimization algorithm uses the Adam optimizer, the initial learning rate is set to lr=0.001, the cross entropy loss function is used, and the entire data set is iterated epoch=200 times; during the training process, the program calculates the accuracy of the model in the test set once after completing a complete data iteration, and saves the model weight with the highest current accuracy; in subsequent actual use, only the model weight file and the corresponding inference program code need to be retained for recognition.

[0091] 4) Model deployment: After training, the MARM action recognition model is obtained. Combined with the human body detection model, the YOLOv5 model and the MARM model are deployed on the deep learning machine platform for model reasoning and testing to achieve multi-view action recognition.

[0092] During specific implementation, appropriate modifications and adjustments can be made according to actual needs. The present invention can be applied to the following fields:

[0093] (1) Sports training and game analysis: In sports training, coaches can use multiple cameras to capture the movements of athletes, and use the method of the present invention to detect and identify the movements, thereby analyzing the athletes' technical movements, postures, reaction speed, etc., and providing athletes with more accurate training suggestions. In competitions, referees can use this method to make fair judgments and avoid errors caused by human factors.

[0094] (2) Smart home and security: In a smart home system, comprehensive monitoring of family members can be achieved by installing multiple cameras. Using the motion detection and recognition method of the present invention, the behavior patterns of family members can be automatically identified, such as the elderly falling down, children leaving the safe area, etc., and alarms can be issued in time to improve family safety.

[0095] (3) Medical rehabilitation and health monitoring: In the field of medical rehabilitation, this method can be used to analyze the patient's movement performance, such as gait, joint range of motion, etc., to evaluate the patient's rehabilitation progress and treatment effect. At the same time, for patients who need to stay in bed for a long time or have difficulty in moving, remote health monitoring can be performed by installing a camera and using this method to detect abnormal conditions in a timely manner.

[0096] (4) Autonomous driving and intelligent transportation: In autonomous driving and intelligent transportation systems, by integrating traffic scene data captured by multiple cameras, the motion detection and recognition method of the present invention can be used to analyze the behavior patterns of traffic participants such as pedestrians and vehicles in real time, providing more accurate decision support for autonomous driving vehicles and improving traffic safety.

[0097] (5) The method of the present invention can be applied to intelligent security monitoring and intelligent detection scenarios. The entire implementation process is clear and simple. In addition to the camera, no additional sensor equipment is required. The deployment cost is low and the detection effect is good.

[0098] Embodiment 2:

[0099] Reference Fig. 9 As shown, based on the same inventive concept, an embodiment of the present invention further provides a multi-view action recognition device based on deep learning, comprising:

[0100] The acquisition and preprocessing module is used to acquire multi-view images containing human bodies, extract the human body location area in the image based on the pre-trained YOLOv5 model, and perform preprocessing;

[0101] A construction module is used to construct a multi-view action recognition model, wherein the multi-view action recognition model structure includes: a multi-view perception layer, a Backbone network layer and an output layer;

[0102] A training module, used to train the multi-view action recognition model using the preprocessed data;

[0103] The inference module is deployed to deploy the pre-trained YOLOv5 model and the trained multi-view action recognition model; the image to be processed is used as input to realize multi-view action recognition of human body.

[0104] Through the above modules, the device achieves efficient data acquisition and preprocessing. Through the pre-trained YOLOv5 model, it can quickly and accurately extract the human body position area and perform preprocessing, which significantly improves the efficiency and accuracy of data processing and ensures the quality of input data. The constructed multi-view action recognition model includes a multi-view perception layer, a Backbone network layer, and an output layer. This structural design can effectively perceive and fuse the spatial feature information of multiple perspectives, and improve the model's ability to capture the spatial features and morphological changes of actions.

[0105] In addition, the model is trained using preprocessed data to ensure the high quality and consistency of the training data. By deploying the pre-trained YOLOv5 model and the trained multi-view action recognition model, real-time, multi-view action recognition of the processed images can be achieved. This not only improves the accuracy and robustness of recognition, but also adapts to real-time application requirements in complex scenarios.

[0106] Embodiment 3:

[0107] The embodiment of the present invention further provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0108] Memory, used to store computer programs;

[0109] The processor, when used to execute the program stored in the memory, can implement a multi-view action recognition method based on deep learning as any one of the embodiments in Example 1.

[0110] In this embodiment, a complete hardware solution is provided by integrating a processor, a communication interface, a memory and a communication bus through a computer device, which simplifies the implementation and deployment of the system. The processor executes the program on the memory, which can efficiently implement the multi-view action recognition method based on deep learning, and improve the response speed and processing power of the system. The design of the computer device makes it have good flexibility and scalability, and can adapt to the multi-view action recognition needs in different scenarios.

[0111] Embodiment 4:

[0112] An embodiment of the present invention further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes a multi-view action recognition method based on deep learning as described in Example 1.

[0113] The program instructions on the computer-readable storage medium can be run on different processors, providing cross-platform compatibility and being applicable to a variety of computing devices. The program instructions on the storage medium are optimized and can be efficiently executed on the processor, thereby improving the overall performance and accuracy of multi-view action recognition.

[0114] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-view action recognition method based on deep learning, characterized in that: The following steps are involved: Obtain multi-view images containing human bodies, extract the human body location area in the image based on the pre-trained YOLOv5 model and perform preprocessing; Constructing a multi-view action recognition model, wherein the multi-view action recognition model structure includes: a multi-view perception layer, a backbone network layer and an output layer; Using the preprocessed data to train the multi-view action recognition model; The pre-trained YOLOv5 model and the trained multi-view action recognition model are deployed; the image to be processed is used as input to realize multi-view action recognition of human body.

2. The multi-view action recognition method based on deep learning according to claim 1, characterized in that: Obtain multi-view images containing human bodies, extract the human body location area in the image based on the pre-trained YOLOv5 model and perform preprocessing; include: Deploy and set up multiple cameras at preset heights and angles to capture images containing human movements in the same scene; Obtaining annotation data of human target frames and action behaviors in the image, and dividing the annotated image into a training set and a test set according to a preset ratio; Use the pre-trained YOLOv5 model to identify and locate the human body area in the image; The human body position area is expanded and cropped to serve as an action recognition image area.

3. The multi-view action recognition method based on deep learning according to claim 1, characterized in that: The multi-view perception layer, as the input layer of the multi-view action recognition model, is used to perceive the spatial feature information of each view and extract the shallow semantic features of the image; The multi-view perception layer is composed of 4 Branch branches with the same structure; each Branch branch adopts a three-layer convolution structure; each convolution structure is composed of a convolution layer, a batch normalization layer, and an activation function layer connected in sequence; the 4 Branch branches each extract the human motion features under the corresponding perspective and splice them through concat.

4. The multi-view action recognition method based on deep learning according to claim 1, characterized in that: The Backbone network layer uses ResNet18 as the basic network structure to extract deep semantic features of the image; the ResNet18 uses the BasicBlock module to build the network, introduces residual connections, and adds a channel attention module to give different importance weights to different channels.

5. The multi-view action recognition method based on deep learning according to claim 4, characterized in that: The channel attention module includes: a) A global pooling layer, which is used to perform a global pooling operation on the feature map of each channel to obtain a one-dimensional vector; the vector contains the feature information of all positions of the channel; b) Fully connected layer, used to fully connect the vector output by the global pooling layer and learn the weight of each channel through nonlinear transformation; c) Normalization layer, which is used to normalize the weights of the output of the fully connected layer so that the sum of the weights of all channels is 1; the weight of each channel is used as the attention coefficient of each channel; The channel attention module multiplies the obtained channel attention coefficient vector by the original feature map element by channel, and weights the features of each channel.

6. The multi-view action recognition method based on deep learning according to claim 1, characterized in that: The output layer, consisting of a fully connected layer, is used to predict the action category.

7. A multi-view action recognition device based on deep learning, characterized in that: include: The acquisition and preprocessing module is used to acquire multi-view images containing human bodies, extract the human body location area in the image based on the pre-trained YOLOv5 model, and perform preprocessing; A construction module is used to construct a multi-view action recognition model, wherein the multi-view action recognition model structure includes: a multi-view perception layer, a Backbone network layer and an output layer; A training module, used to train the multi-view action recognition model using the preprocessed data; The inference module is deployed to deploy the pre-trained YOLOv5 model and the trained multi-view action recognition model; the image to be processed is used as input to realize multi-view action recognition of human body.

8. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor, when used to execute the program stored in the memory, can implement a multi-view action recognition method based on deep learning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, enable the processor to execute a multi-view action recognition method based on deep learning as described in any one of claims 1 to 6.

Images