Human Motion Recognition Method, Device, Computer Equipment and Storage Medium

Abstract

A human action recognition method disclosed by the present invention includes: acquiring a video stream; performing discrete sampling processing and caching on the acquired video stream; detecting image data; performing unsupervised clustering processing on depth image data packets; extracting optical flow information; performing spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; and calculating a category score activation result, and obtaining an action recognition classification result of the current sampling batch according to the category score activation result. An apparatus, a computer device, and a storage medium for implementing the above human action recognition method are also disclosed. The present invention improves the engineering scenario adaptation performance, reduces the optical flow calculation error perturbation, and improves the action recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior recognition methods, and in particular, to a human action recognition method, device, computer device, and storage medium. Background Art

[0002] Video behavior recognition is a type of task that enables a computer to learn to understand human behaviors through algorithms. Different from the image classification task, it mainly focuses on the classification problem of behaviors in the spatio-temporal semantics composed of video frames within a period of time. Since the booming development of deep learning in the field of images, video behavior recognition has been more used in the field of security monitoring, and its own algorithm research has also experienced a development process from traditional algorithms to deep learning methods.

[0003] The traditional algorithm (iDT algorithm) calculates the dense optical flow information between two consecutive frames of images, and then calculates the optical flow field of a series of consecutive video frames. Then, the motion trajectory information of the object is obtained from the optical flow field, and higher-level feature information encodings such as HOG (Histogram of Oriented Gradients), HOF (Histogram of Optical Flow), and MBH (Motion Boundary Histogram) are further extracted from the motion trajectory information. These feature information encodings are sent into an SVM (Support Vector Machine) for classification training. This type of algorithm is based on optical flow information, usually has a high computational complexity, and for long video segments, the sampling strategy becomes a crucial factor affecting the algorithm speed and accuracy. At the same time, due to the background optical flow noise caused by camera movement, its robustness in actual engineering scenarios is not very ideal.

[0004] Compared with the traditional optical flow-based action recognition algorithm, the action recognition algorithm based on a deep convolutional neural network borrows the powerful modeling ability of the convolutional neural network and absorbs and learns from the traditional optical flow method. In the extraction of optical flow information, the CNN modeling method is used to obtain the optical flow information, and then it is sent into the next network for high-level information extraction and classification. The common action recognition algorithms based on convolutional neural networks can generally be divided into two categories: one is the algorithm architecture based on 3D convolutional neural networks. Compared with 2D convolution, 3D convolution adds a time dimension and can directly process a series of video frame sequences; the other is the algorithm architecture based on 2D convolutional neural networks. The algorithm model architecture of this type can be divided into a 3D convolution decomposition architecture of 2D+1D and a two-stream network architecture. The former decomposes the 3D convolution into a 2D convolution in space and a 1D convolution in the time dimension. The input of the latter network has two branches. One is the image sequence input branch, which mainly extracts spatial information, and the other branch is the optical flow sequence input, which mainly extracts the spatio-temporal information of the frame sequence. The two branches finally perform feature-level fusion and are sent into a classifier for classification.

[0005] Whether it is a 3D convolutional architecture or a 2D convolutional two-stream architecture, the calculation of optical flow depends on a pre-trained neural network. Due to data defects (the background of the training data is single, and its extraction of optical flow is not very ideal for complex backgrounds and large noise situations), the optical flow extraction cannot be adapted to specific scenarios, that is, it is difficult for the optical flow extractor to be adapted to engineering scenarios. At the same time, most of the above solutions are based on the RGB-D (RGB image - depth image) or RGB data input mode, and there are a large number of uncertain factors in the migration from the RGB-D data domain to the IR-D (near-infrared image - depth image) data domain of the TOF camera, which is not conducive to engineering practice applications. In addition, there is currently no complete algorithm framework based on TOF implemented.

[0006] For this reason, the applicant has conducted beneficial exploration and research and found a method to solve the above problems. The technical solution to be introduced below was generated under this background. Summary of the Invention

[0007] One of the technical problems to be solved by the present invention is: to provide a human action recognition method that improves the adaptability performance of engineering scenarios, reduces the error perturbation of optical flow calculation, and improves the recognition accuracy in view of the deficiencies of the prior art.

[0008] Another technical problem to be solved by the present invention is: to provide a device for implementing the above human action recognition method.

[0009] Another technical problem to be solved by the present invention is: to provide a computer device for implementing the above human action recognition method.

[0010] Another technical problem to be solved by the present invention is: to provide a computer-readable storage medium for implementing the above human action recognition method.

[0011] As a human action recognition method according to the first aspect of the present invention, it includes:

[0012] Obtain a video stream containing depth images and near-infrared images from a TOF device module or a TOF video;

[0013] Perform discrete sampling processing on the obtained video stream and cache the image data obtained by the discrete sampling processing;

[0014] Detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, depth image data packets, and near-infrared image data packets;

[0015] Perform unsupervised clustering on the depth image data packet to obtain a background mask sequence, and perform background removal on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed;

[0016] Perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information;

[0017] Perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; and

[0018] Splice the spatio-temporal feature representation information with the scene space information to calculate the category score activation result, and obtain the action recognition classification result of the current sampling batch according to the category score activation result.

[0019] In a preferred embodiment of the present invention, the discrete sampling process is performed on the acquired video stream, and the image data obtained by the discrete sampling process is cached, including:

[0020] Perform discrete sampling on the acquired video stream through the first image collector and the second image collector respectively. The sampling step of the first image collector is L1, and the sampling step of the second image collector is L2, where L1 > L2 and L1, L2 are integers;

[0021] Perform mean and variance processing on the image data collected by the first image collector and the second image collector respectively; and

[0022] Store the processed image data into the first data buffer and the second data buffer for caching respectively.

[0023] In a preferred embodiment of the present invention, the image data in the cache is detected, and the image data that meets the detection requirements is subjected to spatial calculation processing and data packaging processing to obtain scene space information, a depth image data packet, and a near-infrared image data packet, including:

[0024] Judge whether the size in the first data buffer is greater than or equal to N, where N ≥ 2 and is an integer. If the judgment is yes, take out N first IR-D image data from the first data buffer, and set the size in the first data buffer to 0;

[0025] Determine whether the size in the second data buffer is greater than or equal to (N - 1)L1 / L2, where (N - 1)L1 / L2 is an integer. If the determination is yes, take out (N - 1)L1 / L2 second IR-D image data from the second data buffer, and set the size in the second data buffer to 0;

[0026] Obtain the IR image data from the N first IR-D image data, perform data packing processing on all the obtained IR image data to obtain an IR image data packet, and then send the IR image data packet to the slow branch network SSCNN for spatial calculation processing to obtain scene spatial information;

[0027] Obtain the Depth image data from (N - 1)L1 / L2 second IR-D image data, perform data packing processing on all the obtained Depth image data to obtain a depth image data packet; and

[0028] Obtain the IR image data from (N - 1)L1 / L2 second IR-D image data, perform data packing processing on all the obtained IR image data to obtain a near-infrared image data packet.

[0029] In a preferred embodiment of the present invention, use the K-nearest neighbor algorithm to perform unsupervised clustering processing on each depth image in the depth image data packet to obtain a background mask sequence.

[0030] In a preferred embodiment of the present invention, the background removal processing of the depth image data packet and the near-infrared image data packet according to the background mask sequence is specifically to multiply the depth image data packet and the near-infrared image data packet by the mask value at the corresponding position serial number in the background mask sequence respectively to obtain a depth image sequence and a near-infrared image sequence with the background removed.

[0031] In a preferred embodiment of the present invention, the time-order pairing processing of the depth image sequence and the near-infrared image sequence is specifically to perform pairwise pairing in the time order on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence.

[0032] In a preferred embodiment of the present invention, use an optical flow extractor to perform optical flow information extraction processing on the spliced depth image pairing sequence and near-infrared image pairing sequence to obtain optical flow information.

[0033] In a preferred embodiment of the present invention, use a fast branch optical flow network FOFCNN to perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information.

[0034] In a preferred embodiment of the present invention, the concatenated spatio-temporal feature representation information and the scene space information are subjected to class score activation calculation processing through the feature fusion network FusionMLP.

[0035] As a human action recognition method device according to the second aspect of the present invention, it includes:

[0036] A video stream acquisition module, which is used to acquire a video stream containing a depth image and a near-infrared image from a TOF device module or a TOF video;

[0037] A discrete sampling processing module, which is used to perform discrete sampling processing on the acquired video stream and cache the image data obtained from the discrete sampling processing;

[0038] An image detection processing module, which is used to detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet;

[0039] An image background processing module, which is used to perform unsupervised clustering processing on the depth image data packet to obtain a background mask sequence, and perform background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence without background;

[0040] An optical flow information extraction module, which is used to perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and extract optical flow information after splicing the depth image pairing sequence and the near-infrared image pairing sequence;

[0041] A spatio-temporal feature representation module, which is used to perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; and

[0042] An action recognition and classification processing module, which is used to splice the spatio-temporal feature representation information and the scene space information to calculate the class score activation result, and obtain the action recognition and classification result of the current sampling batch according to the class score activation result.

[0043] As a computer device for implementing the above human action recognition method according to the third aspect of the present invention, it includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Obtain a video stream containing depth images and near-infrared images from a TOF device module or a TOF video;

[0045] Perform discrete sampling processing on the obtained video stream, and cache the image data obtained from the discrete sampling processing;

[0046] Detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, depth image data packets, and near-infrared image data packets;

[0047] Perform unsupervised clustering processing on the depth image data packets to obtain a background mask sequence, and perform background removal processing on the depth image data packets and near-infrared image data packets respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed;

[0048] Perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information;

[0049] Perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; and

[0050] Splice the spatio-temporal feature representation information and the scene space information to calculate the category score activation result, and obtain the action recognition classification result of the current sampling batch according to the category score activation result.

[0051] As a computer-readable storage medium for implementing the above-mentioned human action recognition method according to the fourth aspect of the present invention, a computer program is stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Obtain a video stream containing depth images and near-infrared images from a TOF device module or a TOF video;

[0053] Perform discrete sampling processing on the obtained video stream, and cache the image data obtained from the discrete sampling processing;

[0054] Detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, depth image data packets, and near-infrared image data packets;

[0055] Perform unsupervised clustering processing on the depth image data packets to obtain a background mask sequence, and perform background removal processing on the depth image data packets and near-infrared image data packets respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed;

[0056] Perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information;

[0057] Perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; and

[0058] Splice the spatio-temporal feature representation information with the scene space information to calculate the category score activation result, and obtain the action recognition classification result of the current sampling batch according to the category score activation result.

[0059] Due to the adoption of the above technical solutions, the beneficial effects of the present invention are as follows:

[0060] 1. The present invention uses the upper-level task to supervise the training of the two-stream classifier and the optical flow extractor, incorporates the optical flow extractor into the training task framework of the two-stream classifier, and can obtain an optical flow extractor that is more suitable for the engineering scenario, improving the engineering scenario adaptation performance and stability robustness;

[0061] 2. Use the self-supervised clustering algorithm (K-nearest neighbor) to cluster the input of the optical flow extractor to obtain a background class mask, and then remove the background elements in the image sequence, which can effectively avoid the optical flow calculation error perturbation caused by the relative movement of the camera and background noise;

[0062] 3. The present invention not only includes the 2D optical flow information of the IR near-infrared image, but also includes the 2D optical flow information from the depth distance. The 2D optical flow information of the depth distance can be understood as the movement trajectory of the human body on the z-axis in the three-dimensional space, while the 2D optical flow information of the IR near-infrared image can be understood as the movement trajectories of the human body on the x-axis and y-axis in the three-dimensional space. Compared with the 2D optical flow information based on RGB in the RGB-D scheme, more comprehensive human body movement spatio-temporal information can be obtained, improving the action recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0064] Figure 1 It is a schematic flowchart of the human action recognition method of the present invention.

[0065] Figure 2It is a schematic diagram of an application embodiment of the human motion recognition method of the present invention.

[0066] Figure 3 It is a schematic structural diagram of the device for implementing the human motion recognition method of the present invention.

[0067] Figure 4 It is the internal structure diagram of the computer device of the present invention. Detailed implementation manners

[0068] In order to make the technical means, creative features, achieved purposes and functions implemented by the present invention easy to understand, the present invention will be further described below with reference to specific illustrations.

[0069] See Figure 1 , what is shown in the figure is a human motion recognition method, including the following steps:

[0070] Step S10, obtain a video stream containing depth images and near-infrared images from a TOF device module or a TOF video.

[0071] Step S20, perform discrete sampling processing on the obtained video stream, and cache the image data obtained by the discrete sampling processing.

[0072] Step S30, detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet.

[0073] Step S40, perform unsupervised clustering processing on the depth image data packet to obtain a background mask sequence, and perform background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed. In step S40, the K-nearest neighbor algorithm is used to perform unsupervised clustering processing on each depth image in the depth image data packet to obtain a background mask sequence. Performing background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence specifically means multiplying the depth image data packet and the near-infrared image data packet by the mask value at the corresponding position serial number in the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed.

[0074] Step S50: Perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information. In step S50, performing chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively specifically means performing pairwise chronological pairing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence. An optical flow extractor is used to perform optical flow information extraction processing on the spliced depth image pairing sequence and near-infrared image pairing sequence to obtain optical flow information.

[0075] Step S60: Perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information. In step S60, a fast-branch optical flow network FOFCNN is used to perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information.

[0076] Step S70: Splice the spatio-temporal feature representation information with the scene space information and calculate the category score activation result, and obtain the action recognition classification result of the current sampling batch according to the category score activation result. In step S70, a feature fusion network FusionMLP is used to perform category score activation calculation processing on the spliced spatio-temporal feature representation information and scene space information.

[0077] In step S20, perform discrete sampling processing on the obtained video stream and cache the image data obtained by the discrete sampling processing, including the following steps:

[0078] Step S21: Respectively perform discrete sampling processing on the obtained video stream through a first image collector and a second image collector; wherein, the sampling step of the first image collector is L1, the sampling step of the second image collector is L2, L1 > L2 and L1, L2 are integers.

[0079] Step S22: Respectively perform mean and variance processing on the image data collected by the first image collector and the second image collector.

[0080] Step S23: Store the processed image data into a first data buffer and a second data buffer respectively for caching.

[0081] In step S30, detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet, including the following steps:

[0082] Step S31: Determine whether the size in the first data buffer is greater than or equal to N, where N≥2 and is an integer. If the determination is yes, take out N first IR-D image data from the first data buffer, and set the size in the first data buffer to 0.

[0083] Step S32: Determine whether the size in the second data buffer is greater than or equal to (N - 1)L1 / L2, where (N - 1)L1 / L2 is an integer. If the determination is yes, take out (N - 1)L1 / L2 second IR-D image data from the second data buffer, and set the size in the second data buffer to 0.

[0084] Step S33: Obtain the IR image data from the N first IR-D image data, perform data packaging processing on all the obtained IR image data to obtain an IR image data packet, and then send the IR image data packet to the slow branch network SSCNN for spatial calculation processing to obtain scene spatial information.

[0085] Step S34: Obtain the Depth image data from the (N - 1)L1 / L2 second IR-D image data, perform data packaging processing on all the obtained Depth image data to obtain a depth image data packet.

[0086] Step S35: Obtain the IR image data from the (N - 1)L1 / L2 second IR-D image data, perform data packaging processing on all the obtained IR image data to obtain a near-infrared image data packet.

[0087] See Figure 2 , the figure shows an application embodiment of the human action recognition method of the present invention, which is implemented based on TOF and includes the implementation of an optical flow extractor-two-stream classifier joint training architecture and a background separation joint optical flow calculation method, specifically including the steps:

[0088] Step 1: Obtain a video stream containing a depth image (Depth) and a near-infrared image (IR) from a TOF device module or a TOF video;

[0089] Step 2: Perform discrete sampling processing on the obtained video stream and cache the image data obtained from the discrete sampling processing. Specifically, the sampling step of the slow image collector is 16, that is, it samples once every 16 frames; the sampling step of the fast image collector is 2, that is, it samples once every 2 frames; the data collected each time is an IR-D data pair; then perform area mean and variance operations on the images (IR-D) collected by the image collector respectively, and store the processed image pairs into Slow-Stack and Fast-Stack respectively;

[0090] Step 3: Detect the cached image data. When the size of Slow-Stack is greater than or equal to 2 (the slow collector collects twice with a collection interval step of 16), take out two pairs of cached data (IR-D) from Slow-Stack, denoted as IRDS2. At this time, the size of Slow-Stack is reset to 0. When the size of Fast-Stack is greater than or equal to 8 (the collection step of the slow collector is 8 times that of the fast collector. Theoretically, when the slow collector has completed 2 collections, the fast collector has completed 8 collections), take out 8 pairs of (IR-D) data from Fast-Stack, denoted as IRDF8. At this time, the size of Fast-Stack is reset to 0. If the sizes of both data buffers (Slow-Stack / Fast-Stack) meet the determination conditions, pack the IR data in IRDS2 to form a data packet IRS2 and send it into the SSCNN network to calculate the spatial information, and denote the obtained spatial information representation as SSO. Pack the IR in IRDF8 to form a data packet IRF8. Similarly, pack the Depth in IRDF8 to form a data packet DF8. At this time, IRF8 and DF8 contain 8 IR images and 8 Depth images respectively.

[0091] Step 4: Calculate the background mask. Use the K-nearest neighbor algorithm to perform unsupervised clustering on each image in DF8 to obtain the background mask sequence MDF8. Multiply DF8 and IRF8 by the mask values at the corresponding position numbers in MDF8 respectively to obtain the depth map sequence and near-infrared sequence after removing the background, denoted as DMF8 and IRMF8 respectively.

[0092] Step 5: Pair IRMF8 and DMF8 pairwise in chronological order (for example: frame 0 is paired with frame 1, frame 1 and frame 2 are paired, etc.) to obtain the paired sequences IRP7 (IR-IR) and DP7 (D-D) (7 represents the number of pairs. After pairwise pairing of 8 images, there are 7 pairs of images). Finally, splice IRP7 and DP7 and send them into the OEU optical flow extractor to extract the optical flow information (the OEU optical flow extractor is essentially also a CNN network), and denote the obtained optical flow information as POF.

[0093] Step 6: Send POF into the FOFCNN network to obtain the spatio-temporal feature representation based on the optical flow, denoted as FOF.

[0094] Step 7: Splice FOF and SSO and send them into FusionMLP to calculate the final class score activation, and obtain the action recognition classification result of the current sampling batch according to the score activation.

[0095] The present invention utilizes an upper-layer task to supervise the training of a two-stream classifier and an optical flow extractor, incorporates the optical flow extractor into the training of the two-stream classifier, can obtain an optical flow extractor more adapted to the engineering scenario, and improves the engineering scenario adaptation performance.

[0096] The present invention uses a self-supervised clustering algorithm (K-nearest neighbor) to cluster the input to the optical flow extractor, obtains a background class mask, and further removes background elements in the image sequence, which can effectively avoid the disturbance of optical flow calculation errors caused by relative camera motion and background noise.

[0097] The input of the optical flow extractor of the present invention not only includes the 2D optical flow information of the IR near-infrared image, but also includes the 2D optical flow information from the depth distance. The 2D optical flow information of the depth distance can be understood as the movement trajectory of the human body on the z-axis in the three-dimensional space, while the 2D optical flow information of the IR near-infrared image can be understood as the movement trajectories of the human body on the x-axis and y-axis in the three-dimensional space. Compared with the 2D optical flow information based on RGB in the RGB-D scheme, more comprehensive spatio-temporal information of human motion can be obtained, improving the action recognition accuracy.

[0098] See Figure 3 , the figure shows a human action recognition device, including a video stream acquisition module 100, a discrete sampling processing module 200, an image detection processing module 300, an image background processing module 400, an optical flow information extraction module 500, a spatio-temporal feature representation module 600, and an action recognition classification processing module 700.

[0099] The video stream acquisition module 100 is used to acquire a video stream containing a depth image and a near-infrared image from a TOF device module or a TOF video.

[0100] The discrete sampling processing module 200 is used to perform discrete sampling processing on the acquired video stream and cache the image data obtained from the discrete sampling processing.

[0101] The image detection processing module 300 is used to detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet.

[0102] The image background processing module 400 is used to perform unsupervised clustering processing on the depth image data packet to obtain a background mask sequence, and perform background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed.

[0103] The optical flow information extraction module 500 is used to perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively, obtain the depth image pairing sequence and the near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information.

[0104] The spatio-temporal feature representation module 600 is used to perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information.

[0105] The action recognition and classification processing module 700 is used to splice the spatio-temporal feature representation information with the scene space information to calculate the category score activation result, and obtain the action recognition and classification result of the current sampling batch according to the category score activation result.

[0106] Each module in the human action recognition device of the present invention can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0107] The present invention also provides a computer device for implementing the above-mentioned human action recognition method. The computer device can be a server, and its internal structure diagram can be as Figure 4 shown. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as user information, record information, and files. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a human action recognition method as described above.

[0108] Those skilled in the art can understand that Figure 4 the structure shown in

[0109] is only a block diagram of a part of the structure related to the technical solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0110] Step S10, obtain a video stream containing a depth image and a near-infrared image from a TOF device module or a TOF video;

[0111] Step S20, perform discrete sampling processing on the obtained video stream, and cache the image data obtained from the discrete sampling processing;

[0112] Step S30, detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene spatial information, a depth image data packet, and a near-infrared image data packet;

[0113] Step S40, perform unsupervised clustering processing on the depth image data packet to obtain a background mask sequence, and perform background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed;

[0114] Step S50, perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence and then extract optical flow information;

[0115] Step S60, perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information;

[0116] Step S70, splice the spatio-temporal feature representation information and the scene spatial information and then calculate the category score activation result, and obtain the action recognition classification result of the current sampling batch according to the category score activation result.

[0117] The present invention also provides a computer-readable storage medium for implementing the above-mentioned human action recognition method, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0118] Step S10, obtain a video stream containing a depth image and a near-infrared image from a TOF device module or a TOF video;

[0119] Step S20, perform discrete sampling processing on the obtained video stream, and cache the image data obtained from the discrete sampling processing;

[0120] Step S30, detect the image data in the cache, and perform spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene spatial information, a depth image data packet, and a near-infrared image data packet;

[0121] Step S40: Perform unsupervised clustering processing on the depth image data packets to obtain a background mask sequence, and perform background removal processing on the depth image data packets and the near-infrared image data packets respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed;

[0122] Step S50: Perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information;

[0123] Step S60: Perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information;

[0124] Step S70: Splice the spatio-temporal feature representation information with the scene space information to calculate the category score activation result, and obtain the action recognition classification result of the current sampling batch according to the category score activation result.

[0125] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0126] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification is only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A human motion recognition method, characterized in that, Including: Obtaining a video stream containing a depth image and a near-infrared image from a TOF device module or a TOF video; Performing discrete sampling processing on the obtained video stream and caching the image data obtained from the discrete sampling processing; Detecting the image data in the cache, and performing spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet; Performing unsupervised clustering processing on the depth image data packet to obtain a background mask sequence, and respectively performing background removal processing on the depth image data packet and the near-infrared image data packet according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed; Performing chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splicing the depth image pairing sequence and the near-infrared image pairing sequence and then extracting optical flow information; Performing spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; And Splicing the spatio-temporal feature representation information and the scene space information and then calculating a class score activation result, and obtaining an action recognition classification result of the current sampling batch according to the class score activation result.

2. The human body motion recognition method according to claim 1, characterized in that, The performing discrete sampling processing on the obtained video stream and caching the image data obtained from the discrete sampling processing includes: Performing discrete sampling processing on the obtained video stream through a first image collector and a second image collector respectively, the sampling step of the first image collector is L1, the sampling step of the second image collector is L2, L1 > L2 and L1, L2 are integers; Respectively performing mean value and variance processing on the image data collected by the first image collector and the second image collector; and Respectively storing the processed image data into a first data buffer and a second data buffer for caching.

3. The human body motion recognition method according to claim 2, wherein, The detecting the image data in the cache, and performing spatial calculation processing and data packaging processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet includes: Judging whether the size in the first data buffer is greater than or equal to N, N ≥ 2 and is an integer, if the judgment is yes, taking out N first IR-D image data from the first data buffer and setting the size in the first data buffer to 0; Judging whether the size in the second data buffer is greater than or equal to (N - 1)L1 / L2, (N - 1)L1 / L2 is an integer, if the judgment is yes, taking out (N - 1)L1 / L2 second IR-D image data from the second data buffer and setting the size in the second data buffer to 0; Obtaining the IR image data in the N first IR-D image data, performing data packaging processing on all the obtained IR image data to obtain an IR image data packet, and then sending the IR image data packet to a slow branch network SSCNN for spatial calculation processing to obtain scene space information; Obtain the Depth image data from (N - 1)L1 / L2 second IR-D image data, and perform data packing processing on all the obtained Depth image data to obtain a depth image data packet; and Obtain the IR image data from (N - 1)L1 / L2 second IR-D image data, and perform data packing processing on all the obtained IR image data to obtain a near-infrared image data packet.

4. The human motion recognition method according to claim 1, characterized in that Use the K-nearest neighbor algorithm to perform unsupervised clustering processing on each depth image in the depth image data packet to obtain a background mask sequence.

5. The human body motion recognition method according to claim 1, characterized in that Perform background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence. Specifically, multiply the depth image data packet and the near-infrared image data packet by the mask value at the corresponding position serial number in the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed.

6. The human motion recognition method according to claim 1, wherein, Perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively. Specifically, perform pairwise chronological pairing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence.

7. The human body motion recognition method according to claim 1, wherein Extract optical flow information from the spliced depth image pairing sequence and near-infrared image pairing sequence through an optical flow extractor to obtain optical flow information.

8. The human body motion recognition method according to claim 1, characterized in that, Perform spatio-temporal feature representation processing on the optical flow information through a fast-branch optical flow network FOFCNN to obtain spatio-temporal feature representation information.

9. The human body motion recognition method according to claim 1, characterized in that, Perform category score activation calculation processing on the spliced spatio-temporal feature representation information and scene space information through a feature fusion network FusionMLP.

10. A human body motion recognition method and device, characterized in that, Include: A video stream acquisition module, which is used to acquire a video stream containing a depth image and a near-infrared image from a TOF device module or a TOF video; A discrete sampling processing module, which is used to perform discrete sampling processing on the acquired video stream and cache the image data obtained from the discrete sampling processing; An image detection processing module, which is used to detect the image data in the cache, and perform spatial calculation processing and data packing processing on the image data that meets the detection requirements to obtain scene space information, a depth image data packet, and a near-infrared image data packet; An image background processing module, which is used to perform unsupervised clustering processing on the depth image data packet to obtain a background mask sequence, and perform background removal processing on the depth image data packet and the near-infrared image data packet respectively according to the background mask sequence to obtain a depth image sequence and a near-infrared image sequence with the background removed; An optical flow information extraction module, which is used to perform chronological pairing processing on the depth image sequence and the near-infrared image sequence respectively to obtain a depth image pairing sequence and a near-infrared image pairing sequence, and splice the depth image pairing sequence and the near-infrared image pairing sequence to extract optical flow information; A spatio-temporal feature representation module, which is used to perform spatio-temporal feature representation processing on the optical flow information to obtain spatio-temporal feature representation information; and An action recognition and classification processing module, which is configured to splice the spatio-temporal feature representation information and the scene space information, calculate the category score activation result, and obtain the action recognition and classification result of the current sampling batch according to the category score activation result.

11. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the human action recognition method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps according to any one of claims 1 to 9 are implemented.

Images