Human behavior recognition method, system, device and storage medium

Abstract

This invention discloses a method, system, device, and storage medium for recognizing human behavior. The method includes: generating an image frame sequence based on acquired static and / or dynamic image information of the human behavior to be recognized; inputting the image frame sequence into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition to obtain the behavior category of the human being to be recognized. This invention achieves human behavior recognition based on optical flow feature extraction networks, lightweight networks, and temporal networks, mitigating the influence of adverse factors such as network depth, light intensity, external object occlusion, and shooting orientation, improving recognition speed and accuracy, and achieving better recognition results.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, system, device, and storage medium for recognizing human behavior. Background Technology

[0002] With the improvement of computing power and the development of artificial intelligence and computer vision technologies, human-computer interaction technologies based on human behavior recognition are widely used. For example, human behavior recognition technologies based on infrared feature capture, contour extraction and computer vision technologies can achieve good human behavior recognition functions and achieve a good human-computer interaction experience.

[0003] Human behavior recognition results are typically determined in two ways: first, based on manually designed human behavior feature extraction, which usually involves designing effective human behavior features for specific datasets to model human behavior; and second, based on deep convolutional neural network (CNN) human behavior feature extraction, which can autonomously learn deep features from large amounts of data to capture human behavior features and then recognize human behavior. However, both of these methods have drawbacks, including poor generalization ability (using manually designed human behavior features), high complexity (many calculations are required based on manually designed features, and the calculation process and steps are complicated), low accuracy (for example, the acquired features are too limited, and features are difficult to calculate under complex environmental conditions), slow recognition speed (the number of parameters in the deep neural network calculation process is complex, and while the accuracy is relatively improved, the calculation speed is sacrificed), and large space consumption (the network layers are too deep, resulting in an excessive number of parameters and thus occupying too much memory space).

[0004] In summary, under the current stage of technological research, the speed and accuracy of recognition are affected to varying degrees by factors such as the computational complexity and speed of deep neural networks, as well as external environmental factors (such as light intensity, external object obstruction, and shooting orientation), resulting in poor final recognition results. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of existing human behavior recognition methods, which are slow in recognition speed and low in recognition accuracy, resulting in poor recognition effect, and to provide a human behavior recognition method, system, device and storage medium.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution:

[0007] The first aspect of this invention provides a method for recognizing human behavior, the method comprising:

[0008] Acquire image information of the human behavior to be identified, wherein the image information includes static image information and / or dynamic image information;

[0009] Generate the image frame sequence of the human behavior to be identified based on the static image information and / or the dynamic image information;

[0010] The image frame sequence is input into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition to obtain the behavior category of the human body to be identified.

[0011] Preferably, the image information includes non-infrared image information and / or infrared image information, the non-infrared image information includes non-infrared static image information and / or non-infrared dynamic image information, and the infrared image information includes infrared static image information and / or infrared dynamic image information.

[0012] The step of generating the image frame sequence of the human behavior to be identified based on the static image information and / or the dynamic image information includes:

[0013] Generate an RGB (color system) image frame sequence of the human behavior to be identified based on the non-infrared static image information and / or the non-infrared dynamic image information;

[0014] Generate an infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or the infrared dynamic image information;

[0015] The step of inputting the image frame sequence into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for recognition to obtain the behavior category of the human body to be identified includes:

[0016] Human behavior recognition is performed on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network to obtain at least one recognition result of the human body to be identified;

[0017] The weighted MAX-K fusion algorithm is used to fuse the at least one recognition result to obtain the behavior category of the human body to be identified.

[0018] Preferably, the step of performing human behavior recognition on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network to obtain at least one recognition result for the human body to be identified includes:

[0019] The Fourier feature extraction algorithm is used to extract contour feature information from the RGB image frame sequence;

[0020] The contour feature information is input into the temporal network for human behavior recognition to obtain the posture feature recognition result of the human body to be recognized.

[0021] The RGB image frame sequence is input into the optical flow feature extraction network to obtain a motion image frame sequence that characterizes the running feature information of the human behavior to be identified.

[0022] The motion image frame sequence is input into the lightweight network and / or the temporal network to obtain the motion feature recognition result of the human body to be identified;

[0023] The RGB image frame sequence is input into the lightweight network and / or the temporal network to obtain the appearance feature recognition result of the human body to be identified;

[0024] The infrared image frame sequence is input into the lightweight network and / or the temporal network to obtain the infrared feature recognition result of the human body to be identified;

[0025] The weighted MAX-K fusion algorithm is used to fuse the motion feature recognition result, the appearance feature recognition result, and the infrared feature recognition result to obtain a first fused recognition result;

[0026] The weighted MAX-K fusion algorithm is used to fuse the posture feature recognition result and the first fusion recognition result to obtain the behavior category of the human body to be identified.

[0027] Preferably, the step of performing human behavior recognition on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network to obtain at least one recognition result for the human body to be identified further includes:

[0028] The motion feature recognition result and the appearance feature recognition result are fused using the weighted MAX-K fusion algorithm to obtain a second fused recognition result;

[0029] The weighted MAX-K fusion algorithm is used to fuse the posture feature recognition result and the second fusion recognition result to obtain the behavior category of the human body to be identified.

[0030] Preferably, the step of generating the RGB image frame sequence of the human behavior to be identified based on the non-infrared static image information and / or the non-infrared dynamic image information includes:

[0031] Extract the non-infrared static image information and / or the non-infrared dynamic image information's non-infrared image frames according to a preset transmission frame number;

[0032] The non-infrared image frames are subjected to data augmentation processing to obtain the RGB image frame sequence of the human behavior to be identified.

[0033] Preferably, the step of generating the infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or the infrared dynamic image information includes:

[0034] Infrared image frames of the infrared static image information and / or the infrared dynamic image information are extracted according to a preset number of transmission frames.

[0035] The infrared image frames are subjected to data augmentation processing to obtain the infrared image frame sequence of the human behavior to be identified.

[0036] Preferably, the step of inputting the RGB image frame sequence into the optical flow feature extraction network to obtain a motion image frame sequence characterizing the movement feature information of the human behavior to be identified includes:

[0037] The RGB image frame sequence is input into the optical flow feature extraction network at a preset interval of frames, and the network pruning algorithm is used to obtain a motion image frame sequence that represents the running feature information of the human behavior to be identified.

[0038] A second aspect of the present invention provides a human behavior recognition system, the recognition system comprising an acquisition module, a generation module, and a recognition module;

[0039] The acquisition module is used to acquire image information of the human behavior to be identified, and the image information includes static image information and / or dynamic image information;

[0040] The generation module is used to generate an image frame sequence of the human behavior to be identified based on the static image information and / or the dynamic image information.

[0041] The recognition module is used to input the image frame sequence into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition, so as to obtain the behavior category of the human body to be identified.

[0042] Preferably, the image information includes non-infrared image information and / or infrared image information, the non-infrared image information includes non-infrared static image information and / or non-infrared dynamic image information, and the infrared image information includes infrared static image information and / or infrared dynamic image information; the generation module includes a first generation unit and a second generation unit, and the recognition module includes a recognition unit and a fusion unit;

[0043] The first generation unit is used to generate the RGB image frame sequence of the human behavior to be identified based on the non-infrared static image information and / or the non-infrared dynamic image information;

[0044] The second generation unit is used to generate an infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or the infrared dynamic image information;

[0045] The recognition unit is used to perform human behavior recognition on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network, so as to obtain at least one recognition result of the human body to be identified;

[0046] The fusion unit is used to fuse the at least one recognition result using a weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

[0047] Preferably, the identification unit includes a third extraction subunit, a first identification subunit, an acquisition subunit, a second identification subunit, a third identification subunit, a fourth identification subunit, a first fusion subunit, and a second fusion subunit;

[0048] The third extraction subunit is used to extract contour feature information from the RGB image frame sequence using the Fourier feature extraction algorithm;

[0049] The first recognition subunit is used to input the contour feature information into the temporal network for human behavior recognition, so as to obtain the posture feature recognition result of the human body to be recognized;

[0050] The acquisition subunit is used to input the RGB image frame sequence into the optical flow feature extraction network to obtain a motion image frame sequence that characterizes the running feature information of the human behavior to be identified.

[0051] The second recognition subunit is used to input the motion image frame sequence into the lightweight network and / or the temporal network to obtain the motion feature recognition result of the human body to be identified;

[0052] The third recognition subunit is used to input the RGB image frame sequence into the lightweight network and / or the temporal network to obtain the appearance feature recognition result of the human body to be recognized;

[0053] The fourth recognition subunit is used to input the infrared image frame sequence into the lightweight network and / or the temporal network to obtain the infrared feature recognition result of the human body to be identified;

[0054] The first fusion subunit is used to fuse the motion feature recognition result, the appearance feature recognition result, and the infrared feature recognition result using the weighted MAX-K fusion algorithm to obtain a first fusion recognition result;

[0055] The second fusion subunit is used to fuse the posture feature recognition result and the first fusion recognition result using the weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

[0056] Preferably, the identification unit further includes a third fusion subunit and a fourth fusion subunit;

[0057] The third fusion subunit is used to fuse the motion feature recognition result and the appearance feature recognition result using the weighted MAX-K fusion algorithm to obtain a second fusion recognition result;

[0058] The fourth fusion subunit is used to fuse the posture feature recognition result and the second fusion recognition result using the weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

[0059] Preferably, the first generation unit includes a first extraction subunit and a first processing subunit;

[0060] The first extraction subunit is used to extract the non-infrared static image information and / or the non-infrared dynamic image information's non-infrared image frames according to a preset number of transmission frames;

[0061] The first processing subunit is used to perform data enhancement processing on the non-infrared image frames to obtain the RGB image frame sequence of the human behavior to be identified.

[0062] Preferably, the second generation unit includes a second extraction subunit and a second processing subunit;

[0063] The second extraction subunit is used to extract infrared image frames of the infrared static image information and / or the infrared dynamic image information according to a preset number of transmission frames;

[0064] The second processing subunit is used to perform data enhancement processing on the infrared image frames to obtain the infrared image frame sequence of the human behavior to be identified.

[0065] Preferably, the acquisition subunit is used to input the RGB image frame sequence into the optical flow feature extraction network at a preset interval of frames, and use a network pruning algorithm to obtain a motion image frame sequence that represents the motion characteristics of the human behavior to be identified.

[0066] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the human behavior recognition method as described in the first aspect.

[0067] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human behavior recognition method as described in the first aspect.

[0068] The positive and progressive effects of this invention are as follows:

[0069] This invention generates an image frame sequence of the human behavior to be identified based on the acquired static and / or dynamic image information. The image frame sequence is then input into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition to determine the behavior category of the human being to be identified. This invention achieves human behavior recognition based on optical flow feature extraction networks, lightweight networks, and temporal networks, mitigating the impact of adverse factors such as network depth, light intensity, external occlusion, and shooting orientation, thereby improving recognition speed and accuracy and achieving better recognition results. Attached Figure Description

[0070] Figure 1 This is a first flowchart of the human behavior recognition method according to Embodiment 1 of the present invention.

[0071] Figure 2 This is a second flowchart of the human behavior recognition method according to Embodiment 1 of the present invention.

[0072] Figure 3 This is a schematic diagram of the human behavior recognition system according to Embodiment 2 of the present invention.

[0073] Figure 4 This is a schematic diagram of the structure of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0074] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0075] Example 1

[0076] This embodiment provides a method for recognizing human behavior. This method can be used on a terminal device, which can be either moving or stationary. Terminal devices include, but are not limited to, smartphones, smart cameras, tablets, interactive gaming platforms, and devices running Linux (an operating system), Android (a mobile operating system), or other operating systems. Figure 1 As shown, the identification method includes:

[0077] Step 101: Obtain image information of the human behavior to be identified, including static image information and / or dynamic image information;

[0078] In this embodiment, image information of the human behavior to be identified is acquired through a camera device, that is, static image information and / or dynamic image information of the human behavior to be identified is collected through the camera device. In addition, the facial expressions, gestures, body postures, and information such as ears, nose, eyes, and mouth of the human being to be identified can be acquired through the static image information and / or dynamic image information.

[0079] It should be noted that the camera device can be a regular camera or an infrared camera; still image information can be picture information (i.e., single-frame image information), and moving image information can be video information (i.e., multi-frame image information).

[0080] Step 102: Generate an image frame sequence of the human behavior to be identified based on static image information and / or dynamic image information;

[0081] In this embodiment, an image frame sequence representing the human behavior to be identified is generated based on the acquired static image information and / or dynamic image information.

[0082] Step 103: Input the image frame sequence into the optical flow feature extraction network and / or lightweight network and / or temporal network for human behavior recognition to obtain the behavior category of the human body to be identified.

[0083] In this embodiment, the behavior categories include, but are not limited to, raising and swinging arms, kicking and straddling, lying down, jumping, cycling, standing, squatting, and walking.

[0084] In this embodiment, the optical flow extraction network can use an improved network, SpyNet or FlowNet (i.e., adding a low-rank decomposition algorithm or network pruning algorithm to SpyNet or FlowNet to reduce parameters, so as to improve the extraction speed and accuracy of the optical flow extraction network); the lightweight network uses MobileNet; and the temporal network uses LSTM (Long Short Term Memory Network).

[0085] In one feasible solution, the image information includes non-infrared image information and / or infrared image information; the non-infrared image information includes non-infrared static image information and / or non-infrared dynamic image information; and the infrared image information includes infrared static image information and / or infrared dynamic image information. Figure 2 As shown, step 102 in this identification method includes:

[0086] Step 1021: Generate an RGB image frame sequence of the human behavior to be identified based on non-infrared static image information and / or non-infrared dynamic image information;

[0087] In this embodiment, a regular camera (e.g., a light-sensing camera) is used to capture non-infrared static image information and / or non-infrared dynamic image information of the human body to be identified. It should be noted that the non-infrared static image information and / or non-infrared dynamic image information are ordinary picture information and / or ordinary video information.

[0088] Generate a sequence of RGB image frames representing the appearance features of the human body to be identified based on ordinary image information and / or ordinary video information.

[0089] Step 1022: Generate an infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or infrared dynamic image information;

[0090] In this embodiment, an infrared camera (e.g., an infrared detection camera or an infrared sensor) or a camera with infrared imaging function is used to obtain infrared static image information and / or infrared dynamic image information of the human body to be identified. It should be noted that the infrared static image information and / or infrared dynamic image information are infrared image information and / or infrared video information.

[0091] Generate a sequence of infrared image frames that represent thermal information of human behavior to be identified, based on infrared image information and / or infrared video information.

[0092] Step 103 includes:

[0093] Step 1031: Use Fourier feature extraction algorithm and / or optical flow feature extraction network and / or lightweight network and / or temporal network to perform human behavior recognition on image frame sequence and / or infrared image frame sequence to obtain at least one recognition result of the human body to be identified;

[0094] In this embodiment, the Fourier feature extraction algorithm is a statistical analysis method. The model has time series analysis capabilities. By obtaining binary image sequences of various human behaviors, Fourier features with rotation, translation and scale invariance are extracted from the sequence images to be identified. The extracted feature vectors are classified using a temporal network based on the Fourier descriptor based on the center distance to obtain the recognition results of human behaviors.

[0095] Step 1032: Use the weighted MAX-K fusion algorithm to fuse at least one recognition result to obtain the behavior category of the human body to be identified.

[0096] In a feasible solution, such as Figure 2 As shown, step 1031 in this identification method includes:

[0097] Step 1031-1: Extract contour feature information from the RGB image frame sequence using the Fourier feature extraction algorithm;

[0098] Step 1031-2: Input the contour feature information into a temporal network for human behavior recognition to obtain the posture feature recognition result of the human body to be recognized;

[0099] In this embodiment, the RGB image frame sequence is substituted frame by frame into the Fourier feature extraction algorithm to extract the human body contour feature information in the RGB image frame sequence, and the extracted human body contour feature information is substituted into the temporal network to perform human behavior recognition in the RGB image frame sequence, so as to obtain the posture feature recognition result of the human body to be identified.

[0100] It should be noted that the temporal network in step 1031-2 is a type of recurrent neural network used to capture long-term dependencies, mainly used to capture features of human behavior in long-term video sequences.

[0101] Step 1031-3: Input the RGB image frame sequence into the optical flow feature extraction network to obtain a motion image frame sequence that represents the running feature information of the human behavior to be identified;

[0102] In one feasible scheme, step 1031-3 includes: inputting the RGB image frame sequence into the optical flow feature extraction network at a preset interval of frames, and using a network pruning algorithm to obtain a motion image frame sequence that represents the running feature information of the human behavior to be identified.

[0103] In the specific implementation process, taking a preset interval of 30 frames as an example, the input RGB image frame sequence is input into the optical flow feature extraction network at 30 frame intervals, and a network pruning algorithm is used in the optical flow feature extraction network to extract the motion image frame sequence (i.e., optical flow image frame sequence) that represents the motion feature information of human behavior.

[0104] In one feasible scheme, the optical flow feature extraction network used in step 1031-3 can be divided into a two-layer optical flow feature extraction network. The upper optical flow feature extraction network is used to extract feature information under small displacement motion, and the lower optical flow feature extraction network is used to extract feature information under large displacement motion. The feature information extracted by the two-layer optical flow feature extraction network is then fused to obtain the required motion image frame sequence (i.e., optical flow image frame sequence).

[0105] Then, the network pruning algorithm in step 1031-3 is added to the two-layer optical flow feature extraction network. The network pruning algorithm is used to remove redundant parameters of the two-layer optical flow feature extraction network during the optical flow feature extraction process, thereby reducing the memory occupied by the two-layer optical flow feature extraction network and improving the speed of optical flow image frame sequence extraction.

[0106] It should be noted that the optical flow feature extraction network is used to convert the initially extracted RGB image frame sequence representing spatial domain feature information into a motion image frame sequence representing temporal domain feature information (i.e., optical flow image frame sequence).

[0107] Step 1031-4: Input the motion image frame sequence into a lightweight network and / or a temporal network to obtain the motion feature recognition result of the human body to be identified;

[0108] In this embodiment, the motion image frame sequence is substituted only into a lightweight network to obtain the motion feature recognition result of the human body to be identified; or the motion image frame sequence is substituted only into a temporal network to obtain the motion feature recognition result of the human body to be identified; or the motion image frame sequence is substituted into both a lightweight network and a temporal network to obtain the motion feature recognition result of the human body to be identified.

[0109] It should be noted that simultaneously feeding the motion image frame sequence into both the lightweight network and the temporal network can yield more accurate motion feature recognition results for the human body to be identified.

[0110] Step 1031-5: Input the RGB image frame sequence into a lightweight network and / or a temporal network to obtain the appearance feature recognition result of the human body to be identified;

[0111] In this embodiment, the generated RGB image frame sequence used to characterize the appearance features of the human behavior to be identified is substituted into a lightweight network, or into a temporal network (i.e., the temporal network is a long short-term memory neural network), or into both a lightweight network and a temporal network, to obtain the appearance feature recognition result characterizing the human behavior to be identified.

[0112] Step 1031-6: Input the infrared image frame sequence into a lightweight network and / or a temporal network to obtain the infrared feature recognition result of the human body to be identified;

[0113] In this embodiment, the generated infrared image frame sequence used to characterize the thermal information of the human behavior to be identified is substituted into a lightweight network, or into a temporal network (i.e., the temporal network is a long short-term memory neural network), or into both a lightweight network and a temporal network, to obtain the infrared feature recognition result of the human body to be identified.

[0114] Step 1031-7: Use the weighted MAX-K fusion algorithm to fuse the motion feature recognition result, appearance feature recognition result and infrared feature recognition result to obtain the first fused recognition result;

[0115] Step 1031-8: Use the weighted MAX-K fusion algorithm to fuse the pose feature recognition result and the first fusion recognition result to obtain the behavior category of the human body to be identified.

[0116] It should be noted that the temporal networks in steps 1031-4, 1031-5, and 1031-6 above are all long short-term memory neural networks.

[0117] The lightweight networks in steps 1031-4, 1031-5, and 1031-6 can have the same or different network structures, which is not specifically limited here. In addition, the lightweight networks in steps 1031-5 and 1031-6 mainly extract features from the image frame sequence through convolutional and pooling layers in the network, and substitute them into the temporal network to obtain the temporal information in the image frame sequence. The recognition results (i.e., motion feature information, appearance feature information, and infrared feature information) are obtained in the temporal network.

[0118] In one feasible embodiment, step 1031 further includes:

[0119] Step 1031-11: Use the weighted MAX-K fusion algorithm to fuse the motion feature recognition results and the appearance feature recognition results to obtain the second fused recognition result;

[0120] Steps 1031-12: Use the weighted MAX-K fusion algorithm to fuse the pose feature recognition results and the second fusion recognition results to obtain the behavior category of the human body to be identified.

[0121] In this embodiment, during the process of obtaining the first or second fused recognition result using the weighted MAX-K fusion algorithm, the weights can be adjusted if limited by internal or external conditions. For example, if infrared detection images cannot be detected, the weight of the infrared detection recognition result can be set to 0, while the remaining recognition results still use the weighted MAX-K fusion algorithm, thereby improving the generalization of the real-time process.

[0122] It should be noted that, in the specific implementation process, when the acquired image information is only non-infrared image information (i.e., only ordinary image information of the human body to be identified is acquired through an ordinary camera), the posture feature recognition result obtained in step 1031-2 can be directly used as the behavior category of the human body to be identified, or the motion feature recognition result obtained in step 1031-4 can be directly used as the behavior category of the human body to be identified, or the appearance feature recognition result obtained in step 1031-5 can be directly used as the behavior category of the human body to be identified, or the recognition result obtained by fusing the motion feature recognition result obtained in step 1031-4 and the appearance feature recognition result obtained in step 1031-5 can be fused again with the posture feature recognition result obtained in step 1031-2 and the result obtained can be used as the behavior category of the human body to be identified.

[0123] When the acquired image information is only infrared image information (i.e., only the infrared image information of the human body to be identified is acquired through an infrared camera), the infrared feature recognition result obtained in step 1031-6 can be directly used as the behavior category of the human body to be identified.

[0124] When the acquired image information includes both infrared image information and infrared image information (i.e., acquiring ordinary image information of the human body to be identified through an ordinary camera and acquiring infrared image information of the human body to be identified through an infrared camera), the fusion result obtained by fusing at least one of the posture features obtained in step 1031-2, the motion feature recognition result obtained in step 1031-4, and the appearance feature recognition result obtained in step 1031-5 with the infrared feature recognition result obtained in step 1031-6 can be used as the behavior category of the human body to be identified.

[0125] In this embodiment, the motion feature recognition results obtained in step 1031-4, the appearance feature recognition results obtained in step 1031-5, and the infrared feature recognition results obtained in step 1031-6 are fused using a weighted MAX-K fusion algorithm according to the ratio of weight K and 1-K to obtain a first fusion recognition result. Alternatively, the motion feature recognition results obtained in step 1031-4 and the appearance feature recognition results obtained in step 1031-5 are fused using a weighted MAX-K fusion algorithm according to the ratio of weight K and 1-K to obtain a second fusion recognition result. Then, the first fusion recognition result or the second fusion recognition result is fused again with the posture features obtained in step 1031-2 using a weighted MAX-K fusion algorithm according to the ratio of weight K and 1-K to obtain the behavior category of the human body to be identified.

[0126] For example, in a specific implementation, firstly, the contour feature information in the RGB image frame sequence of the human image information to be identified is obtained through the Fourier feature extraction algorithm, and the contour feature information is substituted into a temporal network to obtain the pose feature recognition result of the human body to be identified; then, the optical flow feature extraction network is used to convert the initially extracted RGB image frame sequence representing spatial domain feature information into a motion image frame sequence representing temporal domain feature information (i.e., optical flow image frame sequence), and the motion image frame sequence is substituted into a lightweight network and / or a temporal network to obtain the motion feature recognition result representing human behavior recognition. Finally, the RGB image frame sequence is substituted into a lightweight network and / or a temporal network to obtain the appearance feature recognition result representing the human behavior to be identified. The results include infrared image frame sequences obtained through an infrared probe; the infrared image frame sequences are then substituted into a lightweight network and / or a temporal network to obtain infrared feature recognition results characterizing human behavior; the obtained motion feature recognition results, appearance feature recognition results, and / or infrared feature recognition results of human behavior recognition are then weighted and fused using MAX-K to obtain a first fusion recognition result or a second fusion recognition result; finally, the posture feature recognition result obtained through the Fourier feature extraction algorithm is substituted into the first fusion recognition result or the second fusion recognition result obtained through the lightweight network and / or the weighted MAX-K fusion algorithm to accurately determine the behavior category of the human behavior to be recognized.

[0127] This embodiment uses contour feature extraction based on Fourier feature extraction algorithm, infrared feature extraction, feature extraction based on lightweight network and / or temporal network, and feature fusion algorithm to identify human behavior. It solves the problem of low accuracy and speed in existing human behavior recognition methods. Compared with existing human behavior recognition methods, it has higher recognition speed, recognition accuracy and generalization.

[0128] In one feasible embodiment, step 1021 includes:

[0129] Step 1021-1: Extract non-infrared image frames containing non-infrared static image information and / or non-infrared dynamic image information according to the preset transmission frame number;

[0130] Step 10212: Perform data augmentation processing on the non-infrared image frames to obtain an RGB image frame sequence of the human behavior to be identified.

[0131] In this embodiment, the preset number of transmission frames is set according to the actual situation. For example, the preset number of transmission frames can be set to 25fps, or it can be set to other values. No specific limitation is made here.

[0132] In the specific implementation process, taking a preset transmission frame rate of 25fps as an example, a regular light sensor camera is used to capture non-infrared static image information and / or non-infrared dynamic image information. The input regular non-infrared static image information and / or non-infrared dynamic image information data stream is extracted into non-infrared image frames at 25fps, and the extracted non-infrared image frames are data-enhanced to obtain the required RGB image frame sequence that characterizes the appearance features of the human body to be identified.

[0133] In one feasible embodiment, step 1022 includes:

[0134] Step 1022-1: Extract infrared image frames containing infrared static image information and / or infrared dynamic image information according to the preset number of transmission frames;

[0135] Step 1022-2: Perform data enhancement processing on the infrared image frames to obtain an infrared image frame sequence of the human behavior to be identified.

[0136] In this embodiment, the preset number of transmission frames is set according to the actual situation. For example, the preset number of transmission frames can be set to 25fps, or it can be set to other values. No specific limitation is made here.

[0137] In the specific implementation process, taking a preset transmission frame rate of 25fps as an example, an infrared detection camera is used to capture infrared static image information and / or infrared dynamic image information. The input infrared static image information and / or infrared dynamic image information data stream is extracted into infrared image frames at 25bps, and the extracted infrared image frames are data-enhanced to obtain the required infrared image frame sequence that represents the thermal information of the human body to be identified.

[0138] This embodiment generates an image frame sequence of the human behavior to be identified based on the acquired static and / or dynamic image information. The image frame sequence is then input into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition to determine the behavior category of the human being to be identified. This implementation achieves human behavior recognition based on an optical flow feature extraction network, a lightweight network, and a temporal network, mitigating the impact of adverse factors such as network depth, light intensity, external occlusion, and shooting orientation, thereby improving recognition speed and accuracy and achieving better recognition results.

[0139] Example 2

[0140] This embodiment provides a human behavior recognition system that can be used in terminal devices, which can be mobile or stationary. Terminal devices include, but are not limited to, smartphones, smart cameras, tablets, interactive gaming platforms, and devices running Linux, Android, or other operating systems. Figure 3As shown, the identification system includes an acquisition module 21, a generation module 22, and an identification module 23;

[0141] The acquisition module 21 is used to acquire image information of the human behavior to be identified, including static image information and / or dynamic image information;

[0142] In this embodiment, image information of the human behavior to be identified is acquired through a camera device, that is, static image information and / or dynamic image information of the human behavior to be identified is collected through the camera device. In addition, the facial expressions, gestures, body postures, and information such as ears, nose, eyes, and mouth of the human being to be identified can be acquired through the static image information and / or dynamic image information.

[0143] It should be noted that the camera device can be a regular camera or an infrared camera; still image information can be picture information (i.e., single-frame image information), and moving image information can be video information (i.e., multi-frame image information).

[0144] The generation module 22 is used to generate a sequence of image frames of human behavior to be identified based on static image information and / or dynamic image information;

[0145] In this embodiment, an image frame sequence representing the human behavior to be identified is generated based on the acquired static image information and / or dynamic image information.

[0146] The recognition module 23 is used to input the image frame sequence into the optical flow feature extraction network and / or the lightweight network and / or the temporal network for human behavior recognition, so as to obtain the behavior category of the human body to be recognized.

[0147] In this embodiment, the behavior categories include, but are not limited to, raising and swinging arms, kicking and straddling, lying down, jumping, cycling, standing, squatting, and walking.

[0148] In this embodiment, the optical flow extraction network can use an improved network, SpyNet or FlowNet (i.e., adding a low-rank decomposition algorithm or network pruning algorithm to SpyNet or FlowNet to reduce parameters, so as to improve the extraction speed and accuracy of the optical flow extraction network); the lightweight network uses MobileNet; and the temporal network uses LSTM (Long Short Term Memory Network).

[0149] In one feasible scheme, the image information includes non-infrared image information and / or infrared image information, the non-infrared image information includes non-infrared static image information and / or non-infrared dynamic image information, and the infrared image information includes infrared static image information and / or infrared dynamic image information.

[0150] like Figure 3As shown, the generation module 22 includes a first generation unit 221 and a second generation unit 222, and the recognition module 23 includes a recognition unit 231 and a fusion unit 232.

[0151] The first generation unit 221 is used to generate an RGB image frame sequence of human behavior to be identified based on non-infrared static image information and / or non-infrared dynamic image information.

[0152] In this embodiment, a regular camera (e.g., a light-sensing camera) is used to capture non-infrared static image information and / or non-infrared dynamic image information of the human body to be identified. It should be noted that the non-infrared static image information and / or non-infrared dynamic image information are ordinary picture information and / or ordinary video information.

[0153] Generate a sequence of RGB image frames representing the appearance features of the human body to be identified based on ordinary image information and / or ordinary video information.

[0154] The second generation unit 222 is used to generate an infrared image frame sequence of human behavior to be identified based on infrared static image information and / or infrared dynamic image information.

[0155] In this embodiment, an infrared camera (e.g., an infrared detection camera or an infrared sensor) or a camera with infrared imaging function is used to obtain infrared static image information and / or infrared dynamic image information of the human body to be identified. It should be noted that the infrared static image information and / or infrared dynamic image information are infrared image information and / or infrared video information.

[0156] Generate a sequence of infrared image frames that represent thermal information of human behavior to be identified, based on infrared image information and / or infrared video information.

[0157] The recognition unit 231 is used to perform human behavior recognition on image frame sequences and / or infrared image frame sequences using Fourier feature extraction algorithm and / or optical flow feature extraction network and / or lightweight network and / or temporal network, so as to obtain at least one recognition result of the human body to be identified.

[0158] In this embodiment, the Fourier feature extraction algorithm is a statistical analysis method. The model has time series analysis capabilities. By obtaining binary image sequences of various human behaviors, Fourier features with rotation, translation and scale invariance are extracted from the sequence images to be identified. The extracted feature vectors are classified using a temporal network based on the Fourier descriptor based on the center distance to obtain the recognition results of human behaviors.

[0159] The fusion unit 232 is used to fuse at least one recognition result using a weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

[0160] In a feasible solution, such as Figure 3 As shown, the identification unit 231 includes a third extraction subunit 2311, a first identification subunit 2312, an acquisition subunit 2313, a second identification subunit 2314, a third identification subunit 2315, a fourth identification subunit 2316, a first fusion subunit 2317, and a second fusion subunit 2318.

[0161] The third extraction subunit 2311 is used to extract contour feature information from the RGB image frame sequence using the Fourier feature extraction algorithm;

[0162] The first recognition subunit 2312 is used to input contour feature information into a temporal network for human behavior recognition, so as to obtain the posture feature recognition result of the human body to be recognized.

[0163] In this embodiment, the RGB image frame sequence is substituted frame by frame into the Fourier feature extraction algorithm to extract the human body contour feature information in the RGB image frame sequence, and the extracted human body contour feature information is substituted into the temporal network to perform human behavior recognition in the RGB image frame sequence, so as to obtain the posture feature recognition result of the human body to be identified.

[0164] It should be noted that the aforementioned temporal network is a type of recurrent neural network used to capture long-term dependencies, primarily for feature capture of human behavior in long-term video sequences.

[0165] The acquisition subunit 2313 is used to input the RGB image frame sequence into the optical flow feature extraction network to obtain a motion image frame sequence that represents the running feature information of the human behavior to be identified.

[0166] In one feasible scheme, the acquisition subunit 2313 is used to input the RGB image frame sequence into the optical flow feature extraction network according to a preset interval of frames, and use the network pruning algorithm to obtain the motion image frame sequence that represents the running feature information of the human behavior to be identified.

[0167] In the specific implementation process, taking a preset interval of 30 frames as an example, the acquisition subunit 2313 is used to input the input RGB image frame sequence into the optical flow feature extraction network in a 30-frame interval manner, and to use the network pruning algorithm in the optical flow feature extraction network to extract the motion image frame sequence (i.e., optical flow image frame sequence) that represents the motion feature information of human behavior.

[0168] In one feasible scheme, the optical flow feature extraction network can be divided into a two-layer optical flow feature extraction network. The upper optical flow feature extraction network is used to extract feature information under small displacement motion, and the lower optical flow feature extraction network is used to extract feature information under large displacement motion. The feature information extracted by the two-layer optical flow feature extraction network is then fused into the image to obtain the required motion image frame sequence (i.e., optical flow image frame sequence).

[0169] Then, the network pruning algorithm is added to the two-layer optical flow feature extraction network to remove redundant parameters of the two-layer optical flow feature extraction network during the optical flow feature extraction process, thereby reducing the memory occupied by the two-layer optical flow feature extraction network and improving the speed of optical flow image frame sequence extraction.

[0170] It should be noted that the optical flow feature extraction network is used to convert the initially extracted RGB image frame sequence representing spatial domain feature information into a motion image frame sequence representing temporal domain feature information (i.e., optical flow image frame sequence).

[0171] The second recognition subunit 2314 is used to input the motion image frame sequence into a lightweight network and / or a temporal network to obtain the motion feature recognition result of the human body to be recognized.

[0172] In this embodiment, the motion image frame sequence is substituted only into a lightweight network to obtain the motion feature recognition result of the human body to be identified; or the motion image frame sequence is substituted only into a temporal network to obtain the motion feature recognition result of the human body to be identified; or the motion image frame sequence is substituted into both a lightweight network and a temporal network to obtain the motion feature recognition result of the human body to be identified.

[0173] It should be noted that simultaneously feeding the motion image frame sequence into both the lightweight network and the temporal network can yield more accurate motion feature recognition results for the human body to be identified.

[0174] The third recognition subunit 2315 is used to input the RGB image frame sequence into a lightweight network and / or a temporal network to obtain the appearance feature recognition result of the human body to be recognized.

[0175] In this embodiment, the generated RGB image frame sequence used to characterize the appearance features of the human behavior to be identified is substituted into a lightweight network, or into a temporal network (i.e., the temporal network is a long short-term memory neural network), or into both a lightweight network and a temporal network, to obtain the appearance feature recognition result characterizing the human behavior to be identified.

[0176] The fourth recognition subunit 2316 is used to input the infrared image frame sequence into a lightweight network and / or a temporal network to obtain the infrared feature recognition result of the human body to be identified.

[0177] In this embodiment, the generated infrared image frame sequence used to characterize the thermal information of the human behavior to be identified is substituted into a lightweight network, or into a temporal network (i.e., the temporal network is a long short-term memory neural network), or into both a lightweight network and a temporal network, to obtain the infrared feature recognition result of the human body to be identified.

[0178] The first fusion subunit 2317 is used to fuse the motion feature recognition result, appearance feature recognition result and infrared feature recognition result using the weighted MAX-K fusion algorithm to obtain the first fusion recognition result;

[0179] The second fusion subunit 2318 is used to fuse the posture feature recognition result and the first fusion recognition result using the weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

[0180] It should be noted that the temporal networks in the second recognition subunit 2314, the third recognition subunit 2315, and the fourth recognition subunit 2316 are all long short-term memory neural networks.

[0181] The lightweight networks in the second recognition subunit 2314, the third recognition subunit 2315, and the fourth recognition subunit 2316 can have the same network structure or different network structures, which is not specifically limited here. In addition, the lightweight networks in the third recognition subunit 2315 and the fourth recognition subunit 2316 mainly extract features from the image frame sequence through convolutional layers and pooling layers in the network, and substitute them into the temporal network to obtain the temporal information in the image frame sequence, and obtain the recognition results under various types of information (i.e., motion feature information, appearance feature information, and infrared feature information) in the temporal network.

[0182] In a feasible solution, such as Figure 3 As shown, the identification unit 231 also includes a third fusion subunit 2319 and a fourth fusion subunit 2310;

[0183] The third fusion subunit 2319 is used to fuse the motion feature recognition result and the appearance feature recognition result using the weighted MAX-K fusion algorithm to obtain the second fusion recognition result;

[0184] The fourth fusion subunit 2310 is used to fuse the posture feature recognition result and the second fusion recognition result using the weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

[0185] In this embodiment, during the process of obtaining the first or second fused recognition result using the weighted MAX-K fusion algorithm, the weights can be adjusted if limited by internal or external conditions. For example, if infrared detection images cannot be detected, the weight of the infrared detection recognition result can be set to 0, while the remaining recognition results still use the weighted MAX-K fusion algorithm, thereby improving the generalization of the real-time process.

[0186] It should be noted that in the specific implementation process, when the acquired image information is only non-infrared image information (i.e., only ordinary image information of the human body to be identified is acquired through ordinary cameras), the posture feature recognition result can be directly used as the behavior category of the human body to be identified, the motion feature recognition result can be directly used as the behavior category of the human body to be identified, the appearance feature recognition result can be directly used as the behavior category of the human body to be identified, or the recognition result obtained by fusing the motion feature recognition result and the appearance feature recognition result can be fused with the posture feature recognition result again to obtain the behavior category of the human body to be identified.

[0187] When the acquired image information is only infrared image information (i.e., only the infrared image information of the human body to be identified is acquired through an infrared camera), the infrared feature recognition result can be directly used as the behavior category of the human body to be identified.

[0188] When the acquired image information includes both infrared image information and infrared image information (i.e., acquiring ordinary image information of the human body to be identified through an ordinary camera and acquiring infrared image information of the human body to be identified through an infrared camera), the fusion result obtained by fusing at least one of the posture features, motion features, and appearance features with the infrared features can be used as the behavior category of the human body to be identified.

[0189] In this embodiment, the motion feature recognition result, appearance feature recognition result, and infrared feature recognition result are fused using a weighted MAX-K fusion algorithm according to the weights K and 1-K to obtain a first fused recognition result. Alternatively, the motion feature recognition result and appearance feature recognition result are fused using a weighted MAX-K fusion algorithm according to the weights K and 1-K to obtain a second fused recognition result. Then, the first fused recognition result or the second fused recognition result is fused with the posture feature again using a weighted MAX-K fusion algorithm according to the weights K and 1-K to obtain the behavior category of the human body to be identified.

[0190] For example, in a specific implementation, firstly, the contour feature information in the RGB image frame sequence of the human image information to be identified is obtained through the Fourier feature extraction algorithm, and the contour feature information is substituted into a temporal network to obtain the pose feature recognition result of the human body to be identified; then, the optical flow feature extraction network is used to convert the initially extracted RGB image frame sequence representing spatial domain feature information into a motion image frame sequence representing temporal domain feature information (i.e., optical flow image frame sequence), and the motion image frame sequence is substituted into a lightweight network and / or a temporal network to obtain the motion feature recognition result representing human behavior recognition. Finally, the RGB image frame sequence is substituted into a lightweight network and / or a temporal network to obtain the appearance feature recognition result representing the human behavior to be identified. The results include infrared image frame sequences obtained through an infrared probe; the infrared image frame sequences are then substituted into a lightweight network and / or a temporal network to obtain infrared feature recognition results characterizing human behavior; the obtained motion feature recognition results, appearance feature recognition results, and / or infrared feature recognition results of human behavior recognition are then weighted and fused using MAX-K to obtain a first fusion recognition result or a second fusion recognition result; finally, the posture feature recognition result obtained through the Fourier feature extraction algorithm is substituted into the first fusion recognition result or the second fusion recognition result obtained through the lightweight network and / or the weighted MAX-K fusion algorithm to accurately determine the behavior category of the human behavior to be recognized.

[0191] This embodiment uses contour feature extraction based on Fourier feature extraction algorithm, infrared feature extraction, feature extraction based on lightweight network and / or temporal network, and feature fusion algorithm to identify human behavior. It solves the problem of low accuracy and speed in existing human behavior recognition methods. Compared with existing human behavior recognition methods, it has higher recognition speed, recognition accuracy and generalization.

[0192] In a feasible solution, such as Figure 3 As shown, the first generation unit 221 includes a first extraction subunit 2211 and a first processing subunit 2212;

[0193] The first extraction subunit 2211 is used to extract non-infrared image frames of non-infrared static image information and / or non-infrared dynamic image information according to a preset number of transmission frames.

[0194] The first processing subunit 2212 is used to perform data enhancement processing on non-infrared image frames to obtain an RGB image frame sequence of human behavior to be identified.

[0195] In this embodiment, the preset number of transmission frames is set according to the actual situation. For example, the preset number of transmission frames can be set to 25fps, or it can be set to other values. No specific limitation is made here.

[0196] In the specific implementation process, taking a preset transmission frame rate of 25fps as an example, a regular light sensor camera is used to capture non-infrared static image information and / or non-infrared dynamic image information. The input regular non-infrared static image information and / or non-infrared dynamic image information data stream is extracted into non-infrared image frames at 25fps, and the extracted non-infrared image frames are data-enhanced to obtain the required RGB image frame sequence that characterizes the appearance features of the human body to be identified.

[0197] In a feasible solution, such as Figure 3 As shown, the second generation unit 222 includes a second extraction subunit 2221 and a second processing subunit 2222;

[0198] The second extraction subunit 2221 is used to extract infrared image frames containing infrared static image information and / or infrared dynamic image information according to a preset number of transmission frames.

[0199] The second processing subunit 2222 is used to perform data enhancement processing on the infrared image frames to obtain an infrared image frame sequence of human behavior to be identified.

[0200] In this embodiment, the preset number of transmission frames is set according to the actual situation. For example, the preset number of transmission frames can be set to 25fps, or it can be set to other values. No specific limitation is made here.

[0201] In the specific implementation process, taking a preset transmission frame rate of 25fps as an example, an infrared detection camera is used to capture infrared static image information and / or infrared dynamic image information. The input infrared static image information and / or infrared dynamic image information data stream is extracted into infrared image frames at 25bps, and the extracted infrared image frames are data-enhanced to obtain the required infrared image frame sequence that represents the thermal information of the human body to be identified.

[0202] This embodiment generates an image frame sequence of the human behavior to be identified based on the acquired static and / or dynamic image information. The image frame sequence is then input into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition to determine the behavior category of the human being to be identified. This implementation achieves human behavior recognition based on an optical flow feature extraction network, a lightweight network, and a temporal network, mitigating the impact of adverse factors such as network depth, light intensity, external occlusion, and shooting orientation, thereby improving recognition speed and accuracy and achieving better recognition results.

[0203] Example 3

[0204] Figure 4This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the human behavior recognition method of Embodiment 1. Figure 4 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0205] like Figure 4 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0206] Bus 33 includes a data bus, an address bus, and a control bus.

[0207] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0208] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0209] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the human behavior recognition method of Embodiment 1 of the present invention.

[0210] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generating device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 36. Figure 4 As shown, network adapter 36 communicates with other modules of the model-generated device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0211] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0212] Example 4

[0213] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the human behavior recognition method provided in Embodiment 1.

[0214] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0215] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to execute the human behavior recognition method described in Embodiment 1.

[0216] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0217] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A method of recognizing human behavior, characterized by, The identification method includes: Acquire image information of the human behavior to be identified, wherein the image information includes static image information and / or dynamic image information; Generate the image frame sequence of the human behavior to be identified based on the static image information and / or the dynamic image information; The image frame sequence is input into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition to obtain the behavior category of the human body to be identified. The image information includes non-infrared image information and / or infrared image information. The non-infrared image information includes non-infrared static image information and / or non-infrared dynamic image information. The infrared image information includes infrared static image information and / or infrared dynamic image information. The step of generating the image frame sequence of the human behavior to be identified based on the static image information and / or the dynamic image information includes: The RGB image frame sequence of the human behavior to be identified is generated based on the non-infrared static image information and / or the non-infrared dynamic image information. Generate an infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or the infrared dynamic image information; The step of inputting the image frame sequence into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for recognition to obtain the behavior category of the human body to be identified includes: Human behavior recognition is performed on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network to obtain at least one recognition result of the human body to be identified; The weighted MAX-K fusion algorithm is used to fuse the at least one recognition result to obtain the behavior category of the human body to be identified; The step of performing human behavior recognition on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network to obtain at least one recognition result for the human body to be identified includes: The Fourier feature extraction algorithm is used to extract contour feature information from the RGB image frame sequence; The contour feature information is input into the temporal network for human behavior recognition to obtain the posture feature recognition result of the human body to be recognized. The RGB image frame sequence is input into the optical flow feature extraction network to obtain a motion image frame sequence that characterizes the running feature information of the human behavior to be identified. The motion image frame sequence is input into the lightweight network and / or the temporal network to obtain the motion feature recognition result of the human body to be identified; The RGB image frame sequence is input into the lightweight network and / or the temporal network to obtain the appearance feature recognition result of the human body to be identified; The infrared image frame sequence is input into the lightweight network and / or the temporal network to obtain the infrared feature recognition result of the human body to be identified; The weighted MAX-K fusion algorithm is used to fuse the motion feature recognition result, the appearance feature recognition result, and the infrared feature recognition result to obtain a first fused recognition result; The weighted MAX-K fusion algorithm is used to fuse the posture feature recognition result and the first fusion recognition result to obtain the behavior category of the human body to be identified.

2. The method of recognizing human behavior according to claim 1, wherein The step of performing human behavior recognition on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network to obtain at least one recognition result for the human body to be identified further includes: The motion feature recognition result and the appearance feature recognition result are fused using the weighted MAX-K fusion algorithm to obtain a second fused recognition result; The weighted MAX-K fusion algorithm is used to fuse the posture feature recognition result and the second fusion recognition result to obtain the behavior category of the human body to be identified.

3. The method for recognizing human behavior as described in claim 1, characterized in that, The step of generating the RGB image frame sequence of the human behavior to be identified based on the non-infrared static image information and / or the non-infrared dynamic image information includes: Extract the non-infrared static image information and / or the non-infrared dynamic image information's non-infrared image frames according to a preset transmission frame number; The non-infrared image frames are subjected to data augmentation processing to obtain the RGB image frame sequence of the human behavior to be identified.

4. The method for recognizing human behavior as described in claim 1, characterized in that, The step of generating the infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or the infrared dynamic image information includes: Infrared image frames of the infrared static image information and / or the infrared dynamic image information are extracted according to a preset number of transmission frames. The infrared image frames are subjected to data augmentation processing to obtain the infrared image frame sequence of the human behavior to be identified.

5. The method for recognizing human behavior as described in claim 1, characterized in that, The step of inputting the RGB image frame sequence into the optical flow feature extraction network to obtain a motion image frame sequence characterizing the motion feature information of the human behavior to be identified includes: The RGB image frame sequence is input into the optical flow feature extraction network at a preset interval of frames, and the network pruning algorithm is used to obtain a motion image frame sequence that represents the running feature information of the human behavior to be identified.

6. A human behavior recognition system, characterized in that, The identification system includes an acquisition module, a generation module, and an identification module; The acquisition module is used to acquire image information of the human behavior to be identified, and the image information includes static image information and / or dynamic image information; The generation module is used to generate an image frame sequence of the human behavior to be identified based on the static image information and / or the dynamic image information. The recognition module is used to input the image frame sequence into an optical flow feature extraction network and / or a lightweight network and / or a temporal network for human behavior recognition, so as to obtain the behavior category of the human body to be identified; The image information includes non-infrared image information and / or infrared image information. The non-infrared image information includes non-infrared static image information and / or non-infrared dynamic image information. The infrared image information includes infrared static image information and / or infrared dynamic image information. The generation module includes a first generation unit and a second generation unit. The recognition module includes a recognition unit and a fusion unit. The first generation unit is used to generate the RGB image frame sequence of the human behavior to be identified based on the non-infrared static image information and / or the non-infrared dynamic image information; The second generation unit is used to generate an infrared image frame sequence of the human behavior to be identified based on the infrared static image information and / or the infrared dynamic image information; The recognition unit is used to perform human behavior recognition on the image frame sequence and / or the infrared image frame sequence using the Fourier feature extraction algorithm and / or the optical flow feature extraction network and / or the lightweight network and / or the temporal network, so as to obtain at least one recognition result of the human body to be identified; The fusion unit is used to fuse the at least one recognition result using a weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified; The identification unit includes a third extraction subunit, a first identification subunit, an acquisition subunit, a second identification subunit, a third identification subunit, a fourth identification subunit, a first fusion subunit, and a second fusion subunit; The third extraction subunit is used to extract contour feature information from the RGB image frame sequence using the Fourier feature extraction algorithm; The first recognition subunit is used to input the contour feature information into the temporal network for human behavior recognition, so as to obtain the posture feature recognition result of the human body to be recognized; The acquisition subunit is used to input the RGB image frame sequence into the optical flow feature extraction network to obtain a motion image frame sequence that characterizes the running feature information of the human behavior to be identified. The second recognition subunit is used to input the motion image frame sequence into the lightweight network and / or the temporal network to obtain the motion feature recognition result of the human body to be identified; The third recognition subunit is used to input the RGB image frame sequence into the lightweight network and / or the temporal network to obtain the appearance feature recognition result of the human body to be recognized; The fourth recognition subunit is used to input the infrared image frame sequence into the lightweight network and / or the temporal network to obtain the infrared feature recognition result of the human body to be identified; The first fusion subunit is used to fuse the motion feature recognition result, the appearance feature recognition result, and the infrared feature recognition result using the weighted MAX-K fusion algorithm to obtain a first fusion recognition result; The second fusion subunit is used to fuse the posture feature recognition result and the first fusion recognition result using the weighted MAX-K fusion algorithm to obtain the behavior category of the human body to be identified.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the human behavior recognition method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human behavior recognition method as described in any one of claims 1-5.

Images