Human behavior recognition method and terminal
By constructing and training deep network models, the problem of poor adaptability of human behavior recognition algorithms on edge devices is solved, and efficient and accurate portrait image behavior recognition is achieved on edge devices.
Patent Information
- Application Number
- CN202110880546.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-02
- Publication Date
- 2025-04-15
- Estimated Expiration
- 2041-08-02
AI Technical Summary
In the prior art, the human behavior recognition model based on deep learning has problems such as loss of accuracy and excessive computing power requirements after being converted to edge devices, resulting in the inability to be effectively applied to edge devices.
Build and train deep network models, including normalized layers, convolutional layers, downsampled branches and upsampled branches, image feature extraction and processing are performed end-to-end, model parameters are optimized in combination with specific loss functions, and the computing power of edge devices is adapted to the calculation capabilities of edge devices.
Without taking up too much computing power, the detection efficiency and accuracy of portrait image recognition are improved, the cumulative error of non-end-to-end methods is reduced, and the end-to-end image calculation and recognition can be directly performed on edge devices.
Smart Images
Figure CN113609957B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for performing image analysis on human behavior, and in particular to a method for identifying human behavior with less occupied resources. Background Art
[0002] Behavior classification on edge devices is a challenging task with high practical application value. It can play a huge role in security monitoring, advertising, and other tasks related to video understanding. Some existing behavior recognition methods on edge devices are also based on deep learning. First, a human joint classification model based on deep learning is trained based on data, and then the posture of the human body is classified according to the identified joints. The advantage of this type of method is that it can dynamically combine the image information in the video frame to achieve the effect of identifying part of the action in the video. However, its disadvantage is that the model is often accompanied by the problem of precision loss after being converted to the edge device, especially for non-end-to-end models. During the conversion, because the precision loss of the two models is involved, the superposition of the two will bring greater precision loss. At the same time, due to the reason of inference speed, this type of model requires greater computing power support, and the processing time on the edge device is long, so that the behavior classification model cannot be well applied to the edge device. Summary of the invention
[0003] To this end, it is necessary to provide a human behavior recognition algorithm with low computing power consumption to solve the problem of poor adaptability of human behavior recognition algorithms for edge devices in the existing technology.
[0004] To achieve the above object, the inventor provides a method for human behavior recognition, comprising the following steps:
[0005] S2: Construct a deep network model for identifying abnormal human behavior;
[0006] S3: Train the deep network model and convert it to the edge device;
[0007] S4: Use the deep network model to infer the input action behavior image to be classified, obtain the inferred image feature map, post-process and splice the obtained image feature map to obtain the final image feature map, and analyze the final image feature map to obtain the action classification result and action area box result.
[0008] Specifically, the step S2 constructs a deep network model for identifying abnormal human behavior as follows:
[0009] S21: Use the normalization layer and the convolution layer to normalize the input image and adjust the obtained image feature map;
[0010] S22: input the image feature map obtained in S21 into a downsampling branch to obtain a small-scale image feature map;
[0011] S23: Input the small-scale image feature map obtained in S22 into an upsampling branch to restore the image feature map to half the size of the original image;
[0012] S24: Input the image feature map restored in S23 into each branch network to obtain the image feature map of the corresponding action.
[0013] Specifically, in step S21, the normalization parameter of the normalization layer is 255; the parameters of the convolution layer are: the convolution kernel is 1×1, the step size is 1, and the number of filters is 8;
[0014] In step S22, the structure of the downsampling branch is composed of a downsampling module and an encoding module in sequence, wherein the downsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 2, and a convolution layer with a convolution kernel of 3×3, a step size of 1, and a filter number of 16; the encoding module includes two convolution layers, and the convolution kernels of the two convolution layers are both 3×3 and the step size is 1;
[0015] In step S23, the structure of the upsampling branch is sequentially composed of an upsampling module and a decoding module, the upsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 4; the decoding module is composed of a convolution layer with a convolution kernel of 5×5 and a step size of 1;
[0016] In step S24, the branch network structure is composed of a convolution layer with a convolution kernel of 3×3, a step size of 2, and 16 filters, a convolution layer with a convolution kernel of 3×3, a step size of 2, and 5 filters, and an upsampling layer with a sampling size of 2×2.
[0017] Furthermore, the structure of the downsampling branch is sequentially composed of 1 downsampling module and 6 encoding modules, and the numbers of filters of 12 convolutional layers in the 6 encoding modules are 16, 32, 32, 48, 48, 48, 48, 48, 48, 48, 48, 48 respectively;
[0018] The structure of the upsampling branch is composed of 7 combination modules in sequence, each combination module is composed of an upsampling module and a decoding module in sequence; the number of filters of the convolutional layer of the decoding module in the 7 combination modules is 48, 48, 48, 48, 32, and 16 respectively.
[0019] Specifically, the step S3 of training the deep network model includes:
[0020] S31: Obtain a training set of action behavior images to be classified, find abnormal action areas in each image, and mark the coordinates of four vertices of the abnormal action area as true value labels;
[0021] S32: inputting the labeled training set of action behavior images to be classified into the deep network model, and obtaining an image feature atlas through inference;
[0022] S33: Calculate the loss of each image feature map in the image feature map set and the true value label of the corresponding image, then back-propagate the loss to the deep network model, and update the parameter weights of the deep network model;
[0023] Repeat step S33 to traverse each image feature map in the image feature map set, and continuously update the parameter weights of the deep network model until the model is stable.
[0024] Specifically, the step 31 also includes an image preprocessing step, which is: the camera carried by the edge device collects images of abnormal human behavior and marks the abnormal human behavior areas in the images; after marking, the images are augmented and grayscaled to expand the data volume to obtain the images required for model training; the abnormal behaviors include at least one of the following: playing with mobile phones, turning heads, lying on the table, or looking up.
[0025] Specifically, the step S33 specifically includes: setting the learning rate of the deep network model, calculating the loss of each image feature map in the image feature map set and the true value label of the corresponding image according to the learning rate and the loss function, obtaining the loss value, performing back propagation of the loss on the deep network model according to the loss value, and updating the parameter weights of the deep network model.
[0026] Preferably, in step S33,
[0027] The learning rate of the model is 0.0001, and the loss function used by the model is:
[0028] Loss=(Loss ce +Loss mse )*w (1)
[0029] Among them, Loss ce is the multi-classification cross entropy loss, Loss mse is the mean square error loss, w is the weight calculated according to the number of label categories and label values;
[0030] Loss ce The calculation of is as follows:
[0031] Loss ce =categorical_crossentropy(y true,y pred ) (2)
[0032] where y true is the true value label, y pred is the model prediction value, categorical_crossentropy represents multi-classification cross entropy calculation;
[0033] Loss mse The calculation of is as follows:
[0034] Loss mse =mean((y true -y pred ) 2 ) (3)
[0035] Among them, mean represents the mean operation;
[0036] The calculation of w is as follows:
[0037] w=(1 / n_class)*non_zero_class / k (4)
[0038] Where k is the sum of the true value labels in its length and width dimensions,
[0039] K = sum(y true ) (5)
[0040] n_class is the number of action categories, non_zero_class is the array after the value of array k is modified with 0.1 as the boundary, and the value of the part greater than 0.1 is 1, otherwise it is 0:
[0041] non_zero_class=k>0.1 (6).
[0042] Furthermore, in step S3, converting the model to the edge device includes converting the deep network model trained by the GPU into an embedded neural network processor inference model, and applying the converted deep network model to the edge device through an interface.
[0043] Furthermore, step S4 specifically includes the following steps:
[0044] The acquired image feature maps are post-processed and spliced to obtain the final image feature maps, and the final image feature maps are analyzed to obtain the action classification results and action region box results.
[0045] Step S41: inputting the action behavior image to be classified into the deep network model for inference to obtain an inferred image feature map;
[0046] Step S42: post-processing the acquired image feature map, specifically: the image feature map includes channel image feature maps output by channels corresponding to each branch network, and each channel image feature map output by a branch network corresponds to an action behavior type; the maximum value is obtained for each pixel point at the same position of different channel image feature maps, the pixel point with the maximum value, the coordinates of the pixel point and the channel to which the pixel point belongs are retained, all the pixel points with the maximum value are spliced according to their coordinates to obtain the final image feature map, a square area corresponding to the pixel points of the same channel is drawn in the final image feature map, the proportion of the pixel points of the same channel in the square area is calculated, and the proportion is compared with a preset value; if the proportion is greater than the preset value, the action behavior type corresponding to the channel to which the corresponding pixel point belongs is judged as the action behavior category in the image, and the corresponding square area is the detection box area of the action behavior.
[0047] The present invention also discloses a human behavior recognition terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements each step of the above-mentioned human behavior recognition method when executing the computer program.
[0048] Different from the existing technology, the above technical solution can recognize portrait images without taking up too much computing power, thus improving the detection efficiency. At the same time, in terms of the recognition of portrait image behavior, it is compatible with edge devices, and can directly perform end-to-end image calculation and recognition on the device side, greatly reducing the cumulative error caused by the non-end-to-end method in step-by-step recognition, and the recognition accuracy is higher. The model structure is simple, the output results are intuitive, and it can directly locate the type and area of abnormal action behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flow chart of the human behavior recognition method described in the specific implementation mode;
[0050] Figure 2 A flowchart showing a specific implementation of the hierarchical architecture of the deep network model described in the specific implementation method;
[0051] Figure 3 A flowchart for executing a deep network model training as described in a specific implementation method;
[0052] Figure 4 It is a flow chart of the method for obtaining classification results described in the specific implementation method;
[0053] Figure 5 This is a diagram showing the effect of action recognition described in a specific implementation method. DETAILED DESCRIPTION
[0054] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.
[0055] See also Figure 1 , is a human behavior recognition method, comprising the following steps:
[0056] S2: Construct a deep network model for identifying abnormal human behavior;
[0057] S3: Train the deep network model and convert it to the edge device;
[0058] S4: Use the deep network model to infer the input action behavior image to be classified, obtain the inferred image feature map, post-process and splice the obtained image feature map to obtain the final image feature map, and analyze the final image feature map to obtain the action classification result and action area box result.
[0059] The technical solution of the present invention designs the construction, training, inference and post-processing steps of the deep network model. The features of the specific behavior actions to be identified are analyzed and processed, and output in an end-to-end manner, which greatly improves the accuracy of the deep network model results. At the same time, the deep network model has a simple structure, good scalability, and the output results are easy to observe directly, and the execution efficiency is high.
[0060] The step S2 constructs a deep network model for identifying abnormal human behavior as follows:
[0061] S21: normalizing the input video image using a normalization layer and a convolution layer and adjusting the size of the obtained image feature information;
[0062] S22: Input the image feature map obtained in S21 into a downsampling branch to obtain a small-scale image feature;
[0063] S23: Input the small-scale image feature map obtained in S22 into an upsampling branch to restore the image feature map to half the size of the original image;
[0064] S24: Input the image feature map restored in S23 into the classification branch network to obtain a feature map of the corresponding action.
[0065] In some specific embodiments, the number of channels of the output image feature map of each branch network is the number of action categories to be identified plus 1. This is because an additional channel of the background category is required.
[0066] In some more specific embodiments, in step S21, the normalization parameter of the normalization layer is 255; the parameters of the convolution layer are: the convolution kernel is 1×1, the step size is 1, and the number of filters is 8;
[0067] In step S22, the structure of the downsampling branch is composed of a downsampling module and an encoding module in sequence, wherein the downsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 2, and a convolution layer with a convolution kernel of 3×3, a step size of 1, and a filter number of 16; the encoding module includes two convolution layers, and the convolution kernels of the two convolution layers are both 3×3 and the step size is 1;
[0068] In step S23, the structure of the upsampling branch is sequentially composed of an upsampling module and a decoding module, the upsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 4; the decoding module is composed of a convolution layer with a convolution kernel of 5×5 and a step size of 1;
[0069] In step S24, the branch network structure is composed of a convolution layer with a convolution kernel of 3×3, a step size of 2, and 16 filters, a convolution layer with a convolution kernel of 3×3, a step size of 2, and 5 filters, and an upsampling layer with a sampling size of 2×2.
[0070] In some further embodiments, the structure of the downsampling branch is composed of 1 downsampling module and 6 encoding modules in sequence, and the numbers of filters of the 12 convolutional layers in the 6 encoding modules are 16, 32, 32, 48, 48, 48, 48, 48, 48, 48, 48, 48, respectively; the structure of the upsampling branch is composed of 7 combination modules in sequence, and each combination module is composed of an upsampling module and a decoding module in sequence. The numbers of filters of the convolutional layers of the decoding modules in the 7 combination modules are 48, 48, 48, 48, 48, 32, and 16, respectively.
[0071] The structure of the downsampling branch is composed of a downsampling module and 6 encoding modules in sequence. The downsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 2, and a convolution layer with a convolution kernel of 3×3, a step size of 1, and 16 filters; the encoding module contains two convolution layers, and the convolution kernels of the two convolution layers are both 3×3 and the step size is 1. The number of filters of the 12 convolution layers in the 6 encoding modules are 16, 32, 32, 48, 48, 48, 48, 48, 48, 48, 48, 48, respectively.
[0072] In step S23, the structure of the upsampling branch is composed of 7 combination modules in sequence, each combination module is composed of an upsampling module and a decoding module in sequence, wherein each upsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 4; the 7 decoding modules are composed of convolution layers with a convolution kernel of 5×5, a step size of 1, and the number of filters are 48, 48, 48, 48, 32, and 16 respectively.
[0073] In step S24, the branch network structure is composed of a convolution layer with a convolution kernel of 3×3, a step size of 2, and 16 filters, a convolution layer with a convolution kernel of 3×3, a step size of 2, and 5 filters, and an upsampling layer with a sampling size of 2×2.
[0074] Through a deep network model, abnormal behavior can be directly identified in an end-to-end manner. When designing the model, the image features are obtained by using structures such as normalization layers, convolution layers, downsampling branches, and upsampling branches, and then the branch network is used to obtain feature maps of different actions. This design makes the overall structure of the model simpler and more direct, the model size is moderate, and the results can be obtained within 150ms. The above solution can adapt well to the hardware conditions on edge devices.
[0075] In some other embodiments, in order to better train the deep network model, such as Figure 1 As shown, the method further includes the steps of: S1: obtaining an image of abnormal human behavior, marking the abnormal human behavior area in the image, and performing image preprocessing. By marking the abnormal human behavior area, the abnormal human behavior to be identified can be obtained, and the marked abnormal human behavior is used as the learning material of the deep network model. The above steps can improve the recognition rate of the deep network model.
[0076] In the present invention, the step S3 of training the deep network model includes:
[0077] S31: Obtain a training set of action behavior images to be classified, find abnormal action areas in each image, and mark the coordinates of four vertices of the abnormal action area as true value labels;
[0078] S32: inputting the labeled training set of action behavior images to be classified into the deep network model, and obtaining an image feature atlas through inference;
[0079] S33: Calculate the loss of each image feature map in the image feature map set and the true value label of the corresponding image, then back-propagate the loss to the deep network model, and update the parameter weights of the deep network model;
[0080] Repeat step S33 to traverse each image feature map in the image feature map set, and continuously update the parameter weights of the deep network model until the model is stable.
[0081] According to the above training steps, the specific actions to be identified can be labeled and features that are helpful for identification can be found. The loss function used can better reflect the error between the real image pixel area and the true value label image pixel area, making the learning goal of the model clearer, and obtaining a more accurate area of the behavior action through continuous iterative loss calculation.
[0082] In the present invention, the step 31 also includes an image preprocessing step, specifically: the camera carried by the edge device collects images of abnormal human behavior and marks the abnormal human behavior areas in the images; after marking, the images are augmented and grayscaled to expand the data volume to obtain the images required for model training; the abnormal behaviors include at least one of the following: playing with mobile phones, turning heads, lying on the table, or looking up.
[0083] Through the above image preprocessing steps, the amount of data for model training can be reduced through a small number of data sets, which increases the data that may exist in real scenes and makes the generalization effect of the model better.
[0084] In some specific embodiments, in order to better train the deep network model, the step S33 may also specifically include: setting the learning rate of the deep network model, calculating the loss of each image feature map in the image feature map set and the true value label of the corresponding image according to the learning rate and the loss function, obtaining the loss value, performing back propagation of the loss of the deep network model according to the loss value, and updating the parameter weights of the deep network model. By setting the learning rate and loss function through the above steps, iterating and repeating step S33, the trained deep network model can be better converged.
[0085] Specifically, the learning rate of the model is 0.0001, and the loss function used by the model is:
[0086] Loss=(Loss ce +Loss mse )*w (1)
[0087] Among them, Loss ce is the multi-classification cross entropy loss, Loss mse is the mean square error loss, w is the weight calculated according to the number of label categories and label values;
[0088] Loss ce The calculation of is as follows:
[0089] Loss ce =categorical_crossentropy(y true ,y pred ) (2)
[0090] where y true is the true value label, y pred is the model prediction value, categorical_crossentropy represents multi-classification cross entropy calculation;
[0091] Loss mseThe calculation of is as follows:
[0092] Loss mse =mean((y true -y pred ) 2 ) (3)
[0093] Among them, mean represents the mean operation;
[0094] The calculation of w is as follows:
[0095] w=(1 / n_class)*non_zero_class / k (4)
[0096] Where k is the sum of the true value labels in its length and width dimensions,
[0097] K = sum(y true ) (5)
[0098] n_class is the number of action categories, non_zero_class is the array after the value of array k is modified with 0.1 as the boundary, and the value of the part greater than 0.1 is 1, otherwise it is 0:
[0099] non_zero_class=k>0.1 (6).
[0100] Among them, sum represents the sum operation.
[0101] like Figure 4 In the specific embodiment shown, step S4 specifically includes the following steps:
[0102] Step S41: Input the action behavior picture to be classified into the model obtained in step S32 to obtain the inferred image feature map. The action behavior picture that needs to be judged is different according to the annotation of the material training set. Step S42: Post-process the acquired image feature map, specifically: the image feature map includes the channel image feature map of the channel output corresponding to each branch network, and each channel image feature map output by a branch network corresponds to a type of action behavior; find the maximum value of each pixel point at the same position of different channel image feature maps, retain the pixel point of the maximum value, the coordinates of the pixel point and the channel to which the pixel point belongs, splice all the pixel points of the maximum value according to their coordinates to obtain the final image feature map, draw a square area corresponding to the same channel pixel point in the final image feature map, calculate the proportion of the same channel pixel point in the square area, and compare the proportion with the preset value; if the proportion is greater than the preset value, the action behavior type corresponding to the channel to which the corresponding pixel point belongs is judged as the action behavior category in the image, and the action behavior type here can include abnormal behavior actions to be identified, or background actions (i.e. actions without abnormal behavior). The corresponding square area is the detection frame area of the action behavior. In this embodiment, in step S41, it is only necessary to input the video image into the model to obtain the inferred image feature map. In step S42, in order to obtain the final image feature map, the preset value in the present invention is adjusted according to experience, actual needs, etc., and the value range is generally 0.25-0.7. In general, we set it to 0.5.
[0103] The present invention can adjust the accuracy and recall of the model recognition results according to the set threshold value. Using a high preset value will reduce the number of detection results, but improve the accuracy of the detection results, reduce false detections, and reduce the recall rate; using a low preset value will increase the number of detection results, but reduce the accuracy, increase false detections, and improve the recall rate. Therefore, the present invention balances the accuracy and recall of the model recognition results by setting an appropriate threshold value.
[0104] like Figure 5 As shown, we also show the detection box results for the action of playing with the phone and turning the head.
[0105] The advantages of this step of the present invention are: the end-to-end recognition solution greatly reduces the accuracy loss associated with the model when it is converted to the edge device, and the single-model solution also saves the time of the entire recognition process, and it only takes 150ms to complete the recognition of an image.
[0106] The model structure of the present invention is simple and the model size is moderate. The model can obtain the corresponding feature map results in different channels for different actions, and the effect is direct and obvious after visualization, which is convenient for verification. The use of downsampling branches allows us to obtain a small-scale image feature, expand the receptive field, reduce the parameters of the model, and improve the efficiency of the model.
[0107] The above scheme of the present invention also has a weighted loss with a weight w that better fits the error between the true value label and the predicted value. When performing model training, better results can be obtained and a model that is closer to the label can be obtained.
[0108] The present invention makes the model scalable by using different branch networks. For new actions that need to be added later, it is only necessary to add new data sets and add new output branches to the model to predict the feature results of the new actions, and other parts do not need to be modified.
[0109] The present invention also discloses a human behavior recognition terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements each step of the above-mentioned human behavior recognition method when executing the computer program.
[0110] It should be noted that, although the above embodiments have been described in this article, the scope of patent protection of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields are all included in the scope of protection of the patent of the present invention.
Claims
1. A human behavior recognition method, characterized in that: The steps include: S2: Construct a deep network model for abnormal human behavior recognition, specifically: S21: Use the normalization layer and the convolution layer to normalize the input image and adjust the obtained image feature map; S22: input the image feature map obtained in S21 into a downsampling branch to obtain a small-scale image feature map; S23: Input the small-scale image feature map obtained in S22 into an upsampling branch to restore the image feature map to half the size of the original image; S24: input the image feature map restored in S23 into each branch network to obtain an image feature map of the corresponding action; S3: Train the deep network model and convert it to the edge device; Training a deep network model specifically includes: S31: Obtain a training set of action behavior images to be classified, find abnormal action areas in each image, and mark the coordinates of four vertices of the abnormal action area as true value labels; S32: inputting the labeled training set of action behavior images to be classified into the deep network model, and obtaining an image feature atlas through inference; S33: Calculate the loss of each image feature map in the image feature map set and the true value label of the corresponding image, set the learning rate of the deep network model, calculate the loss of each image feature map in the image feature map set and the true value label of the corresponding image according to the learning rate and the loss function, obtain the loss value, perform back propagation of the loss on the deep network model according to the loss value, and update the parameter weights of the deep network model; the learning rate of the model is 0.0001, and the loss function used by the model is: Loss=(Loss ce +Loss mse )*w (1) Among them, Loss ce is the multi-classification cross entropy loss, Loss mse is the mean square error loss, w is the weight calculated according to the number of label categories and label values; Loss ce The calculation of is as follows: Loss ce =categorical_crossentropy(y true ,y pred ) (2) Where ytrue is the true value label, ypred is the model prediction value, and categorical_crossentropy represents multi-classification cross entropy calculation; Loss mse The calculation of is as follows: Loss mse =mean((y true -y pred ) 2 ) (3) Among them, mean represents the mean operation; The calculation of w is as follows: w=(1 / n_class)*non_zero_class / k (4) Where k is the sum of the true value labels in its length and width dimensions, K=sum(y true ) (5) n_class is the number of action categories, non_zero_class is the array after the value of array k is modified with 0.1 as the boundary, and the value of the part greater than 0.1 is 1, otherwise it is 0: non_zero_class=k>0.1 (6) Repeat step S33 to traverse each image feature map in the image feature map set, and continuously update the parameter weights of the deep network model until the model is stable; S4: Use the deep network model to infer the input action behavior image to be classified to obtain the inferred image feature map; post-process and splice the obtained image feature map to obtain the final image feature map, and analyze the final image feature map to obtain the action classification result and action area box result.
2. The human behavior recognition method according to claim 1, characterized in that: In step S21, the normalization parameter of the normalization layer is 255; the parameters of the convolution layer are: the convolution kernel is 1×1, the step size is 1, and the number of filters is 8; In step S22, the structure of the downsampling branch is composed of a downsampling module and an encoding module in sequence, wherein the downsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 2, and a convolution layer with a convolution kernel of 3×3, a step size of 1, and a filter number of 16; the encoding module includes two convolution layers, and the convolution kernels of the two convolution layers are both 3×3 and the step size is 1; In step S23, the structure of the upsampling branch is sequentially composed of an upsampling module and a decoding module, the upsampling module is composed of a separable convolution layer with a convolution kernel of 5×5, a step size of 2, and a depth multiplier of 4; the decoding module is composed of a convolution layer with a convolution kernel of 5×5 and a step size of 1; In step S24, the structure of the branch network is composed of a convolution layer with a convolution kernel of 3×3, a step size of 2, and 16 filters, a convolution layer with a convolution kernel of 3×3, a step size of 2, and 5 filters, and an upsampling layer with a sampling size of 2×2.
3. The human behavior recognition method according to claim 2, characterized in that: The structure of the downsampling branch is composed of a downsampling module and 6 encoding modules in sequence, and the numbers of filters of the 12 convolutional layers in the 6 encoding modules are 16, 32, 32, 48, 48, 48, 48, 48, 48, 48, 48, 48 respectively; The structure of the upsampling branch is composed of 7 combination modules in sequence, each combination module is composed of an upsampling module and a decoding module in sequence; the number of filters of the convolutional layer of the decoding module in the 7 combination modules is 48, 48, 48, 48, 32, and 16 respectively.
4. The human behavior recognition method according to claim 1, characterized in that: The step 31 also includes an image preprocessing step, specifically: the camera carried by the edge device collects images of abnormal human behavior and marks the abnormal human behavior areas in the images; after marking, the images are augmented and grayscaled to expand the data volume to obtain the images required for model training; the abnormal behaviors include at least one of the following: playing with mobile phones, turning the head, lying on the table, or looking up.
5. The human behavior recognition method according to claim 1, characterized in that: In step S4, the acquired image feature map is post-processed and spliced to obtain a final image feature map, and the final image feature map is analyzed to obtain an action classification result and an action region frame result, which specifically includes the following steps: Step S41: inputting the action behavior image to be classified into the deep network model for inference to obtain an inferred image feature map; Step S42: post-processing the acquired image feature map, specifically: the image feature map includes channel image feature maps output by channels corresponding to each branch network, and each channel image feature map output by a branch network corresponds to an action behavior type; the maximum value is obtained for each pixel point at the same position of different channel image feature maps, the pixel point with the maximum value, the coordinates of the pixel point and the channel to which the pixel point belongs are retained, all the pixel points with the maximum value are spliced according to their coordinates to obtain the final image feature map, a square area corresponding to the pixel points of the same channel is drawn in the final image feature map, the proportion of the pixel points of the same channel in the square area is calculated, and the proportion is compared with a preset value; if the proportion is greater than the preset value, the action behavior type corresponding to the channel to which the corresponding pixel point belongs is judged as the action behavior category in the image, and the corresponding square area is the detection frame area of the action behavior.
6. A human behavior recognition terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the human behavior recognition method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Abnormal driving behavior detection method based on multi-task depth convolution neural network
CN109034134A
A traffic vehicle density estimation method based on space-based monitoring
CN109948524A