A behavior analysis model training method, a behavior analysis method, and a device thereof

Abstract

The application discloses a behavior analysis model training method, a behavior analysis method and equipment thereof. The model training method comprises the following steps: constructing a continuous frame human eye training data set; generating a human eye cluster image based on the continuous frame human eye training data set; inputting the human eye cluster image into a to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification comprises an eye behavior category and a first prediction probability thereof; obtaining a first prediction loss of the to-be-trained analysis model based on the eye behavior classification of the continuous frame human eye training data set and a real eye behavior label; and training the to-be-trained analysis model by using the first prediction loss to obtain a final behavior analysis model. The behavior analysis model training method can effectively reduce parameter redundancy, complex calculation and storage consumption of model training by encoding a plurality of human eye image frames of the continuous frame human eye training data set into the same human eye cluster image, and improves the efficiency of model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image recognition, in particular to a behavior analysis model training method, a behavior analysis method and a device thereof. BACKGROUND

[0002] Blinking behavior is a normal physiological response, which naturally reflects biological activity. In order to avoid eye fatigue, while reflecting the good interaction state of human with the outside world, blinking is inevitable. With the popularization of intelligent technology in real life, blinking behavior is widely used in living body detection, fatigue monitoring, micro-expression analysis, lie detection system and other scenes.

[0003] However, the current blinking detection academic achievements based on single-frame spatial features lack timing information, which naturally causes information loss for behavior recognition; the blinking detection academic achievements based on sensing devices and devices have the problems of high device cost, difficult expansion to general scenarios, and low practicality; the blinking detection academic achievements based on multi-frame timing weighted fusion have the problem that complex calculation and storage consumption will make the algorithm difficult to use. SUMMARY

[0004] The present application provides a behavior analysis model training method, a behavior analysis method and a device thereof.

[0005] To solve the above technical problems, the first technical solution provided by the present application is to provide a behavior analysis model training method, which comprises:

[0006] Constructing a continuous frame human eye training data set, wherein the continuous frame human eye training data set comprises a plurality of continuous human eye image frames;

[0007] Generating a human eye cluster image based on the continuous frame human eye training data set;

[0008] Inputting the human eye cluster image into a to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification comprises an eye behavior category and a first prediction probability thereof;

[0009] Obtaining a first prediction loss of the to-be-trained analysis model based on the eye behavior classification of the continuous frame human eye training data set and a real eye behavior label, and training the to-be-trained analysis model by using the first prediction loss to obtain a final behavior analysis model.

[0010] Among them, the constructing a continuous frame human eye training data set comprises:

[0011] An eye detection network is used to detect the human eye image in the monitoring video stream to determine the human eye image including the eye region as the starting frame;

[0012] The continuous-frame human eye training dataset is composed of a plurality of human eye images based on a starting frame.

[0013] The continuous-frame human eye training dataset is composed of a plurality of human eye images based on a starting frame.

[0014] The continuous-frame human eye training dataset is composed of a plurality of human eye images based on a starting frame.

[0015] The continuous-frame human eye training dataset is composed of a plurality of human eye images based on a starting frame.

[0016] The continuous-frame human eye training dataset is composed of a plurality of human eye images based on a starting frame.

[0017] The eye detection network is used to locate an eye region in the human eye image.

[0018] The human eye image is cropped according to the eye region to obtain a human eye region image.

[0019] The continuous-frame human eye training dataset is composed of a plurality of human eye region images.

[0020] The human eye image is cropped according to the eye region to obtain a human eye region image.

[0021] A face alignment algorithm is used to correct the plurality of human eye images so that the directions of the eye regions in the corrected human eye images are consistent.

[0022] The human eye image is cropped according to the eye region to obtain a human eye region image.

[0023] The continuous-frame human eye training dataset is composed of a plurality of human eye images based on a starting frame.

[0024] The plurality of human eye image frames in the continuous-frame human eye training dataset are encoded into the same image according to a preset image encoding rule to generate the human eye cluster image.

[0025] The preset image encoding rule includes the time sequence of the plurality of human eye image frames in the continuous-frame human eye training dataset.

[0026] The analysis model to be trained includes a first analysis model to be trained and a second analysis model to be trained.

[0027] The analysis model training method further includes:

[0028] inputting the human eye cluster image into a first to-be-trained analysis model to obtain a predicted human eye state logical clue feature map, wherein the predicted human eye state logical clue feature map comprises a first predicted probability of each human eye image frame;

[0029] obtaining a second predicted loss of the first to-be-trained analysis model based on the predicted human eye state logical clue feature map and a real human eye state logical clue feature map, and training the first to-be-trained analysis model by using the second predicted loss;

[0030] inputting the predicted human eye state logical clue feature map into a second to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification comprises an eye behavior category and a third predicted probability thereof;

[0031] obtaining a third predicted loss of the second to-be-trained analysis model based on the eye behavior classification of the continuous frame human eye training data set and a real eye behavior label, and training the second to-be-trained analysis model by using the third predicted loss.

[0032] The first to-be-trained analysis model comprises a shallow convolution kernel and a deep convolution kernel.

[0033] The inputting the human eye cluster image into a first to-be-trained analysis model to obtain a predicted human eye state logical clue feature map comprises:

[0034] inputting the human eye cluster image into the first to-be-trained analysis model;

[0035] extracting local features of the human eye cluster image by using the shallow convolution kernel, wherein the local features represent feature relationships of adjacent human eye image frames;

[0036] extracting global features of the human eye cluster image by using the deep convolution kernel, wherein the global features represent feature relationships of human eye image frames in the human eye cluster image;

[0037] generating the predicted human eye state logical clue feature map based on the local features and the global features of the human eye cluster image.

[0038] To solve the above technical problems, a second technical solution provided by the present application is to provide a behavior analysis method, which comprises:

[0039] obtaining a monitoring video stream;

[0040] inputting the monitoring video stream into the behavior analysis model to obtain an output behavior analysis category and a predicted probability of each category;

[0041] selecting a behavior analysis category with the highest predicted probability as a behavior analysis result of the monitoring video stream.

[0042] The behavior analysis model is obtained by the behavior analysis model training method.

[0043] The behavior analysis model is obtained by the behavior analysis model training method.

[0044] The behavior analysis model is obtained by the behavior analysis model training method.

[0045] If yes, output the behavior analysis category as the behavior analysis result of the monitoring video stream.

[0046] If no, output the behavior analysis result that the behavior analysis category event does not occur.

[0047] To solve the above technical problems, the third technical solution provided by the present application is to provide a terminal device, which comprises a processor and a memory connected to the processor, wherein the memory stores program instructions; and the processor is configured to execute the program instructions stored in the memory to implement the behavior analysis model training method and / or the behavior analysis method.

[0048] To solve the above technical problems, the fourth technical solution provided by the present application is to provide a computer readable storage medium, which stores program instructions, and the program instructions are executed to implement the behavior analysis model training method and / or the behavior analysis method.

[0049] In the behavior analysis model training method provided by the present application, a terminal device constructs a continuous frame human eye training data set, wherein the continuous frame human eye training data set comprises a plurality of continuous human eye image frames; a human eye cluster image is generated based on the continuous frame human eye training data set; the human eye cluster image is input into a to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification comprises an eye behavior category and a first prediction probability; a first prediction loss of the to-be-trained analysis model is obtained based on the eye behavior classification of the continuous frame human eye training data set and a real eye behavior label; and the to-be-trained analysis model is trained by using the first prediction loss to obtain a final behavior analysis model. The behavior analysis model training method can effectively reduce the parameter redundancy, complex calculation and storage consumption of model training, and improve the efficiency of model training by encoding a plurality of human eye image frames of the continuous frame human eye training data set into a human eye cluster image. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings. Among them:

[0051] Figure 1 is a flowchart of an embodiment of the behavior analysis model training method provided by the present application;

[0052] Figure 2 is a flowchart of the behavior analysis model algorithm training and testing provided by the present application;

[0053] Figure 3 is a schematic diagram of human eye detection and spatial pose transformation provided by the present application;

[0054] Figure 4 is a schematic diagram of human eye frame sequence labeling provided by the present application;

[0055] Figure 5 is a schematic diagram of human eye data encoding provided by the present application;

[0056] Figure 6 is a flowchart of another embodiment of the behavior analysis model training method provided by the present application;

[0057] Figure 7 is a data flow diagram of the behavior analysis model provided by the present application;

[0058] Figure 8 is an example diagram of the human eye state logical clue feature map provided by the present application;

[0059] Figure 9 is a flowchart of an embodiment of the behavior analysis method provided by the present application;

[0060] Figure 10 is a structural schematic diagram of an embodiment of the terminal device provided by the present application;

[0061] Figure 11 is a structural schematic diagram of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described clearly and completely in the following description with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0063] The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0064] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an embodiment of the behavior analysis model training method provided by the present application, Figure 2 is a flowchart of the behavior analysis model algorithm training and testing provided by the present application.

[0065] The behavior analysis model training method can run in a browser or an application, and can be applied to a terminal device. The terminal device of the present application can be a server, or a system cooperating with a local terminal. Accordingly, each part of the terminal device, such as each unit, subunit, module, and sub-module, can be provided in the server, or can be provided in the server and the local terminal, respectively.

[0066] Further, the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing a distributed server, or as a single software or software module, which is not limited here. In some possible implementation manners, the abnormal motion state detection method of the embodiments of the present application can be implemented by a processor calling computer readable instructions stored in a memory.

[0067] As shown in Figure 1 , the specific steps of the behavior analysis model training method of the embodiments of the present application are as follows:

[0068] Step S11: constructing a continuous frame human eye training data set, wherein the continuous frame human eye training data set includes a plurality of continuous human eye image frames.

[0069] In the embodiments of the present application, the terminal device needs to intercept a plurality of continuous human eye image frames to form a continuous frame human eye training data set as a training set of the behavior analysis model for any input human eye image.

[0070] Before intercepting the plurality of continuous human eye image frames, the terminal device needs to determine the starting frame of the continuous frame human eye training data set. Specifically, the terminal device can use an eye detection network to detect the human eye image in the monitoring video stream to determine the human eye image including the eye region as the starting frame. It should be noted that the eye detection network of the embodiments of the present application is trained in advance through a large number of eye images and non-eye images, so that the eye detection network can detect and identify the eye region in the image.

[0071] In addition, the embodiments of this application use a pre-trained eye detection network to perform regional image recognition of the eye region in human eye images. Compared with using facial key points, such as human eye key points, to locate and recognize the eye region, the embodiments of this application directly perform image recognition of the eye region, which can reduce the impact of key feature point localization accuracy and improve the efficiency of constructing a continuous frame human eye training dataset.

[0072] Specifically, the methods for obtaining images of the human eye region in this application include, but are not limited to, the following four:

[0073] (a) Image cropping after locating the eye region based on key points.

[0074] (b) Image cropping after detecting the human eye region based on an eye detection network.

[0075] (c) The eye region is located and the image is cropped based on the three-dimensional standard face alignment method.

[0076] (d) Based on traditional features, the eye contour is located and then the eye image is extracted by binarization.

[0077] like Figure 2 As shown, the terminal device can capture face data from a video stream, capture the eye region using a face alignment method, and combine this with eye state labels to construct a continuous frame eye training dataset. Specifically, for any input eye image, the terminal device uses a face alignment algorithm based on a lightweight eye detection network and spatial pose correction to locate and correct the eye region, align adjacent eye images according to their order, crop the eye region images, and generate a set of eye sequences, i.e., a continuous frame eye training dataset.

[0078] Furthermore, after determining the starting frame, the terminal device selects n consecutive frames of human eye images, performs grayscale processing on all human eye images, and then generates a human eye frame data set D0. Then, the terminal device can set the frame step size. For example, when the frame step size is 1, the terminal device selects human eye images from the human eye frame data set D0 at intervals of one frame to construct a continuous frame human eye training dataset, thereby selecting k sets of continuous frame human eye training datasets D = {Di|i = 0, 1, 2, ..., k} of the same length as the training set for the behavior analysis model.

[0079] It should be noted that in other embodiments, the continuous frame human eye training data set can also directly use color images, that is, no grayscale processing is required, and the behavior analysis model can also train the behavior analysis function for color images. Different models and different algorithms may have certain differences in input requirements for image data, such as RGB color images, grayscale images, texture feature images, infrared light images, event camera images, and the like. The embodiments of the present application are only an example of grayscale images, and can also be extended to the above-mentioned multiple feature images. Among them, the grayscale image can be regarded as a grayscale processing of the RGB color image, which is a single-channel image.

[0080] Specifically, as shown in Figure 3 , the terminal device uses a face alignment algorithm to correct the continuous multiple frames of human eye images, for example, uses the rotation operation shown in Figure 3 to perform spatial pose transformation on the human eye images, so that the directions of the eye regions in the corrected human eye images are consistent.

[0081] Then, the terminal device can crop the corrected human eye images according to the eye regions to obtain human eye region images, and use the human eye region images to construct a continuous frame human eye training data set. Specifically, the terminal device expands the human eye region outward on the basis of the detected human eye frame, that is, Figure 3 , so that the human eye frame is expanded to a square, that is, a human eye frame with an aspect ratio of 1, and then the human eye region image is extracted.

[0082] Further, after preparing the continuous frame human eye training data set, the terminal device also needs to manually label the state of each human eye image in the continuous frame human eye training data set, and the labeling value can be adjusted according to different needs or habits.

[0083] Specifically, as shown in Figure 4 , the Figure 4 is a set of human eye frame sequence annotations. Figure 4 The human eye frame sequence is composed of 16 frames of human eye images with frame numbers from 0 to 15, wherein the class label of the human eye image representing open eyes is labeled as 0, and the class label of the human eye image representing closed eyes is labeled as 1.

[0084] Among them, the human eye frame sequence annotation is that starting from the starting frame, after collecting n frames of samples, the state value of the human eye image is manually labeled according to the state of the human eye in each frame of sample, such as open eyes as 0, closed eyes as 1, and squinting eyes as 0.5. It should be noted that the regression value is necessary for the training regression task of the trained analysis model, and secondly, the annotation of the above-mentioned regression value is not limited to the values 0, 0.5, 1, etc. It can also be other numerical values or forms, which are not listed here.

[0085] Step S12: Generate a human eye cluster image based on the continuous frame human eye training dataset.

[0086] In this embodiment, the terminal device scales each group of human eye images in the continuous frame human eye training dataset D to a uniform size and generates feature maps, i.e., 2D human eye cluster images, according to certain image encoding rules. Specifically, as follows... Figure 5 As shown, Figure 5 (a) is an image of the human eye cluster. Figure 5 (b) is the encoding format of the human eye image. Figure 5 (c) represents the actual eye behavior labels, which are generated by combining the annotation results of the human eye frame sequence according to the image coding rules.

[0087] Specifically, the image encoding rule can be to arrange several human eye images from a continuous frame human eye training dataset according to the order of their acquisition time, such as... Figure 5 In (b), from top to bottom and from left to right, sequentially... Figure 4 Encode 16 frames of human eye images to generate Figure 5 (a) A cluster image of the human eye.

[0088] Specifically, human eye cluster image encoding involves generating a new image based on labeled consecutive frames of human eye images according to certain encoding rules. Each eye image corresponds to an original frame number such as... Figure 5 As shown in (b). Accordingly, the manually annotated pixel-level state values ​​in the human eye cluster image are as follows: Figure 5 As shown in (c), these represent the human eye states in the region, including open eyes, closed eyes, and squinting, which serve as the basis for training the analysis model.

[0089] It should be noted that the image encoding method is not unique. For example, this application can also use methods such as rotating, flipping, or mirroring the image to change the encoding rules and distribute the image, thereby obtaining different human eye cluster images.

[0090] Furthermore, this application selects consecutive images based on time sequence, and then arranges the selected consecutive images according to... Figure 5 (b) uses an encoding method to encode images, combining them into a complete human eye cluster image, rather than a set of images. The human eye cluster image contains both the spatial features of a single human eye image and the continuous temporal features of multiple human eye images, which can solve the problems of parameter redundancy and network time consumption in the 3D convolution process. Furthermore, the behavior analysis model in this application only requires a single human eye cluster image as input, rather than individual human eye images within a cluster, which offers a performance advantage.

[0091] Step S13: inputting the human eye cluster image into the analysis model to be trained to obtain the eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification includes an eye behavior category and a first prediction probability of the eye behavior category.

[0092] In the embodiment of the present application, the terminal device inputs the human eye cluster image generated in step S12 into the analysis model to be trained to extract the image features of the human eye cluster image, and outputs the eye behavior classification of the continuous frame human eye training data set according to the image features of the human eye cluster image, wherein the eye behavior classification includes an eye behavior category and a prediction probability of the eye behavior category.

[0093] Specifically, the eye behavior category in the embodiment of the present application includes but is not limited to: open eyes, closed eyes, blinking, etc.

[0094] Step S14: obtaining a first prediction loss of the analysis model to be trained based on the eye behavior classification of the continuous frame human eye training data set and the real eye behavior label, and training the analysis model to be trained by using the first prediction loss to obtain a final behavior analysis model.

[0095] In the embodiment of the present application, the analysis model to be trained constructs the first prediction loss according to the difference between the eye behavior classification of the continuous frame human eye training data set and the artificially annotated real eye behavior label, and trains the analysis model to be trained based on the first prediction loss, thereby training the final behavior analysis model.

[0096] In the embodiment of the present application, the terminal device constructs a continuous frame human eye training data set, generates a human eye cluster image based on the continuous frame human eye training data set, inputs the human eye cluster image into the analysis model to be trained to obtain the eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification includes an eye behavior category and a first prediction probability of the eye behavior category, obtains a first prediction loss of the analysis model to be trained based on the eye behavior classification of the continuous frame human eye training data set and the real eye behavior label, and trains the analysis model to be trained by using the first prediction loss to obtain a final behavior analysis model. The behavior analysis model training method of the present application can effectively reduce the parameter redundancy, complex calculation and storage consumption of model training, and improve the efficiency of model training by encoding a plurality of human eye image frames of the continuous frame human eye training data set into the same human eye cluster image.

[0097] Further, the behavior analysis model of the present application can be divided into a first analysis model to be trained, i.e. a state logic clue perception network, and a second analysis model to be trained, i.e. a blinking behavior analysis network.

[0098] For details, please refer to Figure 6 and Figure 7 , Figure 6 is a flowchart of another embodiment of the behavior analysis model training method provided by the present application, Figure 7is a data flow diagram of the behavior analysis model provided in the present application.

[0099] As shown in Figure 6 , the specific steps of the behavior analysis model training method of the embodiment of the present application are as follows:

[0100] Step S21: input the human eye cluster image into the first to-be-trained analysis model to obtain a predicted human eye state logical clue feature map, wherein the predicted human eye state logical clue feature map includes a first prediction probability of each human eye image frame.

[0101] In the embodiment of the present application, as shown in Figure 7 , the terminal device inputs the human eye cluster image into the state logical clue perception network to obtain a predicted human eye state logical clue feature map. Wherein, please refer to Figure 8 , Figure 8 for an example diagram of the human eye state logical clue feature map output by the state logical clue perception network. The visualization result of the human eye state logical clue feature map on the human eye cluster image is shown in the figure, where region A represents a region with a closed eye state, region B represents a region with an open eye state, and region C represents a transition region from an open eye state to a closed eye state.

[0102] Taking the human eye cluster image as input, the state logical clue perception network with Hourglass network as backbone is used to regress the eye region containing the closed eye state, and the human eye state logical clue feature map is obtained.

[0103] Specifically, the state logical clue perception network includes shallow convolution kernels and deep convolution kernels, and the image features of the human eye cluster image are extracted by using the shallow convolution kernels and the deep convolution kernels. The state logical clue perception network uses the shallow convolution kernels to extract shallow features on the human eye cluster image with a fixed step size, i.e. local features of the human eye cluster image, which are used to represent the feature relationship of adjacent human eye images. The state logical clue perception network uses the deep convolution kernels to extract deep features, i.e. global features, which are used to represent the feature relationship of human eye images in the human eye cluster image. Wherein, with the expansion of the perception area, the deep convolution kernels can abstract high-order semantics of the image by integrating shallow texture and deep temporal features.

[0104] Since the human eye cluster image has both single-frame texture information and continuous interval 16-frame temporal information, through 2D convolution, pooling, upsampling and other processing, the human eye state logical clue feature map with the same scale as the input image can be finally output, as shown in Figure 8 , wherein the larger the area A is, the higher the probability of being closed eye is, and the larger the area B is, the higher the probability of being open eye is.

[0105] Step S22: Obtain a second prediction loss of the first to-be-trained analysis model based on the predicted human eye state logical clue feature map and the real human eye state logical clue feature map, and train the first to-be-trained analysis model by using the second prediction loss.

[0106] In the embodiments of the present application, the state logical clue perception network is based on Figure 8 the predicted human eye state logical clue feature map and Figure 5 the real human eye state logical clue feature map of (c) to calculate a prediction loss, and the prediction loss is used for model training.

[0107] Step S23: Input the predicted human eye state logical clue feature map into the second to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification includes an eye behavior category and a third prediction probability thereof.

[0108] In the embodiments of the present application, as shown in Figure 7 , the human eye state logical clue feature map is used as the input of the blink behavior analysis network to classify the eye behavior in the input video frame sequence. During the process, the state logical clue perception network uses a regression loss as supervision, and the blink behavior analysis network uses a cross-entropy loss as a supervision signal to promote network optimization and convergence.

[0109] It should be noted that the state logical clue perception network is a convolutional neural network based on human eye cluster images, which abstracts the features in time sequence and space of the human eye cluster images, extracts sequence behavior changes and spatial distribution features, and thus judges whether a specified action occurs in a time sequence. It is different from the network based on graph convolution. In addition, the blink behavior analysis network as an analysis tool of the human eye state logical clue feature map can use, but is not limited to, a public classification network such as VGG, ResNet, etc.

[0110] Step S24: Obtain a third prediction loss of the second to-be-trained analysis model based on the eye behavior classification of the continuous frame human eye training data set and the real eye behavior label, and train the second to-be-trained analysis model by using the third prediction loss.

[0111] In the embodiments of the present application, the use of feature point information is avoided as much as possible, a light-weight eye detection network and a spatial pose correction method are relied on to obtain human eye region data, so as to avoid the influence of feature point positioning error on the algorithm performance and improve the algorithm accuracy and robustness; multi-frame data is used to avoid the loss of time sequence information, at the same time, a state logical clue perception network is used to perceive the human eye state in the sequence and complete blink behavior analysis, so as to avoid the influence of too long history state. This method not only reduces the difficulty of data labeling, but also can introduce effective time sequence features, thereby improving the algorithm performance; through image coding technology, the use of 3D convolution kernel is avoided, and instead, the characteristics of 2D convolution kernel parameter sharing are used to avoid the large redundancy of model parameters, at the same time, the features in time sequence and texture are retained. In fact, compared with 3D convolution, 2D convolution has great improvement in parameter redundancy, complex calculation and storage consumption, and is convenient for application in actual scenes; by encoding the continuous frame human eye training data set into a complete image and inputting it at one time, there is great advantage in effective coding, effective use and performance of information.

[0112] Please continue to refer to Figure 9 , Figure 9 is a flowchart of an embodiment of the behavior analysis method provided by the present application.

[0113] As shown in Figure 9 , the specific steps of the behavior analysis model training method of the embodiments of the present application are as follows:

[0114] Step S31: Obtain a monitoring video stream.

[0115] Step S32: Input the monitoring video stream into the behavior analysis model, obtain the output behavior analysis category, and the prediction probability of each category.

[0116] Step S33: Select the behavior analysis category with the highest prediction probability as the behavior analysis result of the monitoring video stream.

[0117] For any inputted face-containing video stream, the terminal device uses the same face alignment algorithm as the training process to locate and align the eye image, and intercepts the eye region. Then, after determining the starting frame, n consecutive eye images are selected, all eye images are respectively subjected to grayscale processing, and then eye frame data groups are generated. Finally, the eye cluster image is generated by the same method as the training process.

[0118] The terminal device loads the state logical clue perception network and the blink behavior analysis network trained to the convergent state, takes the eye cluster image as the input of the state logical clue perception network, generates the eye state logical clue feature map, and takes the eye state logical clue feature map as the input of the blink behavior analysis network, and finally obtains the probability of each behavior category.

[0119] The terminal device selects the behavior with the maximum probability value in each behavior as the result of algorithm analysis, and takes the probability value as the confidence degree of the behavior occurrence. The final blinking result is output by comparing with a threshold value.

[0120] Specifically, the blinking behavior analysis is completed based on a blinking behavior analysis network taking a human eye state logical clue feature map as input. The network takes a human eye state map output by a state logical clue perception network as input, extracts abstract features, and finally obtains each blinking behavior probability.

[0121] In the present application, the blinking behavior includes but is not limited to opening eyes, closing eyes, squinting eyes, blinking eyes, etc. On the ground level, the behavior with the maximum probability will be selected. When the behavior probability value exceeds a set threshold value, it is considered that the behavior occurs, otherwise it is considered that the blinking event does not occur.

[0122] The above embodiments are only one common case of the present application, and do not limit the technical scope of the present application. Therefore, any slight modification, equivalent change or modification of the above content according to the essence of the present application still belongs to the scope of the technical scheme of the present application.

[0123] Please refer to Figure 10 , Figure 10 is a structural schematic diagram of an embodiment of the terminal device provided by the present application. The terminal device includes a memory 52 and a processor 51 connected to each other.

[0124] The memory 52 is used to store program instructions for implementing any one of the above behavior analysis model training methods and / or behavior analysis methods.

[0125] The processor 51 is used to execute the program instructions stored in the memory 52.

[0126] The processor 51 can also be called a CPU (Central Processing Unit). The processor 51 can be an integrated circuit chip with a processing capability of signaling. The processor 51 can also be a general-purpose processor, a digital signaling processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0127] The memory 52 can be a memory stick, a TF card, etc., and can store all information in the terminal device, including input raw data, computer programs, intermediate running results and final running results. It stores and retrieves information according to the specified location of the controller. With the memory, the terminal device has a memory function and can work normally. The memory of the terminal device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to the purpose, and there is also a classification method of dividing external memory and internal memory. The external memory is usually a magnetic medium or an optical disc, etc., which can store information for a long time. The internal memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but only used to temporarily store programs and data, and the data will be lost when the power is off.

[0128] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic, for example, the division of models or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0129] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0130] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0131] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a system server or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application.

[0132] Referring to Figure 11 , a structural diagram of a computer readable storage medium of the present application. The storage medium of the present application stores a program file 61 capable of realizing all the behavior analysis model training methods and / or behavior analysis methods described above, wherein the program file 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method of each embodiment of the present application. And the storage device mentioned above includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various media that can store program codes, or computer, server, mobile phone, tablet and other terminal devices.

[0133] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for training a behavior analysis model, characterized in that, The behavior analysis model training method comprises: constructing a continuous frame human eye training data set, wherein the continuous frame human eye training data set comprises a plurality of continuous human eye image frames; encoding the plurality of human eye image frames in the continuous frame human eye training data set into the same image according to a preset image encoding rule to generate a human eye cluster image; inputting the human eye cluster image into a to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification comprises an eye behavior category and a first prediction probability thereof; obtaining a first prediction loss of the to-be-trained analysis model based on the eye behavior classification of the continuous frame human eye training data set and a real eye behavior label, and training the to-be-trained analysis model by using the first prediction loss to obtain a final behavior analysis model.

2. The behavior analysis model training method according to claim 1, wherein the continuous frame human eye training data set is constructed by: detecting human eye images in a monitoring video stream by using an eye detection network to determine human eye images comprising eye regions as starting frames; selecting continuous multiple frames of human eye images based on the starting frames to form the continuous frame human eye training data set.

3. The behavior analysis model training method according to claim 2, wherein the continuous frame human eye training data set is formed by selecting continuous multiple frames of human eye images based on the starting frames, comprising: selecting continuous preset frame numbers of human eye images based on the starting frames according to a preset frame step to form the continuous frame human eye training data set.

4. The behavior analysis model training method according to claim 2, wherein the continuous frame human eye training data set is formed by selecting continuous multiple frames of human eye images based on the starting frames, comprising: selecting continuous multiple frames of human eye images based on the starting frames; positioning eye regions in the human eye images by using the eye detection network; cropping the human eye images according to the eye regions to obtain human eye region images; forming the continuous frame human eye training data set based on the continuous multiple frames of human eye region images.

5. The behavior analysis model training method according to claim 4, wherein the human eye region images are obtained by cropping the human eye images according to the eye regions, comprising: correcting the continuous multiple frames of human eye images by using a face alignment algorithm so that the directions of the eye regions in the corrected human eye images are consistent; cropping the corrected human eye images according to the eye regions to obtain the human eye region images.

6. The behavior analysis model training method according to claim 1, wherein the preset image encoding rule comprises a time sequence of acquisition of the plurality of human eye image frames in the continuous frame human eye training data set.

7. The behavior analysis model training method according to claim 1, wherein the to-be-trained analysis model comprises a first to-be-trained analysis model and a second to-be-trained analysis model; and the behavior analysis model training method further comprises: ​ ​ ​ ​ ​ ​ ​ inputting the human eye cluster image into a first to-be-trained analysis model to obtain a predicted human eye state logical clue feature map, wherein the predicted human eye state logical clue feature map comprises a second predicted probability of each human eye image frame; obtaining a second prediction loss of the first to-be-trained analysis model based on the predicted human eye state logical clue feature map and a real human eye state logical clue feature map, and training the first to-be-trained analysis model by using the second prediction loss; inputting the predicted human eye state logical clue feature map into a second to-be-trained analysis model to obtain an eye behavior classification of the continuous frame human eye training data set, wherein the eye behavior classification comprises an eye behavior category and a third predicted probability thereof; obtaining a third prediction loss of the second to-be-trained analysis model based on the eye behavior classification of the continuous frame human eye training data set and a real eye behavior label, and training the second to-be-trained analysis model by using the third prediction loss.

8. The behavior analysis model training method according to claim 7, wherein the first to-be-trained analysis model comprises a shallow convolution kernel and a deep convolution kernel; the inputting the human eye cluster image into a first to-be-trained analysis model to obtain a predicted human eye state logical clue feature map comprises: inputting the human eye cluster image into the first to-be-trained analysis model; extracting local features of the human eye cluster image by using the shallow convolution kernel, wherein the local features represent feature relationships of adjacent human eye image frames; extracting global features of the human eye cluster image by using the deep convolution kernel, wherein the global features represent feature relationships of human eye image frames in the human eye cluster image; generating the predicted human eye state logical clue feature map based on the local features and the global features of the human eye cluster image.

9. A method of behavior analysis, characterized by, the behavior analysis method comprises: obtaining a monitoring video stream; inputting the monitoring video stream into the behavior analysis model to obtain an output behavior analysis category and a predicted probability of each category; selecting a behavior analysis category with the highest predicted probability as a behavior analysis result of the monitoring video stream; wherein the behavior analysis model is trained by the behavior analysis model training method in any one of claims 1 to 8.

10. The behavior analysis method according to claim 9, wherein the selecting a behavior analysis category with the highest predicted probability as a behavior analysis result of the monitoring video stream comprises: determining whether the predicted probability of the behavior analysis category with the highest predicted probability is greater than or equal to a preset probability threshold; if yes, outputting the behavior analysis category as the behavior analysis result of the monitoring video stream; if no, outputting a behavior analysis result of no occurrence of the behavior analysis category event.

11. A terminal device, comprising: The terminal device comprises a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is configured to execute the program instructions stored in the memory to implement the behavior analysis model training method in any one of claims 1 to 8 and / or the behavior analysis method in any one of claims 9 to 10.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program instructions, which, when executed, implement the behavior analysis model training method of any one of claims 1-8 and / or the behavior analysis method of any one of claims 9-10.

Images