Abnormal behavior warning method based on AI analysis and prediction

Abstract

The invention discloses a behavior abnormity alarm method based on AI analysis and prediction, and relates to the technical field of image processing. The method comprises the following steps: S1, collecting related behavior pictures according to a specific application scene, drawing a labeling box in the pictures, and inputting a classification label; s2, carrying out preprocessing operation on the image; s3, the displacement of two continuous frames of images is calculated, and an optical flow graph corresponding to the difference of adjacent video frames is obtained; s4, carrying out feature vector extraction on the optical flow graph to train an abnormal behavior recognition model; s5, performing training test on the abnormal behavior recognition model by using continuous video data; and S6, the abnormal behavior recognition model recognizes behaviors needing to be recognized. The abnormal behavior recognition module is trained after a large number of behavior pictures are obtained and marked, human body behaviors are recognized and analyzed through the trained model, abnormal behavior conditions are recognized according to human body joint movement conditions, and the abnormal behavior recognition accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a behavior abnormality alarm method based on AI analysis and prediction. Background Art

[0002] Most traditional video surveillance systems are in a "record-only, no-judgment" mode, requiring humans to monitor video footage for long periods of time without interruption. This is not only a huge workload, but also easily affected by human sensory fatigue, resulting in missed detections and false detections. In addition, as the scale of surveillance systems continues to expand, the amount of video data increases, and it becomes increasingly difficult to obtain useful information from it, making it difficult to meet the demand for real-time alerts for abnormal behavior.

[0003] Some existing abnormal behavior detection methods are mainly based on preset rules and thresholds, and have poor adaptability to complex and changing behavior patterns. For example, in the field of network security, simply setting a fixed access frequency threshold or specific operation rules to judge abnormal behavior often cannot accurately identify new attack methods or complex abnormal behavior patterns. In industrial production, fixed equipment failure thresholds may not be able to detect potential failure hazards in a timely manner, because the operating status of the equipment may be affected by a variety of factors, which are difficult to fully cover with simple rules. Summary of the invention

[0004] The purpose of the present invention is to provide a behavior abnormality alarm method based on AI analysis and prediction. By acquiring a large number of behavior pictures, annotating them and then training an abnormal behavior recognition module, the trained model is used to identify and analyze human behavior, thereby solving the problem of difficulty and low efficiency in existing abnormal behavior recognition.

[0005] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] The present invention is a behavior abnormality alarm method based on AI analysis and prediction, comprising the following steps:

[0007] Step S1: According to the specific application scenario, collect relevant behavior pictures, draw annotation boxes in the pictures, and enter classification labels;

[0008] Step S2: performing preprocessing operations on the image, including grayscale, normalization and sequence division;

[0009] Step S3: Calculate the displacement of two consecutive frames of images and obtain the optical flow map corresponding to the difference of adjacent video frames;

[0010] Step S4: extracting feature vectors from the optical flow map to train an abnormal behavior recognition model;

[0011] Step S5: using continuous video data to train and test the abnormal behavior recognition model;

[0012] Step S6: The abnormal behavior recognition model that has been tested identifies the behaviors that need to be identified, and performs real-time statistics at the bottom of the screen to calculate the proportion of each type of behavior in all behaviors.

[0013] As a preferred technical solution, in step S1, video data of the application scene is obtained, the video data is decomposed frame by frame, and after manually selecting complete human body pictures, the selected pictures are flipped, mirrored, and cropped, and then the picture size, range and specifications are adjusted; a labeling box is drawn in the image, and a classification label is entered and the labeling result is stored. After completion, switch to the next picture for labeling until all images are labeled.

[0014] As a preferred technical solution, in step S2, the specific steps of image preprocessing include:

[0015] Step S21: grayscale the image. Assume that the original image is I(x, y), grayscale is Gary(·), and the grayscale image is G(x, y). The specific calculation formula is as follows:

[0016] G(x,y)←Gary(I(x,y));

[0017] Step S22: formatting the grayscale image; assuming that the resizing function is Gray(·), the image obtained after adjustment is R1(x, y), and the specific calculation formula is as follows:

[0018] R1(x,y)←Re(G(x,y));

[0019] Step S23: Use the maximum and minimum values ​​of the feature to perform normalization processing and scale the feature value to the [Min, Max] interval. The specific calculation formula is as follows:

[0020]

[0021] Where R1 min is the minimum value in R1, R1 max is the maximum value in R1, and R(x, y) is the image after data normalization.

[0022] As a preferred technical solution, in step S3, the pixel point in the image after data normalization is taken as the center to construct a local coordinate system, and the displacement in two adjacent frames of images is calculated using the binomial expansion method. t The expansion of (x, y) is the spatial information of the frame image at time t, and the displacement evaluation is performed. The specific formula is as follows:

[0023]

[0024] In the formula, x is a two-dimensional vector, (x, y) T represents two-dimensional coordinates, A represents a 2×2 symmetric matrix, b represents a 2×1 matrix, and c represents a scalar;

[0025] Then R t The next frame image R is obtained after the image pixels of the frame are shifted t+1 for:

[0026]

[0027] The displacement d generated by each pixel in the frame image is:

[0028]

[0029] As a preferred technical solution, in step S3, the pixel point in the image after data normalization is taken as the center to construct a local coordinate system, and the displacement in two adjacent frames of images is calculated using the binomial expansion method. t The expansion of (x, y) is the spatial information of the frame image at time t, and the displacement evaluation is performed. The specific formula is as follows:

[0030]

[0031] In the formula, x is a two-dimensional vector, (x, y) T represents two-dimensional coordinates, A represents a 2×2 symmetric matrix, b represents a 2×1 matrix, and c represents a scalar;

[0032] Then R t The next frame image R is obtained after the image pixels of the frame are shifted t+1 for:

[0033]

[0034] The displacement d generated by each pixel in the frame image is:

[0035]

[0036] As a preferred technical solution, in step S4, when extracting feature vectors from the optical flow map, the displacement trajectory of each pixel in the optical flow map is assumed to be a point sequence trj= <P1,P2,...,P i ,...,P n >, where P i represents the i-th point, expressed as (x i ,y i ,t i ), xi is the horizontal coordinate of the i-th point, y i is the ordinate of the i-th point, t i is the timestamp of the i-th point; then calculate the point sequence speed v of the trajectory i , acceleration a i and curvature s i , the specific calculation formula is as follows:

[0037] Δt i =t i+1 -t i

[0038]

[0039] In the formula, dist(P i ,P i+1 ) is P i ,P i+1 The Euclidean distance between i is the time interval between time i+1 and time i.

[0040] As a preferred technical solution, in step S4, the abnormal behavior recognition module is obtained by training a convolutional neural network; the convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer connected thereto;

[0041] The input layer is used to input the feature vector extracted from the optical flow map;

[0042] The calculation formula of the convolutional layer is as follows:

[0043]

[0044] In the formula, is the input value of the (i,j)th point in the lth layer, is the output value of the (i, j)th point in the lth layer, mh is the height value of the local receptive field in the lth layer, nw is the width value of the local receptive field in the lth layer, is the value of the convolution kernel (i, j) point in the kth layer, b l is the offset of the lth layer;

[0045] The pooling layer is used to reduce the dimension of the features. The optical flow map feature map is pooled twice, and the size is half of the previous input feature map. The calculation formula is as follows:

[0046] Y i,j =maxX m,n ;

[0047] In the formula, X m,n Indicates the current input value, Y i,j Represents the output value after pooling;

[0048] The connection layer is used to linearly combine the features extracted by the convolution layer and perform a highly linear transformation;

[0049] The output layer uses the softmax function to calculate the probability value of each category. The specific formula is as follows:

[0050]

[0051] In the formula, y i is the output probability value of softmax, x i is the i-th eigenvalue of the softmax input, x j is the j-th eigenvalue of the softmax input, and n is the number of softmax input values.

[0052] As a preferred technical solution, in step S5, the training test of the abnormal behavior recognition model is mainly judged from the accuracy Precision, recall Recall and F value. The specific judgment formula is as follows:

[0053]

[0054] In the formula, TP is a true positive example, FP is a false positive example, and FN is a false negative example.

[0055] The present invention has the following beneficial effects:

[0056] (1) The present invention obtains a large number of behavior images, annotates them, and then trains an abnormal behavior recognition module. The trained model is used to recognize and analyze human behavior, and abnormal behavior is identified based on the movement of human joints, thereby improving the accuracy of abnormal behavior recognition.

[0057] (2) The present invention marks the human joints and movements in the image, identifies human abnormalities, and promptly warns of abnormal or dangerous behaviors, quickly notifies relevant personnel to check, and improves supervision.

[0058] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0060] Figure 1This is a flow chart of an abnormal behavior alarm method based on AI analysis and prediction of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The application environment of this embodiment is as follows: The specific experimental environment parameters of this article are as follows: CPU is Intel Core i9-11900F @ 2.50GHz 8 cores; GPU is Nvidia Ge□Force 3060 11G; operating system is Ubuntu 20.04; programming language is Python 3.7; deep learning framework pytorch is used to establish a network model for training and testing, where the batch_size of the two network models is 64, the learning rate of the CNN network model is 0.001, and the learning rate of the LSTM network model is 0.01.

[0065] Experimental dataset:

[0066] The dataset contains 60 types of actions, including 40 types of daily behavior actions, 9 types of health-related actions, and 11 types of two-person interactions. The self-built behavior dataset is constructed by crawling online videos and shooting by ourselves. There are 5 types of actions in the self-built behavior dataset, and the data volume of each type of action is about 50 to 80 corresponding video clips. Each original video clip is processed by the script to a unified format of 340×256 with a frame rate of 30fps. Finally, the processed video is used to extract the skeleton data through the OpenPose algorithm to create a dataset.

[0067] See also Figure 1 As shown, the present invention is a behavior abnormality alarm method based on AI analysis and prediction, comprising the following steps:

[0068] Step S1: According to the specific application scenario, collect relevant behavior pictures, draw annotation boxes in the pictures, and enter classification labels;

[0069] Step S2: performing preprocessing operations on the image, including grayscale, normalization and sequence division;

[0070] Step S3: Calculate the displacement of two consecutive frames of images and obtain the optical flow map corresponding to the difference of adjacent video frames;

[0071] Step S4: extracting feature vectors from the optical flow map to train an abnormal behavior recognition model;

[0072] Step S5: using continuous video data to train and test the abnormal behavior recognition model;

[0073] Step S6: The abnormal behavior recognition model that has been tested identifies the behaviors that need to be identified, and performs real-time statistics at the bottom of the screen to calculate the proportion of each type of behavior in all behaviors.

[0074] In step S1, the video data of the application scene is obtained, the video data is decomposed frame by frame, and after the complete human body pictures are manually selected, the selected pictures are flipped, mirrored, and cropped, and then the picture size, range and specifications are adjusted; a labeling box is drawn in the image, and the classification label is entered and the labeling result is stored. After completion, the next picture is switched to labeling until all images are labeled; according to different human behaviors of multiple different subjects in the data set, including sitting (a), running (b), standing (c), turning (d), walking (e), lying down (f), jumping (g), backward (h) and other behavior pictures.

[0075] In step S2, the specific steps of image preprocessing include:

[0076] Step S21: grayscale the image. Assume that the original image is I(x, y), grayscale is Gary(·), and the grayscale image is G(x, y). The specific calculation formula is as follows:

[0077] G(x,y)←Gary(I(x,y));

[0078] Step S22: formatting the grayscale image; assuming that the resizing function is Gray(·), the image obtained after adjustment is R1(x, y), and the specific calculation formula is as follows:

[0079] R1(x,y)←Re(G(x,y));

[0080] Step S23: Use the maximum and minimum values ​​of the feature to perform normalization processing and scale the feature value to the [Min, Max] interval. The specific calculation formula is as follows:

[0081]

[0082] Where R1 min is the minimum value in R1, R1 max is the maximum value in R1, and R(x, y) is the image after data normalization.

[0083] In order to reduce the difficulty of extracting neural network features, image simplification can be used to reduce the computational burden. The specific operations are as follows:

[0084] (1) First remove the palm and foot joints;

[0085] (2) Move the skeleton so that the joint point spinebase is located at the origin of the coordinate system;

[0086] (3) Performing rotation operations on human bones;

[0087] (4) Normalize the distance between adjacent joints so that the size of the human skeleton does not affect posture judgment.

[0088] In step S3, the pixel point in the image after data normalization is taken as the center to construct a local coordinate system, and the displacement in two adjacent frames of images is calculated using the binomial expansion method. t The expansion of (x, y) is the spatial information of the frame image at time t, and the displacement evaluation is performed. The specific formula is as follows:

[0089]

[0090] In the formula, x is a two-dimensional vector, (x, y) T represents two-dimensional coordinates, A represents a 2×2 symmetric matrix, b represents a 2×1 matrix, and c represents a scalar;

[0091] Then R t The next frame image R is obtained after the image pixels of the frame are shifted t+1 for:

[0092]

[0093] The displacement d generated by each pixel in the frame image is:

[0094]

[0095] In step S3, the pixel point in the image after data normalization is taken as the center to construct a local coordinate system, and the displacement in two adjacent frames of images is calculated using the binomial expansion method. t The expansion of (x, y) is the spatial information of the frame image at time t, and the displacement evaluation is performed. The specific formula is as follows:

[0096]

[0097] In the formula, x is a two-dimensional vector, (x, y) T represents two-dimensional coordinates, A represents a 2×2 symmetric matrix, b represents a 2×1 matrix, and c represents a scalar;

[0098] Then R t The next frame image R is obtained after the image pixels of the frame are shifted t+1 for:

[0099]

[0100] The displacement d generated by each pixel in the frame image is:

[0101]

[0102] In step S4, when extracting feature vectors from the optical flow map, the displacement trajectory of each pixel in the optical flow map is assumed to be a point sequence trj= <P1,P2,...,P i ,...,P n >, where P i represents the i-th point, expressed as (x i ,y i ,t i ), x i is the horizontal coordinate of the i-th point, y i is the ordinate of the i-th point, t i is the timestamp of the i-th point; then calculate the point sequence speed v of the trajectory i , acceleration a i and curvature s i , the specific calculation formula is as follows:

[0103] Δt i =t i+1 -t i

[0104]

[0105] In the formula, dist(P i ,P i+1 ) is P i ,P i+1 The Euclidean distance between i is the time interval between time i+1 and time i.

[0106] In step S4, the abnormal behavior recognition module is obtained by training a convolutional neural network; the convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer connected thereto;

[0107] The input layer is used to input the feature vector extracted from the optical flow map;

[0108] The calculation formula of the convolutional layer is as follows:

[0109]

[0110] In the formula, is the input value of the (i,j)th point in the lth layer, is the output value of the (i, j)th point in the lth layer, mh is the height value of the local receptive field in the lth layer, nw is the width value of the local receptive field in the lth layer, is the value of the convolution kernel (i, j) point in the kth layer, b l is the offset of the lth layer;

[0111] The pooling layer is used to reduce the dimension of features. The optical flow map feature map is pooled twice, and the size is half of the previous input feature map. The calculation formula is as follows:

[0112] Y i,j =maxX m,n ;

[0113] In the formula, X m,n Indicates the current input value, Y i,j Represents the output value after pooling;

[0114] The connection layer is used to linearly combine the features extracted by the convolutional layer and perform a highly linear transformation;

[0115] The output layer uses the softmax function to calculate the probability value of each category. The specific formula is as follows:

[0116]

[0117] In the formula, y i is the output probability value of softmax, x i is the i-th eigenvalue of the softmax input, x j is the j-th eigenvalue of the softmax input, and n is the number of softmax input values.

[0118] In step S5, the training test of the abnormal behavior recognition model is mainly judged from the accuracy, recall and F value. The specific judgment formula is as follows:

[0119]

[0120] In the formula, TP is a true positive example, FP is a false positive example, and FN is a false negative example.

[0121] It is worth noting that in the above system embodiment, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0122] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium.

[0123] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A behavior abnormality alarm method based on AI analysis and prediction, characterized in that: The steps include: Step S1: According to the specific application scenario, collect relevant behavior pictures, draw annotation boxes in the pictures, and enter classification labels; Step S2: performing preprocessing operations on the image, including grayscale, normalization and sequence division; Step S3: Calculate the displacement of two consecutive frames of images and obtain the optical flow map corresponding to the difference of adjacent video frames; Step S4: extracting feature vectors from the optical flow map to train an abnormal behavior recognition model; Step S5: using continuous video data to train and test the abnormal behavior recognition model; Step S6: The abnormal behavior recognition model that has been tested identifies the behaviors that need to be identified, and performs real-time statistics at the bottom of the screen to calculate the proportion of each type of behavior in all behaviors.

2. According to claim 1, a behavior abnormality alarm method based on AI analysis and prediction is characterized in that: In step S1, the video data of the application scene is obtained, the video data is decomposed frame by frame, and after the pictures are manually selected, they are flipped, mirrored, and cropped, and then the picture size, range and specifications are adjusted; a labeling box is drawn in the image, and the classification label is entered and the labeling result is stored. After completion, the next picture is switched to labeling until all images are labeled.

3. The abnormal behavior alarm method based on AI analysis and prediction according to claim 1 is characterized in that: In step S2, the specific steps of image preprocessing include: Step S21: grayscale the image. Assume that the original image is I(x, y), grayscale is Gary(·), and the grayscale image is G(x, y). The specific calculation formula is as follows: G(x,y)←Gary(I(x,y)); Step S22: formatting the grayscale image; assuming that the resizing function is Gray(·), the image obtained after adjustment is R1(x, y), and the specific calculation formula is as follows: R1(x,y)←Re(G(x,y)); Step S23: Use the maximum and minimum values ​​of the feature to perform normalization processing and scale the feature value to the [Min, Max] interval. The specific calculation formula is as follows: Where R1 min is the minimum value in R1, R1 max is the maximum value in R1, and R(x, y) is the image after data normalization.

4. The abnormal behavior alarm method based on AI analysis and prediction according to claim 1 is characterized in that: In step S3, the pixel point in the image after data normalization is taken as the center to construct a local coordinate system, and the displacement in two adjacent frames of images is calculated using the binomial expansion method. t The expansion of (x, y) is the spatial information of the frame image at time t, and the displacement evaluation is performed. The specific formula is as follows: R t (x)=x T A1x+b1 T x+c1; In the formula, x is a two-dimensional vector, (x, y) T represents two-dimensional coordinates, A represents a 2×2 symmetric matrix, b represents a 2×1 matrix, and c represents a scalar; Then R t The next frame image R is obtained after the image pixels of the frame are shifted t+1 for: The displacement d generated by each pixel in the frame image is:

5. The abnormal behavior alarm method based on AI analysis and prediction according to claim 1 is characterized in that: In step S4, when extracting feature vectors from the optical flow map, the displacement trajectory of each pixel in the optical flow map is assumed to be a point sequence trj= <P1,P2,...,P i ,...,P n >, where P i represents the i-th point, expressed as (x i ,y i ,t i ), x i is the horizontal coordinate of the i-th point, y i is the ordinate of the i-th point, t i is the timestamp of the i-th point; then calculate the point sequence speed v of the trajectory i , acceleration a i and curvature s i , the specific calculation formula is as follows: In the formula, dist(P i ,P i+1 ) is P i ,P i+1 The Euclidean distance between i is the time interval between time i+1 and time i.

6. The abnormal behavior alarm method based on AI analysis and prediction according to claim 1 is characterized in that: In step S4, the abnormal behavior recognition module is obtained by training a convolutional neural network; the convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer connected thereto; The input layer is used to input the feature vector extracted from the optical flow map; The calculation formula of the convolutional layer is as follows: In the formula, is the input value of the (i,j)th point in the lth layer, is the output value of the (i, j)th point in the lth layer, mh is the height value of the local receptive field in the lth layer, nw is the width value of the local receptive field in the lth layer, is the value of the convolution kernel (i, j) point in the kth layer, b l is the offset of the lth layer; The pooling layer is used to reduce the dimension of the features. The optical flow map feature map is pooled twice, and the size is half of the previous input feature map. The calculation formula is as follows: Y i,j =maxX m,n ; Where, X m,n Indicates the current input value, Y i,j Represents the output value after pooling; The connection layer is used to linearly combine the features extracted by the convolution layer and perform a highly linear transformation; The output layer uses the softmax function to calculate the probability value of each category. The specific formula is as follows: In the formula, y i is the output probability value of softmax, x i is the i-th eigenvalue of the softmax input, x j is the j-th eigenvalue of the softmax input, and n is the number of softmax input values.

7. The abnormal behavior alarm method based on AI analysis and prediction according to claim 1 is characterized in that: In step S5, the training test of the abnormal behavior recognition model is mainly judged from the accuracy, recall and F value. The specific judgment formula is as follows: In the formula, TP is a true positive example, FP is a false positive example, and FN is a false negative example.

Images