A risk behavior monitoring method and system based on human motion recognition

Abstract

The present invention discloses a method and system for monitoring risky behaviors based on human motion recognition. The method comprises the following steps: separately collecting infrared image data and structured light image data of the object to be identified; fusing the infrared image data and the structured light image data and inputting them into a trained human detector to output a cropped image of the human body; inputting the cropped image into a trained human motion recognition model to output key point information of the human motion; and determining whether risky behaviors exist based on the key point information of the human motion. The present invention can achieve information complementarity between different modalities, improving the accuracy of human motion recognition under interference and complex lighting conditions, thereby increasing the accuracy of risky behavior judgments and helping to prevent and stop the occurrence of dangerous behaviors.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and more specifically, relates to a risk behavior monitoring method and system based on human motion recognition. Background Art

[0002] With the passage of time, students' mental health is receiving increasing attention. However, due to factors such as a lack of relevant teaching resources, students' mental health is often overlooked, and psychological problems often go unaddressed. This can easily lead students to engage in behaviors such as self-harm, suicide, or harming others. Because such behaviors often occur in low-traffic areas on campus, traditional manual inspections consume a large amount of manpower and material resources and are prone to misidentification. Currently, the main method for preventing and deterring such dangerous behavior is through the use of a large number of campus surveillance cameras.

[0003] Currently, there are two common methods for using surveillance cameras to monitor this type of dangerous behavior. One involves manual observation of the video feeds from campus surveillance cameras. This method consumes considerable human effort and offers low accuracy. The other involves using computers to process the video feeds and analyze student behavior to prevent and stop dangerous behavior. However, due to the lack of efficient and accurate technology for analyzing student behavior and the fact that most surveillance cameras use visible light, the accuracy of student behavior monitoring is low. This accuracy is further reduced in complex lighting conditions, such as low light levels. Summary of the Invention

[0004] In response to at least one defect or improvement need in the prior art, the present invention provides a risk behavior monitoring method and system based on human motion recognition, which can achieve information complementarity of different modalities, improve the accuracy of human motion recognition under interference and complex lighting conditions, thereby improving the accuracy of risk behavior judgment, and helping to prevent and stop the occurrence of dangerous behaviors.

[0005] To achieve the above object, according to a first aspect of the present invention, a risk behavior monitoring method based on human motion recognition is provided, comprising the steps of:

[0006] respectively collecting infrared image data and structured light image data of the object to be identified;

[0007] fusing the infrared image data and the structured light image data and inputting the data into a trained human body detector to output a human body cropped image;

[0008] Inputting the human body cropped image into a trained human action recognition model to output human action key point position information;

[0009] Determine whether there is risky behavior based on the human body movement key point information.

[0010] Preferably, the infrared image data and the structured light image data are preprocessed before the fusion, and the preprocessing includes performing image noise reduction processing on the infrared image data and the structured light image data.

[0011] Preferably, the image noise reduction process comprises the steps of:

[0012] The coordinates of a pixel in the image before denoising are marked as (p, q), and the pixel value is r(p, q). The pixel value of the pixel in the image after denoising is marked as R(p, q). The calculation formula of R(p, q) is:

[0013]

[0014]

[0015] Wherein, K(p′, q′) is the normal distribution equation of the pixel point (p′, q′), (p′, q′) is the coordinate of the pixel point within 3σ of the pixel point (p, q), r(p′, q′) is the pixel value of the pixel point (p′, q′), and ε is a preset positive constant.

[0016] Preferably, the human detector is implemented based on the YOLOv5 network, and the size of its human detection frame is 384×288.

[0017] Preferably, the human motion recognition model includes a backbone network and a heat map regression model. The heat map regression model is a key point heart-shaped heat map data generator. The backbone network uses the key point heart-shaped heat map generated by the heat map regression model to train the entire network. After training, the key point heart-shaped heat map in the picture is directly output based on the input picture.

[0018] Preferably, the key point position information is a key point heat map.

[0019] Preferably, the key point heat map is a key point heart-shaped heat map, and the key point heart-shaped heat map is composed of a circular heat map and two semi-elliptical heat maps.

[0020] Preferably, the determining whether risky behavior exists according to the key point information of the human body movement comprises the steps of:

[0021] Obtaining a motion risk score based on the key point information of the human body motion;

[0022] The action risk score is compared with a preset threshold to determine whether there is a risky behavior.

[0023] Preferably, obtaining the action risk score according to the human action key point information comprises the following steps:

[0024] The human body motion key point information is input into a multivariate linear regression model, the independent variable of the linear regression model is the human body motion key point information, and the dependent variable of the linear regression model is the motion risk score.

[0025] According to a second aspect of the present invention, a risk behavior monitoring system based on human motion recognition is provided, comprising:

[0026] An acquisition module, used to respectively acquire infrared image data and structured light image data of an object to be identified;

[0027] a cropping module, configured to fuse the infrared image data and the structured light image data and input the resultant data into a trained human body detector, and output a human body cropped image;

[0028] A key point acquisition module is used to input the human body cropped image into a trained human action recognition model and output human action key point information;

[0029] The judgment module is used to judge whether there is risky behavior based on the key point information of the human body movement.

[0030] In general, the present invention has the following advantages compared with the prior art:

[0031] (1) The present invention uses human motion recognition technology to predict whether students have dangerous behaviors, thereby improving the efficiency and accuracy of predicting dangerous behaviors using video data.

[0032] (2) The present invention improves the recognition accuracy of human body movements in the presence of interference and complex lighting environments, which is of great significance for accurately predicting students' dangerous behaviors in different environmental and time periods.

[0033] (3) The heart-shaped heat map proposed in this invention can improve the accuracy of heat map regression prediction, thereby improving the accuracy and speed of human motion recognition, which is of great significance for preventing and stopping students' dangerous behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a risk behavior monitoring method according to an embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of a key heart-shaped heat map in the present invention. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0037] like Figure 1 As shown, a risk behavior monitoring method based on human motion recognition according to an embodiment of the present invention includes the following steps:

[0038] S1, respectively collecting infrared image data and structured light image data of the object to be identified.

[0039] Furthermore, the objects to be identified can be students, children, or any other objects whose risky behaviors need to be monitored.

[0040] Furthermore, step S1 includes the following sub-steps:

[0041] S101, use an infrared camera (band 900nm) and a structured light camera to record learners' learning videos.

[0042] S102 , decomposing and matching the images frame by frame to obtain a dual-modal human body image at the same moment in the same scene.

[0043] S2, after fusing the infrared image data and the structured light image data, inputs them into the trained human body detector and outputs a human body cropped image.

[0044] Furthermore, before step S2, the method further includes the following steps: preprocessing the infrared image data and the structured light image data, fusing the preprocessed infrared image data and the structured light image data, and inputting the fused data into the trained human body detector.

[0045] Preprocessing includes image noise reduction on infrared image data and structured light image data

[0046] Furthermore, Gaussian filters are used to perform image noise reduction processing on the infrared image data and the structured light image data. The image noise reduction processing method is:

[0047] The coordinates of the pixel before image noise reduction are marked as (p, q), and the value is r(p, q). The value of the pixel (p, q) after image noise reduction is recorded as R(p, q). (p', q') are the coordinates of the pixel within 3σ of the pixel (p, q), and r(p', q') is the pixel value of the pixel (p', q'). R(p, q) can be calculated from r(p', q') using the following formula:

[0048]

[0049]

[0050] Where ε is a given positive constant, K(p′, q′) is the normal distribution equation of the pixel point (p′, q′), and S represents the set of all pixel coordinates.

[0051] Furthermore, the fusion may be performed by inputting the infrared image data and the structured light image data into a trained image fusion model to obtain a fused image of the infrared image data and the structured light image data.

[0052] Image fusion models can be implemented based on neural networks.

[0053] The fused image is then input into the human detector to obtain a human cropped image of the fused image.

[0054] The purpose of the human detector is to obtain a human cropped image from the fused image. A fused image may contain multiple people. The purpose of obtaining a human cropped image is to facilitate more accurate motion risk prediction results.

[0055] Furthermore, the human body detector is recorded as the Heart-YOLO network, which is implemented based on the YOLOv5 network. It adds detection modules of corresponding sizes after the convolutional layers of different sizes. Each detection module performs a detection and classification process to achieve unified image cropping.

[0056] S3, inputs the human body cropped image into the trained human action recognition model and outputs the key point information of the human action.

[0057] Furthermore, the key point information is a key point heat map. Unlike the common method of using unique position coordinates to mark key points, in the key point heat map, the key point is not a fixed point, but a probability distribution around a central point. Each point in the heat map distributed around the central point has a probability value of becoming a key point.

[0058] Furthermore, the key point heat map is a key point heart-shaped heat map, which is composed of a circular heat map and two semi-elliptical heat maps.

[0059] The following describes in detail how to obtain key point heart-shaped heat maps using a human action recognition model.

[0060] S301, constructing a training sample set, encoding the sample labels in the training sample set into a key point heart-shaped heat map.

[0061] Specifically, the COCO dataset images can be used as the heart-shaped heat map data source for training the human action recognition model, and the ordinary human body key points in the COCO dataset images can be encoded into heart-shaped heat map human body key points.

[0062] The method for encoding common human keypoint labels into a heart-shaped heatmap is as follows:

[0063] Constructing a two-dimensional coordinate system of x-axis and y-axis, for a circular heat map, the range of the heat map can be obtained as follows:

[0064] x 2 +y 2 ≤9ε 2 (1)

[0065] Among them, (x, y) is the horizontal coordinate and vertical coordinate of the midpoint of the two-dimensional plane, and ε is a given positive real number, representing the standard deviation of the heat map.

[0066] The heart-shaped heatmap consists of a circular heatmap and two non-overlapping semi-elliptical heatmaps that extend along the limbs. The circular heatmap has the key point as its center and 3ε as its radius. The centers of the two semi-elliptical heatmaps and the circular heatmap are the same point, and the major axis directions of the two semi-ellipses are the extension directions of the limbs on both sides of the key point. Figure 2 As shown in the figure, ① is a circular heatmap, and ②③ are two semi-elliptical heatmaps. The outermost circle is the boundary of the heatmap, and the middle circle represents the possibility of the area becoming a key point according to the grayscale. The higher the grayscale (the darker the color), the greater the possibility of becoming a key point.

[0067] The equation of the semi-ellipse satisfies: the length of the major axis a is twice the length of the minor axis b, and the minor axis length is the same as the radius of the traditional circular heat map. From (1), it can be obtained that the major axis a and the minor axis b of the ellipse satisfy:

[0068]

[0069] In the circular heat map, the probability that the key point is the (x, y) point, that is, the weight of the heat map G(x, y) satisfies: According to the relationship between a and b in (2), the weight of the elliptical heat map G(x, y) in the cardioid heat map satisfies:

[0070]

[0071] When the limbs are horizontal, it is easy to find that the boundary equation of the ellipse in the heart-shaped heat map satisfies:

[0072] x 2 +4y 2 ≤36ε 2 (4)

[0073] When the limbs are not horizontal, take the joints on the limbs as the center and measure the angles β1 and β2 of the limbs on both sides of the joints in the right horizontal direction. Figure 2 As shown, the coordinate rotation matrices of the limbs on both sides are:

[0074]

[0075] After matrix multiplication of (5), the coordinates of (x, y) after rotation of β1 and β2 are obtained as follows:

[0076]

[0077] Substituting (6) into (4), we can obtain the boundary equations of the ellipse after rotation by β1 and β2:

[0078]

[0079] When the limb is horizontal, the linear constraint of the semi-ellipse obtained from the ellipse is:

[0080] x>0 (8)

[0081] When the angles of the limbs on both sides rotating counterclockwise in the right horizontal direction are β1 and β2, the restriction of the ellipse to a semi-ellipse can be obtained from (6) and (8):

[0082]

[0083] According to the heat map non-overlap principle and (7) (9), the distribution range r1, r2, r3 of the heart-shaped heat map with the key point as the origin and its weight G(x, y) can be obtained as follows:

[0084]

[0085] Among them, r1, r2, r3 satisfy

[0086] r1:x 2 +y 2 ≤36ε 2 ;

[0087] r2:x 2 +y 2 >36ε 2 And xcosβ1-ysinβ1<0 and x 2 (4sin 2 β1+cos 2 β1)+

[0088] y 2 (sin 2 β1+4cos 2β1)+3xysin(2β1)-36ε 2 ≤0;

[0089] r3:x 2 +y 2 >36ε 2 And xcosβ1-ysinβ1<0 and x 2 (4sin 2 β1+cos 2 β1)+

[0090] y 2 (sin 2 β1+4cos 2 β1)+3xysin(2β1)-36ε 2 >0 and

[0091] x 2 (4sin 2 β2+cos 2 β2)+y 2 (sin 2 β2+4cos 2 β2)+

[0092] 3xysin(2β2)-36ε 2 ≤0;

[0093] G(x, y) is also the sample label in the training sample set.

[0094] The obtained G(x, y) will be used to calculate the loss value HeartLoss of the image in step S502.

[0095] After performing the above operations on all the key points of the human body in the image, it can be considered that the heart-shaped heat map data for training the human action recognition model is obtained.

[0096] S302, using the training sample set in step (1) to train a human action recognition model.

[0097] The human action recognition model is denoted as the HeartNet network model. The HeartNet network model is used to output the predicted human key point heat map.

[0098] The HeartNet network model consists of a backbone network (HRNe) and a heatmap regression model (HHR). The heatmap regression model is a data generator. The backbone network uses the heart-shaped heatmap generated by the HHR heatmap regression model to train the entire network. After training, it is directly used to accept input images and output key point heatmaps in the image.

[0099] Calculate the difference between the predicted key point heat map output by the human action recognition model and the label key point heat map as the loss value (HeartLoss) of the human action recognition model.

[0100] Specifically, the images in the training sample set are input into the backbone network HRNet of the HeartNet network model to obtain the predicted key point heat map of the corresponding image. The loss value HeartLoss of the corresponding image can be calculated based on the obtained key point data and the label G(x, y) obtained in step S501, and the weight of the backbone network is updated by the total HeartLoss of the data set. Specifically, for each pixel point contained in the key point heart-shaped heat map, we can obtain the unique coordinates (u1, v1) of the pixel point under correct annotation and the coordinates (u2, v2) of the current predicted key point heat map. The Pixel_HeartLoss of the pixel point satisfies:

[0101]

[0102] HeartLoss is the sum of the Pixel_HeartLoss values ​​of all pixels in the keypoint heart heatmap within the image. Training is considered complete when the total HeartLoss value of the dataset is less than 0.001.

[0103] S303: Input the human body cropped image obtained in step S2 into the trained human action recognition model, and the output result obtained is a heat map of key points of the human body corresponding to the cropped image.

[0104] For a picture, a set of key points of the human body key point heat map can be obtained. For example, if there are 17 key points, 17 human body key point heat maps will be obtained.

[0105] Specifically, the key points of the human body can be preset as: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle.

[0106] S4, judging whether there is risky behavior based on the key point information of human body movements.

[0107] Furthermore, step S4 includes the following sub-steps:

[0108] S401, obtaining a motion risk score based on key point information of human motion.

[0109] Furthermore, obtaining the action risk score according to the key point information of the human action includes the following steps:

[0110] The keypoint information of human motions is input into a multivariate linear regression model (KLR). The independent variable of the linear regression model is the keypoint information of human motions, and the dependent variable is the motion dangerousness score. Historical dangerous motion image data is collected and used to pre-train the linear regression motion model.

[0111] Specifically, HeartNet is first used to pre-process the human body key point position information of m groups of fusion data and store them in the Coor matrix, where represents the coordinates of the kth key point of the pth person. Assume that the transpose of the independent variable coefficient matrix of KLR is θ T , we can get θ from the number of key points of the human body T The matrix form of θ satisfies: T =[θ 0 θ 1 θ 2 … θ z ], the transpose of the p-th independent variable matrix of KLR The number of key points z of the human body can be obtained The matrix form of is: From the first two, we can get the KLR dependent variable prediction function h θ satisfy Combined with the L2 norm, we can make the loss function L(θ) of the KLR model as where ω (i) is the i-th dangerous action image data. The above formula can be used to derive the KLR independent variable coefficient θ update formula: j=0,1,2...z,when When , it can be considered that the KLR action risk scoring model has been trained.

[0112] S402: Compare the action risk score with a preset threshold to determine whether there is a risky behavior.

[0113] Specifically, the third quartile of the risk scores of all dangerous action image data may be set as the threshold T.

[0114] If the action's dangerousness score exceeds the classification threshold T, the action is determined to be dangerous. If the action's dangerousness score does not exceed the classification threshold T, the action is determined to be safe. If all human cutouts in the fused image are safe, the student in the fused image is considered to be performing no dangerous actions. If any human cutout in the fused image is performing a dangerous action, the student in the fused image is considered to be performing a dangerous action, and the emergency plan is activated.

[0115] A risk behavior monitoring system based on human motion recognition according to an embodiment of the present invention includes:

[0116] An acquisition module, used to respectively acquire infrared image data and structured light image data of an object to be identified;

[0117] The cropping module is used to fuse the infrared image data and the structured light image data and input them into the trained human body detector to output a human body cropped image;

[0118] The key point acquisition module is used to input the human body cropped image into the trained human action recognition model and output the key point information of the human action;

[0119] The judgment module is used to judge whether there is risky behavior based on the key point information of human body movements.

[0120] Furthermore, the system also includes a pre-processing module, which performs pre-processing on the infrared image data and the structured light image data before inputting them into the cropping module.

[0121] The implementation principle and technical effects of the system are similar to the above method and will not be repeated here.

[0122] It must be noted that in any of the above embodiments, the method is not necessarily executed in sequence according to the sequence number. As long as it cannot be inferred from the execution logic that it must be executed in a certain order, it means that it can be executed in any other possible order.

[0123] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A risk behavior monitoring method based on human motion recognition, characterized in that: Including steps: respectively collecting infrared image data and structured light image data of the object to be identified; fusing the infrared image data and the structured light image data and inputting the data into a trained human body detector to output a human body cropped image; Inputting the human body cropped image into a trained human action recognition model to output human action key point position information; Determining whether there is risky behavior based on the position information of key points of human body movements; The human action recognition model includes a backbone network and a heat map regression model. The heat map regression model is a key point heart-shaped heat map data generator. The backbone network uses the key point heart-shaped heat map generated by the heat map regression model to train the entire network. After training, the key point heart-shaped heat map in the image is directly output based on the input image; the key point heart-shaped heat map is composed of a circular heat map and two semi-elliptical heat maps.

2. The risk behavior monitoring method based on human motion recognition according to claim 1, characterized in that: The infrared image data and the structured light image data are preprocessed before the fusion, and the preprocessing includes performing image noise reduction processing on the infrared image data and the structured light image data.

3. The risk behavior monitoring method based on human motion recognition according to claim 2, characterized in that: The image noise reduction process comprises the steps of: The pixel coordinates in the image before denoising are marked as (p, q), the pixel value is r(p, q), and the pixel value of the pixel in the image after denoising is marked as R(p, q). The calculation formula of R(p, q) is: Where K(p′,q′) is the normal distribution equation of the pixel point (p′,q′), (p′,q′) is the coordinate of the pixel point within 3σ of the pixel point (p,q), r(p′,q′) is the pixel value of the pixel point (p′,q′), and ε is a preset positive constant.

4. The risk behavior monitoring method based on human motion recognition according to claim 1, characterized in that: The human detector is implemented based on the YOLOv5 network, and the size of its human detection frame is 384×288.

5. The risk behavior monitoring method based on human motion recognition according to claim 1, characterized in that: The determining whether there is a risky behavior based on the position information of the key points of the human body movement comprises the following steps: Obtaining an action risk score based on the position information of the key points of the human body action; The action risk score is compared with a preset threshold to determine whether there is a risky behavior.

6. The risk behavior monitoring method based on human motion recognition according to claim 5, characterized in that: Obtaining the action risk score according to the key point position information of the human body action comprises the following steps: The position information of the key points of human body movements is input into a multivariate linear regression model, the independent variable of the linear regression model is the position information of the key points of human body movements, and the dependent variable of the linear regression model is the movement risk score.

7. A risk behavior monitoring system based on human motion recognition, characterized in that: include: An acquisition module, used to respectively acquire infrared image data and structured light image data of an object to be identified; a cropping module, configured to fuse the infrared image data and the structured light image data and input the resultant data into a trained human body detector, and output a human body cropped image; A key point acquisition module is used to input the human body cropped image into a trained human action recognition model and output the position information of key points of human action; A judgment module, configured to judge whether there is risky behavior based on the position information of key points of human body movements; The human action recognition model includes a backbone network and a heat map regression model. The heat map regression model is a key point heart-shaped heat map data generator. The backbone network uses the key point heart-shaped heat map generated by the heat map regression model to train the entire network. After training, the key point heart-shaped heat map in the image is directly output based on the input image; the key point heart-shaped heat map is composed of a circular heat map and two semi-elliptical heat maps.

Images