Method for detecting mobile phone playing of personnel based on key point refinement

By using the combination of YOLOv5s and Lite-HRNet models in mobile phone detection, combining key point diversification and correction modules, the problem of key point detection error in complex backgrounds is solved, the detection accuracy and accuracy are improved, and the complex scenarios are realized Real-time monitoring of mobile phone behavior under play.

CN118918633APending Publication Date: 2024-11-08SHENZHEN VIRTUAL CLUSTERS INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410595522.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-14
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When detecting mobile phone behaviors, the detection accuracy of key points is reduced due to complex background, human body occlusion and different perspective angles, which can easily lead to misjudgment, especially the detection error of key points on the hands and elbows has a great impact.

Method used

The human body detection model based on the YOLOv5s algorithm is used to detect human bodies and handheld mobile phones, and combined with the key point detection model, the tendency of remote key points overlap prediction is reduced through the key point diversification module and correction module, and Lite-HRNet is used as the key point detection. The module improves the accuracy of key point detection through mutex loss and L2 distance measurement, and combines the logical judgment module to make the final behavioral judgment.

Benefits of technology

It improves the accuracy of mobile phone behavior detection, reduces misjudgment, realizes real-time monitoring video detection in complex scenarios, and enhances the accurate judgment of mobile phone behavior.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918633A_ABST
    Figure CN118918633A_ABST
Patent Text Reader

Abstract

The invention discloses a key point refinement-based person mobile phone playing detection method, which comprises the steps of S1, performing human body detection according to a human body detection model of a YOLOv5s algorithm to generate a human body image; s2, according to a human body attribute detection model of a YOLOv5s algorithm, detecting the part of the hand-held mobile phone in the image; s3, detecting human body key point coordinates according to the key point detection model; s4, predicting the action state of the target person in the current image; and S5, integrating multiple action state prediction results of the target person in the S5, and judging whether a mobile phone playing behavior exists or not. According to the invention, the problem of misjudgment caused by detection errors of key points of mobile phone playing behaviors in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of mobile phone detection, and in particular, relates to a method for detecting people playing with mobile phones based on key point refinement. Background Art

[0002] Playing with mobile phones is a common behavior in modern daily life. In some special scenarios, such as classrooms, roads, toll booths and other places that require people to maintain a high degree of concentration, it is often necessary to supervise and restrict people's mobile phone behavior.

[0003] Merely detecting the behavior of playing with mobile phones by manual means will consume a lot of human and material resources. At the same time, with the rapid development of computer technology and artificial intelligence, the technical solutions for detecting playing with mobile phones based on intelligent video analysis have been increasingly studied and applied. At present, most detection solutions are based on the relationship between human hands and mobile phones. For example, patents CN109614939B and CN111461020B comprehensively judge the behavior of playing with mobile phones based on the logical relationship of human hand key points. Patent CN110674728B uses the change rate and synchronization rate of human hand posture and mobile phone color to predict the behavior of playing with mobile phones. Other technical solutions, such as CN110287906A, use the line of sight prediction based on human facial key points and the change of key points between frames. CN112381068B uses the action classification model based on skeletal key points. The implementation of the above technical solutions all uses human key point detection technology. However, in the actual process of mobile phone playing detection, factors such as complex background, human occlusion and perspective difference will affect the accuracy of key point detection, thereby affecting the behavior detection results. In particular, detection errors of hand and elbow key points can easily lead to misjudgment of mobile phone playing behavior. Summary of the invention

[0004] In order to solve the problem of misjudgment caused by detection errors of key points of existing mobile phone playing behaviors, the present invention provides a method for detecting people playing with mobile phones based on key point refinement.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for detecting a person playing with a mobile phone based on key point refinement, comprising the steps of:

[0007] S1. Perform human body detection according to the human body detection model of the YOLOv5s algorithm to generate human body images;

[0008] S2, detecting the part of the image where the mobile phone is held according to the human attribute detection model of the YOLOv5s algorithm;

[0009] S3, detecting the coordinates of key points of the human body according to the key point detection model;

[0010] S4, predicting the action state of the target person in the current image;

[0011] S5. Combining the target person's action state prediction results in multiple S5s, determine whether there is any behavior of playing with the mobile phone.

[0012] Furthermore, the YOLOv5 model takes an image as input and first processes the image into three-dimensional data of a preset size, where the three dimensions correspond to the length, width, and RGB pixel values ​​of the image, respectively. After processing through several network modules, the model outputs the prediction information of the target in the image, including the target's coordinates, confidence, and target category number; the category number is set to 0.

[0013] Furthermore, a human body image is generated: the human body target is tracked, a human body frame sequence is constructed, and the target frame coordinate information of the same person in consecutive frames is connected in series based on target tracking to form a human body frame sequence. If there is an interruption in the middle, it is recalculated.

[0014] Furthermore, in step S2, the human body image obtained in step S1 is expanded once in both length and width directions of the image;

[0015] The expanded human image is used as the input of the human attribute model, and YOLOv5s is used as the baseline model to predict the bounding box of the part where the phone is held.

[0016] Furthermore, the key point detection model detects the coordinates of the key points of the human body: the human body image portion is processed into 3D data of a preset size, which is input into the key point detection model to calculate the key point information of the person, including the coordinate position and confidence of the key points.

[0017] Further, the key point detection model includes a key point detection module, a key point diversification module and a key point correction module;

[0018] The baseline network of the key point detection module adopts the Lite-HRNet model;

[0019] The keypoint diversification module is used to suppress incorrect predicted coordinates;

[0020] Keypoint diversification module: The keypoint heatmap generated by the keypoint detection module is recorded as Fi(i∈{1,2,....K}), where K is the number of keypoints, and the mutual exclusion loss is used. To measure the distance between the key point heat maps, during the model training process, the mutual exclusion loss Lex is added to the original loss Lhm to form the total loss L total =αL hm +βL ex , where α+β=1, and the weights α and β are learned during the training process. By minimizing Ltotal, the model can avoid key point overlap to a certain extent while learning the key point positions;

[0021] The key point correction module further refines the key point prediction features for the four most common mispredictions in key point detection.

[0022] Key point correction module: Four types of error prediction are defined as key point jitter, inversion, swap, and miss. Let the key point in the current human image be Pi (i∈{1,2,....K}), and use L2 distance to measure the similarity KS between two points. When KS (i,i′) When i′ is greater than 0.85, it is considered as an acceptable correct prediction (Good).

[0023] Furthermore, the key point detection model training process:

[0024] For the original human image and key point labels, a pseudo key point heat map is randomly generated according to the empirical distribution of the prediction results;

[0025] The human image and pseudo keypoint heat map are taken as input and first processed into a one-shot vector by the keypoint correction module;

[0026] Then calculate the coordinates of the key points to achieve the transition from coarse-grained to fine-grained attention and minimize the distance between the predicted coordinates and the label coordinates;

[0027] Key point calibration is completed.

[0028] Furthermore, the coordinates of the part of the mobile phone held in step 3 and the coordinates of the key points obtained in step 4 are used as personnel information and input into the logic judgment module to output the judgment of the action state of the person; the logic judgment module makes the best choice in combination with the actual application scenario. For example, for scenes with a relatively simple background environment, a relatively stable viewing angle of the surveillance camera, and a relatively fixed position of the person, the method of determining the angle and distance of the key points can be used; for scenes with a relatively large human target and a relatively large proportion of mobile phone targets in the surveillance screen, the hand posture change rate and the mobile phone color change rate can be combined for judgment; for scenes with a relatively complex background and fewer people in the screen, the action classification model judgment method can be used.

[0029] Furthermore, in step S5, for the predicted status of a person for n consecutive times, if the ratio of the mobile phone playing status is p or above, it is considered that the current person is playing with the mobile phone, and the final mobile phone playing warning and mobile phone playing picture are output.

[0030] Furthermore, the detailed steps of training the human body detection model and the human body attribute detection model in steps S1 and S2 are as follows:

[0031] Collect mobile phone image data;

[0032] Mark the human body boundary rectangle in the picture;

[0033] Train a human detection model;

[0034] Generate a human image based on the human bounding rectangle marked by the trained human detection model;

[0035] Annotate the area of ​​a person holding a mobile phone in an image of a human body;

[0036] Train a model for human attribute detection.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] Since most existing solutions for detecting people playing with mobile phones need to combine key points of the human body for behavior judgment, in real scenes, especially when the background is complex and frequently blocked, the key point prediction is prone to deviation, which often has a great impact on the judgment result. This patent improves the key point detection model and proposes a set of optimization solutions, and finally realizes the real-time detection of people playing with mobile phones under surveillance videos. In view of the key point prediction deviation that has a great influence in the process of detecting the behavior of playing with mobile phones, this patent proposes a corresponding solution. On the basis of the original human key point detection model, a key point diversification module and a key point correction module are added. Based on the mutual exclusion loss of local key points, the tendency of overlapping prediction of remote key points is reduced. Based on a large number of erroneous empirical distributions, possible key point misdetections are corrected, and reasonable logical judgments are set to increase the control of model errors, which ultimately improves the detection accuracy of mobile phone playing behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The present invention is an overall flow chart of a method for detecting a person playing with a mobile phone based on key point refinement in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0041] like Figure 1 As shown, this embodiment provides a method for detecting a person playing with a mobile phone based on key point refinement, including the steps of:

[0042] S1. Perform human body detection based on the human body detection model of the YOLOv5s algorithm and generate human body images. Set the length of the judgment time T and read the video frames at a frequency of extracting 1 frame every f frames. Detect people in the video frames. Considering the detection accuracy, generalization ability and computational cost of the human body detection model, YOLOv5s is selected as the benchmark model to predict the human body bounding box.

[0043] S2, detecting the part of the image where the mobile phone is held according to the human attribute detection model of the YOLOv5s algorithm;

[0044] S3, detecting the coordinates of key points of the human body according to the key point detection model;

[0045] S4, predicting the action state of the target person in the current image;

[0046] S5. Combining the target person's action state prediction results in multiple S5s, determine whether there is any behavior of playing with the mobile phone.

[0047] The YOLOv5 model takes an image as input and first processes the image into three-dimensional data of a preset size, where the three dimensions correspond to the length, width, and RGB pixel values ​​of the image. After being processed by several network modules, the model outputs the prediction information of the target in the image, including the target's coordinates, confidence, and target category number; there is only one category, person, and the category number is set to 0.

[0048] Generate human images: Track human targets and construct a human frame sequence. Based on target tracking, the target frame coordinate information of the same person in consecutive frames is connected in series to form a human frame sequence. If there is an interruption in the middle, it is recalculated.

[0049] In step S2, the human body image obtained in step S1 is expanded once in both length and width directions of the image;

[0050] The expanded human image is used as the input of the human attribute model, and YOLOv5s is used as the baseline model to predict the bounding box of the part where the mobile phone is held. Through observation experiments, it is known that the mobile phone in the real monitoring scene occupies a small area in the picture, and most of them are in non-frontal perspective. In addition, when people play with their mobile phones, their hands often block the mobile phone. Therefore, directly detecting the part of the mobile phone is very likely to lead to missed detection or false detection. In contrast, this patent detects the part where the mobile phone is held, and sets the target box to the part where the mobile phone and the human hand touch the mobile phone. The human hand part roughly includes the fingers and the 1 / 2 palm part close to the fingers, thereby improving the detection accuracy to a certain extent.

[0051] The key point detection model detects the coordinates of the key points of the human body: the human image part is processed into 3D data of a preset size, input into the key point detection model, and the key point information of the person's body is calculated, including the coordinate position and confidence of the key points.

[0052] The key point detection model includes a key point detection module, a key point diversification module and a key point correction module. Considering the accuracy, stability and computational cost of the key point detection model, this patent selects the Lite-HRNet model as the baseline network of the key point detection module. The original human key point pattern includes 17 key points, including nose, eyes (left and right), ears (left and right), shoulders (left and right), elbows (left and right), wrists (left and right), hips (left and right), knees (left and right), and ankles (left and right). Considering that the behavior of playing with mobile phones is more related to the head and upper limbs, and less related to the key points below the waist, a pattern of 11 key points, including nose, eyes (left and right), ears (left and right), shoulders (left and right), elbows (left and right), and wrists (left and right), is used in training.

[0053] The baseline network of the key point detection module adopts the Lite-HRNet model;

[0054] The keypoint diversification module is used to suppress incorrect predicted coordinates;

[0055] Keypoint diversification module: The keypoint heatmap generated by the keypoint detection module is recorded as Fi(i∈{1,2,....K}), where K is the number of keypoints, and the mutual exclusion loss is used. To measure the distance between the key point heat maps, during the model training process, the mutual exclusion loss Lex is added to the original loss Lhm to form the total loss L total =αL hm +βL ex , where α+β=1, and the weights α and β are learned during the training process. By minimizing Ltotal, the model can avoid key point overlap to a certain extent while learning the key point positions;

[0056] The key point correction module further refines the key point prediction features for the four most common mispredictions in key point detection.

[0057] Key point correction module: Four types of error prediction are defined as key point jitter, inversion, swap, and miss. Let the key point in the current human image be Pi (i∈{1,2,....K}), and use L2 distance to measure the similarity KS between two points. When KS (i,i′) When KS is greater than 0.85, i′ is considered an acceptable correct prediction (Good). Jitter means that the distance between the predicted key point i′ and the key point label i is small, that is, 0.5 < KS (i,i′) ≤0.85; Inversion means that the similarity KS(i*,i′) between the predicted key point i′ and the symmetric key point i* of i is higher than KS(i,i′); Swap means that when other parts of the human body appear in the human body image, the predicted key point i′ is similar to the key points of other human bodies. Similarity Higher than KS(i,i′); Miss means that the predicted key point i′ is far away from the key point label i, in which case 0.1<KS (i,i′) ≤0.5.

[0058] Key point detection model training process:

[0059] For the original human image and key point labels, a pseudo key point heat map is randomly generated according to the empirical distribution of the prediction results;

[0060] The human image and pseudo keypoint heat map are taken as input and first processed into a one-shot vector by the keypoint correction module;

[0061] Then calculate the coordinates of the key points to achieve the transition from coarse-grained to fine-grained attention and minimize the distance between the predicted coordinates and the label coordinates;

[0062] Key point calibration is completed.

[0063] Furthermore, the coordinates of the part of the handheld mobile phone obtained in step S3 and the coordinates of the key points obtained in step S4 are used as personnel information and input into the logic judgment module to output the judgment of the action state of the person; the logic judgment module makes the best choice in combination with the actual application scenario. For example, for scenes with a relatively simple background environment, a relatively stable viewing angle of the surveillance camera, and a relatively fixed position of the person, the method of determining the angle and distance of the key points can be used; for scenes with a relatively large human target and a relatively large proportion of mobile phone targets in the surveillance screen, the hand posture change rate and the mobile phone color change rate can be combined for judgment; for scenes with a relatively complex background and fewer people in the screen, the method of determining by the action classification model can be used.

[0064] In step S5, for the predicted status of a person for n consecutive times, if the ratio of the mobile phone playing status is p or above, it is considered that the current person is playing with the mobile phone, and the final mobile phone playing warning and mobile phone playing picture are output.

[0065] The detailed steps of training the human body detection model and the human body attribute detection model in steps S1 and S2 are: collecting mobile phone playing image data; marking the human body boundary rectangular frame in the picture; training the human body detection model; generating a human body image based on the human body boundary rectangular frame marked by the trained human body detection model; marking the mobile phone holding area in the human body image; training the human body attribute detection model.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] Since most existing solutions for detecting people playing with mobile phones need to combine key points of the human body for behavior judgment, in real scenes, especially when the background is complex and frequently blocked, the key point prediction is prone to deviation, which often has a great impact on the judgment result. This patent improves the key point detection model and proposes a set of optimization solutions, and finally realizes the real-time detection of people playing with mobile phones under surveillance videos. In view of the key point prediction deviation that has a great influence in the process of detecting the behavior of playing with mobile phones, this patent proposes a corresponding solution. On the basis of the original human key point detection model, a key point diversification module and a key point correction module are added. Based on the mutual exclusion loss of local key points, the tendency of overlapping prediction of remote key points is reduced. Based on a large number of erroneous empirical distributions, possible key point misdetections are corrected, and reasonable logical judgments are set to increase the control of model errors, which ultimately improves the detection accuracy of mobile phone playing behavior.

[0068] The above is a detailed introduction to a method for detecting people playing with mobile phones based on key point refinement provided by the present application. The description of the specific embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for detecting people playing with mobile phones based on key point refinement, characterized in that: Includes steps: S1. Perform human body detection according to the human body detection model of the YOLOv5s algorithm to generate human body images; S2, detecting the part of the image where the mobile phone is held according to the human attribute detection model of the YOLOv5s algorithm; S3, detecting the coordinates of key points of the human body according to the key point detection model; S4, predicting the action state of the target person in the current image; S5. Combining the target person's action state prediction results in multiple S5s, determine whether there is any behavior of playing with the mobile phone.

2. According to the method for detecting people playing with mobile phones based on key point refinement according to claim 1, it is characterized in that: The YOLOv5 model takes an image as input and first processes the image into three-dimensional data of a preset size, where the three dimensions correspond to the length, width, and RGB pixel values ​​of the image. After being processed by several network modules, the model outputs the prediction information of the target in the image, including the target's coordinates, confidence, and target category number; the category number is set to 0.

3. According to the method for detecting people playing with mobile phones based on key point refinement according to claim 1, it is characterized in that: Generate human images: Track human targets and construct a human frame sequence. Based on target tracking, the target frame coordinate information of the same person in consecutive frames is connected in series to form a human frame sequence. If there is an interruption in the middle, it is recalculated.

4. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: In step S2, the human body image obtained in step S1 is expanded once in both length and width directions of the image; The expanded human image is used as the input of the human attribute model, and YOLOv5s is used as the baseline model to predict the bounding box of the part where the phone is held.

5. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: The key point detection model detects the coordinates of the key points of the human body: the human image part is processed into 3D data of a preset size, input into the key point detection model, and the key point information of the person's body is calculated, including the coordinate position and confidence of the key points.

6. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: The key point detection model includes a key point detection module, a key point diversification module and a key point correction module; The baseline network of the key point detection module adopts the Lite-HRNet model; The keypoint diversification module is used to suppress incorrect predicted coordinates; Keypoint diversification module: The keypoint heatmap generated by the keypoint detection module is recorded as Fi(i∈{1,2,....K}), where K is the number of keypoints, and the mutual exclusion loss is used. To measure the distance between the key point heat maps, during the model training process, the mutual exclusion loss Lex is added to the original loss Lhm to form the total loss L total =αL hm +βL ex , where α+β=1, and the weights α and β are learned during the training process. By minimizing Ltotal, the model can avoid key point overlap to a certain extent while learning the key point positions; The key point correction module further refines the key point prediction features for the four most common mispredictions in key point detection. Key point correction module: Four types of misprediction are defined as key point jitter, crossover, staggered and miss. Assume that the key point in the current human image is Pi (i∈{1,2,....K}), and the similarity KS between two points is measured by L2 distance. When KS (i,i′) When i′>0.85, i′ is considered an acceptable correct prediction.

7. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: Key point detection model training process: For the original human image and key point labels, a pseudo key point heat map is randomly generated according to the empirical distribution of the prediction results; The human image and pseudo keypoint heat map are taken as input and first processed into a one-shot vector by the keypoint correction module; Then calculate the coordinates of the key points to achieve the transition from coarse-grained to fine-grained attention and minimize the distance between the predicted coordinates and the label coordinates; Key point calibration is completed.

8. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: The coordinates of the part of the handheld mobile phone obtained in step S3 and the coordinates of the key points obtained in step S4 are input into the logic judgment module as personnel information, and the judgment on the action status of the person is output; the logic judgment module makes an optimization based on the actual application scenario.

9. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: In step S5, for the predicted status of a person for n consecutive times, if the ratio of the mobile phone playing status is p or above, it is considered that the current person is playing with the mobile phone, and the final mobile phone playing warning and mobile phone playing picture are output.

10. The method for detecting people playing with mobile phones based on key point refinement according to claim 1, characterized in that: Detailed steps for training the human body detection model and human body attribute detection model in steps S1 and S2: Collect mobile phone image data; Mark the human body boundary rectangle in the picture; Train a human detection model; Generate a human image based on the human bounding rectangle marked by the trained human detection model; Annotate the area of ​​a person holding a mobile phone in an image of a human body; Train a model for human attribute detection.