A method, device and medium for detecting mobile phone usage

Abstract

The present invention discloses a method, device and medium for detecting the use of a mobile phone, and relates to the technical field of computer vision. The method comprises: obtaining a target image, and using a target detection model to detect whether there is a person and a mobile phone in the target image, and obtaining a human body detection frame and a mobile phone detection frame; using a Coco‑17 model to determine the key point features of a human body according to the human body detection frame; determining the positional relationship between a mobile phone and a human body according to the mobile phone detection frame and the key point features of a human body; using a Face Mesh model to determine the state of both eyes according to the human body detection frame; determining the use of a mobile phone according to the positional relationship between the mobile phone and the human body and the state of both eyes; the use of a mobile phone includes: not using a mobile phone, using a mobile phone to make calls, using a mobile phone for entertainment, and using a mobile phone to take pictures. The present invention can detect the illegal use of mobile phones and improve the accuracy of detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and medium for detecting the usage of a mobile phone. Background Art

[0002] Human keypoints detection, also known as human posture estimation, is a relatively basic task in computer vision and a prerequisite for human motion recognition, behavior analysis, and human-computer interaction. Small target detection and occluded object detection have always been difficult in the field of target detection. Human keypoints detection can make up for this shortcoming. Human motion recognition and behavior analysis can help target detection be more accurate and efficient. As a key department, the operation and maintenance site of the bank data center needs to strictly control the use of mobile phones, identify the use of mobile phones for making calls, using mobile phones for entertainment, and using mobile phones for taking pictures, and provide prompts. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device and medium for detecting the use of a mobile phone, so as to detect the illegal use of a mobile phone and improve the accuracy of the detection.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for detecting mobile phone usage, the method comprising:

[0006] Acquire a target image, and use a target detection model to detect whether a person and a mobile phone exist in the target image, and obtain a human body detection frame and a mobile phone detection frame;

[0007] Using the Coco-17 model to determine the key point features of the human body according to the human body detection frame;

[0008] Determine the positional relationship between the mobile phone and the human body according to the mobile phone detection frame and the key point features of the human body;

[0009] Using a Face Mesh model to determine the state of both eyes according to the human body detection frame;

[0010] The use of the mobile phone is determined according to the positional relationship between the mobile phone and the human body and the state of the eyes; the use of the mobile phone includes: not using the mobile phone, using the mobile phone to make calls, using the mobile phone for entertainment, and using the mobile phone to take pictures.

[0011] Optionally, the key point features of the human body include: the positions of the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees and left and right ankles.

[0012] Optionally, the position relationship between the mobile phone and the human body includes: a first position relationship, a second position relationship, a third position relationship, a fourth position relationship and a fifth position relationship;

[0013] The first position relationship is that the positions of the mobile phone and the two wrists of the human body do not overlap;

[0014] The second position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter; the wrist holding the phone is the wrist that coincides with the position of the mobile phone;

[0015] The third position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, and the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes is greater than the set angle;

[0016] The fourth position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, and the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes is less than or equal to the set angle;

[0017] The fifth position relationship is that the position of the mobile phone coincides with that of at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of both ears as the center and the distance between the shoulders as the diameter, and the two eyes are not equidistant from the center point of the mobile phone detection frame.

[0018] Optionally, determining the positional relationship between the mobile phone and the human body according to the mobile phone detection frame and the key point features of the human body specifically includes:

[0019] Determine whether the position of the mobile phone coincides with the position of at least one wrist of the human body according to the positions of the left and right wrists in the mobile phone detection frame and the key point features of the human body;

[0020] If the positions of the mobile phone and the two wrists of the human body do not overlap, the position relationship between the mobile phone and the human body is the first position relationship;

[0021] If the position of the mobile phone coincides with at least one wrist of the human body, judging whether the wrist holding the phone is within a circle with the center of both ears as the center and the distance between both shoulders as the diameter according to the positions of the left and right ears, left and right shoulders and left and right wrists in the key features of the human body;

[0022] If the wrist holding the phone is within a circle with the centers of the two ears as the center and the distance between the two shoulders as the diameter, the positional relationship between the phone and the human body is the second positional relationship;

[0023] If the wrist holding the phone is not in a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, determine whether the two eyes are equidistant from the center point of the phone detection frame based on the positions of the left and right eyes in the key point features of the human body and the position of the center point of the phone detection frame, and calculate the angle between the line connecting the center point of the phone detection frame and the center points of the two eyes and the line connecting the two eyes;

[0024] If the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the center point of the mobile phone detection frame and the line connecting the two eyes and the line connecting the two eyes is greater than the set angle, then the position relationship between the mobile phone and the human body is the third position relationship;

[0025] If the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the center point of the mobile phone detection frame and the line connecting the two eyes and the line connecting the two eyes is less than or equal to the set angle, then the position relationship between the mobile phone and the human body is the fourth position relationship;

[0026] If the distances between the eyes and the center point of the mobile phone detection frame are not equidistant, the positional relationship between the mobile phone and the human body is the fifth positional relationship.

[0027] Optionally, the states of the two eyes include: at least one eye is open and both eyes are closed.

[0028] Optionally, using a Face Mesh model to determine the state of both eyes according to the human body detection frame specifically includes:

[0029] Using the Face Mesh model to determine the key features of the eyes according to the human body detection frame; the key features of the eyes include: the positions of the inner and outer eye corners and the vertices of the rectangle tangent to the eyeball;

[0030] Calculating the eye aspect ratio according to the eye key point features;

[0031] If the eye aspect ratio of at least one eye is greater than a first setting value, the state of the two eyes is that at least one eye is open; the first setting value is greater than zero;

[0032] If the difference between the eye aspect ratio of the two eyes and zero is less than a second setting value, the state of the two eyes is that both eyes are closed; the second setting value is greater than zero and less than the first setting value.

[0033] Optionally, determining the usage of the mobile phone according to the positional relationship between the mobile phone and the human body and the state of the eyes specifically includes:

[0034] If the position relationship between the mobile phone and the human body is the first position relationship, then the mobile phone usage status is that the mobile phone is not used;

[0035] If the position relationship between the mobile phone and the human body is the second position relationship, then the mobile phone usage condition is using the mobile phone to make a call;

[0036] If the position relationship between the mobile phone and the human body is the third position relationship, and the state of the two eyes is that at least one eye is open, then the mobile phone usage is using the mobile phone for entertainment;

[0037] If the position relationship between the mobile phone and the human body is the third position relationship, and the state of the two eyes is that both eyes are closed, then the mobile phone usage status is that the mobile phone is not used;

[0038] If the position relationship between the mobile phone and the human body is the fourth position relationship, and the state of the two eyes is that at least one eye is open, then the mobile phone usage condition is taking photos with the mobile phone;

[0039] If the position relationship between the mobile phone and the human body is the fourth position relationship, and the state of the two eyes is that both eyes are closed, then the mobile phone usage status is that the mobile phone is not used;

[0040] If the position relationship between the mobile phone and the human body is the fifth position relationship, the mobile phone usage status is that the mobile phone is not used.

[0041] Optionally, the method further comprises:

[0042] A judgment result message is generated according to the usage of the mobile phone and synchronized to the unified monitoring and alarm platform.

[0043] A computer device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned method for detecting the usage of a mobile phone.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for detecting mobile phone usage.

[0045] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] The method for detecting mobile phone usage provided by the present invention first uses a target detection model to detect whether there are people and mobile phones in a target image, obtains a human body detection frame and a mobile phone detection frame, then uses the Coco-17 model to determine the key point features of the human body according to the human body detection frame, and then determines the positional relationship between the mobile phone and the human body according to the mobile phone detection frame and the key point features of the human body, and uses the Face Mesh model to determine the state of both eyes according to the human body detection frame, and finally determines the mobile phone usage according to the positional relationship between the mobile phone and the human body and the state of both eyes. The method can be applied to duty and office places where the use of mobile phones is strictly restricted, such as the operation and maintenance scenarios of bank data centers, to detect the illegal use of mobile phones in real time, refine the application scenarios of human body key point detection and small object detection, improve the target detection effect of occluded objects, and improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flow chart of a method for detecting mobile phone usage provided by the present invention;

[0049] Figure 2 A detailed flow chart of the method for detecting the usage of a mobile phone provided by the present invention;

[0050] Figure 3 A schematic diagram of the positions of 17 key points of the human body provided by the present invention;

[0051] Figure 4 This is a schematic diagram of the positions of the six key eye points provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The purpose of the present invention is to provide a method, device and medium for detecting the use of a mobile phone. On the basis of defining the key points of the Coco-17 human body, a top-down detection method is used, and an algorithm for detecting various scenarios such as using a mobile phone to make calls, using a mobile phone for entertainment and using a mobile phone to take pictures is proposed, thereby realizing the detection of illegal use of mobile phones and improving the accuracy of detection.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] like Figure 1 and Figure 2 As shown, the method for detecting the usage of a mobile phone provided by the present invention comprises:

[0056] Step 101: Acquire a target image, and use a target detection model to detect whether there are people and mobile phones in the target image, and obtain a human body detection frame and a mobile phone detection frame.

[0057] Based on the general object detection model, it is possible to determine whether the features of a person and a mobile phone can be detected in the picture at one time. If there is no person or mobile phone in the picture, the mobile phone usage is not used, that is, it is not a mobile phone use behavior.

[0058] Step 102: Using the Coco-17 model, determine the key point features of the human body according to the human body detection frame.

[0059] After detecting a person and a mobile phone, it is further determined whether the person is using the mobile phone to make a call, entertain himself (i.e. playing with the mobile phone, which can include playing games, watching videos, reading novels, etc.), or taking photos. The key point features of the human body are detected based on the human body key point detection model Coco-17, and the positional relationship between the mobile phone detection frame features and the key point features of the human body is further detected. When making a call, playing with the mobile phone, or taking photos, people generally hold the mobile phone in their hands, and the mobile phone and the wrist of the key point of the human body coincide. Therefore, the situation of making a call, playing with the mobile phone, or taking photos can be determined by confirming the positional relationship between the position of the wrist holding the phone and other key points of the human body.

[0060] Coco-17 represents the key points of the human body as 17 joints, namely the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. The task of human key point detection is to detect the human body and the corresponding key point positions from the input image. Therefore, the key point features of the human body include: the positions of the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0061] like Figure 3As shown in the figure, the corresponding relationship between each key point number and the key point of the human body is: "keypoints":{0:"nose",1:"left_eye",2:"right_eye",3:"left_ear",4:"right_ear",5:"left_shoulder",6:"right_shoulder",7:"left_elbow",8:"right_elbow",9:"left_wrist",10:"right_wrist",11:"left_hip",12:"right_hip",13:"left_knee",14:"right_knee",15:"left_ankle",16:"right_ankle"}.

[0062] Step 103: Determine the positional relationship between the mobile phone and the human body according to the mobile phone detection frame and the key point features of the human body.

[0063] Specifically, the position relationship between the mobile phone and the human body includes: a first position relationship, a second position relationship, a third position relationship, a fourth position relationship and a fifth position relationship. The first position relationship is that the positions of the mobile phone and the two wrists of the human body do not overlap. The second position relationship is that the positions of the mobile phone and at least one wrist of the human body overlap, and the wrist holding the phone is in a circle with the center of the two ears as the center and the distance of the two shoulders as the diameter; the wrist holding the phone is the wrist that overlaps with the position of the mobile phone. The third position relationship is that the positions of the mobile phone and at least one wrist of the human body overlap, and the wrist holding the phone is not in a circle with the center of the two ears as the center and the distance of the two shoulders as the diameter, and the eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes is greater than the set angle. The fourth position relationship is that the positions of the mobile phone and at least one wrist of the human body overlap, and the wrist holding the phone is not in a circle with the center of the two ears as the center and the distance of the two shoulders as the diameter, and the eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes is less than or equal to the set angle. The fifth position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, and the two eyes are not equidistant from the center point of the mobile phone detection frame. The wrist holding the phone is the wrist that coincides with the position of the mobile phone.

[0064] Preferably, the criterion for determining whether the positions of the mobile phone and the wrist coincide is to determine whether the distance between the center point of the mobile phone detection frame and the wrist is less than the diagonal length of the mobile phone detection frame. If the distance between the center point of the mobile phone detection frame and the wrist is less than the diagonal length of the mobile phone detection frame, the positions of the mobile phone and the wrist coincide; if the distance between the center point of the mobile phone detection frame and the wrist is greater than or equal to the diagonal length of the mobile phone detection frame, the positions of the mobile phone and the wrist do not coincide.

[0065] The specific process of determining the position relationship between the mobile phone and the human body includes:

[0066] (1) judging whether the position of the mobile phone coincides with the position of at least one wrist of the human body according to the positions of the left and right wrists in the mobile phone detection frame and the key point features of the human body; if the positions of the mobile phone and the two wrists of the human body do not coincide, the position relationship between the mobile phone and the human body is the first position relationship.

[0067] (2) If the position of the mobile phone coincides with that of at least one wrist of the human body, it is determined whether the wrist holding the phone is within a circle with the centers of the two ears as the center and the distance between the two shoulders as the diameter according to the positions of the left and right ears, left and right shoulders and left and right wrists in the key point features of the human body; if the wrist holding the phone is within the circle with the centers of the two ears as the center and the distance between the two shoulders as the diameter, the positional relationship between the mobile phone and the human body is the second positional relationship.

[0068] (3) If the wrist holding the phone is not within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, then determine whether the two eyes are equidistant from the center point of the mobile phone detection frame based on the positions of the left and right eyes in the key point features of the human body and the position of the center point of the mobile phone detection frame, and calculate the angle between the center point of the mobile phone detection frame and the line connecting the two eyes; if the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center points of the two eyes and the line connecting the two eyes is greater than the set angle, then the positional relationship between the mobile phone and the human body is the third positional relationship.

[0069] (4) If the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the center point of the mobile phone detection frame and the line connecting the two eyes and the line connecting the two eyes is less than or equal to the set angle, then the positional relationship between the mobile phone and the human body is the fourth positional relationship. Preferably, the set angle is 15°.

[0070] (5) If the distances between the eyes and the center point of the mobile phone detection frame are not equidistant, the positional relationship between the mobile phone and the human body is the fifth positional relationship.

[0071] The specific process of judging whether the wrist holding the device is within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter is as follows:

[0072] 1. Determine the key points of the left and right shoulders and determine their coordinate positions in the image, namely left_shoulder and right_shoulder.

[0073] 2. Calculate the distance between the key points of the two shoulders, that is, ABS (right_shoulder-left_shoulder).

[0074] 3. Determine the key points of the left and right ears and determine their coordinate positions in the image, namely left_ear and right_ear.

[0075] 4. Calculate the middle point between the binaural key points, that is, cent = ABS (right_ear + left_ear) / 2.

[0076] 5. Determine the key points of the left and right wrists and determine their coordinate positions in the image, namely left_wrist and right_wrist.

[0077] 6. Draw a circle with the middle point of the two ear key points as the center and the distance between the two shoulders as the diameter. The radius of the circle is radius = ABS (right_shoulder-left_shoulder) / 2.

[0078] 7. Calculate the distances between the left and right wrist key points and the center of the circle respectively, i.e., dis_l = ABS (cent-left_wrist) and dis_r = ABS (cent-right_wrist).

[0079] 8. Determine whether the distance between the left and right wrist key points and the center of the circle is less than or equal to the radius. If so, the position relationship between the mobile phone and the human body is the second position relationship. At this time, it can be determined that the mobile phone is used to make a call, that is:

[0080] if dis_r<=radius or dis_l<=radius

[0081] print("calling").

[0082] The specific process of determining whether the center points of the eyes and the phone detection frame are equidistant is as follows:

[0083] 1. Determine the key points of the left and right eyes and determine their coordinate positions in the image, namely left_eye and right_eye.

[0084] 2. Determine the center point of the phone detection frame and determine its coordinate position in the image, that is, phone_center.

[0085] 3. Calculate the distances dis_l_eye and dis_r_eye from the center of the left and right eyes to the phone detection frame using the coordinate data of the corresponding points, i.e., dis_l_eye = ABS (left_eye - phone_center) and dis_r_ey = ABS (right_eye - phone_center).

[0086] 4. Using the coordinate data of the corresponding points, calculate the angle α between the center point of the phone detection frame and the line connecting the center points of both eyes (line_eye2phone) and the line connecting both eyes (specifically, the extension line of the line connecting the left and right eyes) (line_eyes). The calculation formula for α is as follows:

[0087] horizontal_distance=ABS((left_eye_x+right_eye_x) / 2-phone_center_x);

[0088] vertical_distance=ABS((left_eye_y+right_eye_y) / 2-phone_center_y);

[0089] α=math.agrees(math.atan(vertical_distance / horizontal_distance));

[0090] Among them, the coordinates of the left and right eye key points are (left_eye_x, left_eye_y) and (right_eye_x, right_eye_y), and the coordinates of the center point of the phone detection frame are (phone_center_x, phone_center_y).

[0091] 5. Determine whether dis_l_eye and dis_r_eye are approximately equal (specifically, the ratio of the length difference to the length sum is less than or equal to 5%), and α>15°. If so, the position relationship between the mobile phone and the human body is the third position relationship. Assuming that the state of both eyes also meets the conditions at this time, it can be determined that the mobile phone is used for entertainment, that is:

[0092] if ABS(dis_l_eye-dis_r_eye) / (dis_l_eye+dis_r_eye)<=5%andα>15°

[0093] print("palying").

[0094] 6. Determine whether dis_l_eye and dis_r_eye are approximately equal, and α <= 15°. If so, the position relationship between the mobile phone and the human body is the fourth position relationship. Assuming that the state of both eyes also meets the conditions at this time, it can be determined that the mobile phone is used to take pictures, that is:

[0095] if ABS(dis_l_eye-dis_r_eye) / (dis_l_eye+dis_r_eye)<=5% andα<=15°

[0096] print("taking a picture").

[0097] Step 104: using the Face Mesh model to determine the state of both eyes according to the human body detection frame.

[0098] Specifically, the states of the two eyes include: at least one eye is open and both eyes are closed.

[0099] The specific process of determining the state of the above-mentioned eyes includes:

[0100] (1) Use the Face Mesh model to determine the eye key point features based on the human body detection frame. Use the face mesh model Face Mesh to detect the eye key points. Face Mesh returns 468 face key points. Only 12 eye key points are needed in the present invention (6 points for each eye). Draw 6 points in the picture containing the human eye, such as Figure 4 As shown in FIG. 1 , the key features of the eye include: the inner and outer corners of the eye (i.e., P1 and P4, the distance between the two is the longest distance between the eye edge points) and the positions of the vertices of the rectangle tangent to the eyeball (i.e., P2, P3, P5, and P6). When a person blinks, the distances between these six points will change, and the distance relationships between these six points can be used to determine whether there is blinking behavior and the open and closed state of the eyes.

[0101] (2) Calculate the eye aspect ratio based on the eye key point features. The specific formula is:

[0102]

[0103] Among them, EAR (Eye Aspect Ratio) represents the eye aspect ratio, P1 represents the position of the inner corner of the eye, P4 represents the position of the outer corner of the eye, P2 represents the position of the first vertex of the rectangle tangent to the eyeball, P3 represents the position of the second vertex of the rectangle tangent to the eyeball, P5 represents the position of the third vertex of the rectangle tangent to the eyeball, and P6 represents the position of the fourth vertex of the rectangle tangent to the eyeball. |||| represents the L2 norm, which is used to calculate the distance between two vectors.

[0104] Eyes open, see Figure 4 In the left picture, the eyes are closed. Figure 4 From the right figure in the figure, it can be seen that when the eyes are open, the EAR is greater than zero and usually remains constant, while when the eyes are closed, the EAR approaches zero. Therefore, the following judgment method is used:

[0105] (3) If the aspect ratio of at least one eye is greater than a first setting value, the state of the two eyes is that at least one eye is open. The first setting value is greater than zero. The first setting value is generally a real number between 0 and 1, such as 0.3.

[0106] (4) If the difference between the eye aspect ratio of the two eyes and zero is less than a second setting value, then the state of the two eyes is that both eyes are closed. The second setting value is greater than zero and less than the first setting value. The second setting value is generally a relatively small value, such as 0.005.

[0107] Step 105: Determine the usage of the mobile phone according to the positional relationship between the mobile phone and the human body and the state of the eyes. The usage of the mobile phone includes: not using the mobile phone, using the mobile phone to make calls, using the mobile phone for entertainment, and using the mobile phone to take photos.

[0108] For the scenario of using a mobile phone to make a call: determine whether the wrist holding the phone is within a circle with the distance between the shoulders as the diameter. If it is within the range, it is determined to be a call. The basis for judgment is that when making a call, the phone is generally close to the ears and mouth, and will not exceed the distance of the left and right shoulders. Even when the phone is hands-free, it is generally not too far away from the mouth.

[0109] For scenes of using a mobile phone for entertainment or taking photos: if the position of the wrist holding the phone is outside the circle with the distance between the shoulders as the diameter, it is necessary to determine whether at least one eye is open and whether the distance between the left and right eyes and the phone is approximately equal. If the above conditions are met, it is determined to be playing with the phone or taking photos. The basis for this judgment is that when playing with the phone or taking photos, the user faces the phone screen with both eyes open and looks in the same direction; for easy viewing, the phone is generally located in the middle of both eyes, that is, the distance between the phone and the left and right eyes is approximately equal, and the two eyes and the phone form an isosceles triangle, and the distance between the phone and the left and right eyes is the length of the two sides of the triangle.

[0110] Therefore, the specific process of determining mobile phone usage includes:

[0111] (1) If the position relationship between the mobile phone and the human body is the first position relationship, the mobile phone usage status is that the mobile phone is not used.

[0112] (2) If the positional relationship between the mobile phone and the human body is the second positional relationship, then the mobile phone usage is using the mobile phone to make a call.

[0113] (3) If the positional relationship between the mobile phone and the human body is the third positional relationship, and the state of the two eyes is that at least one eye is open, then the mobile phone usage is using the mobile phone for entertainment.

[0114] (4) If the positional relationship between the mobile phone and the human body is the third positional relationship, and the state of the eyes is that both eyes are closed, then the mobile phone usage status is that the mobile phone is not used.

[0115] (5) If the positional relationship between the mobile phone and the human body is the fourth positional relationship, and the state of the two eyes is that at least one eye is open, then the mobile phone usage is using the mobile phone to take photos.

[0116] (6) If the positional relationship between the mobile phone and the human body is the fourth positional relationship, and the state of the eyes is that both eyes are closed, then the mobile phone usage status is that the mobile phone is not used.

[0117] (7) If the position relationship between the mobile phone and the human body is the fifth position relationship, the mobile phone usage status is that the mobile phone is not used.

[0118] Furthermore, the method further comprises:

[0119] Step 106: Generate a judgment result message based on the mobile phone usage and synchronize it to the unified monitoring and alarm platform.

[0120] By extending the above-mentioned mobile phone usage detection method to duty and office locations where mobile phone use is strictly restricted, the judgment result messages can be synchronized to a unified monitoring and alarm platform, which can detect and remind illegal mobile phone use in real time, thereby improving security and compliance management capabilities.

[0121] In summary, the present invention proposes a method for detecting the use of mobile phones in bank operation and maintenance sites by means of human key point detection, which refines the application scenarios of human key point detection and small object detection, and can also improve the target detection effect of occluded objects to a certain extent. Through verification in the operation and maintenance scenario, the results show that the accuracy of detection has been significantly improved without affecting the detection efficiency.

[0122] In one embodiment, a computer program product is also provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0123] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0124] In one embodiment, a computer device is also provided, including a memory and a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0125] Furthermore, the computer device also includes an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned method embodiments are implemented.

[0126] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting mobile phone usage, characterized in that: The method for detecting the use of a mobile phone is a detection method applied to a duty or office site of a bank data center operation and maintenance, and the method for detecting the use of a mobile phone includes: Acquire a target image, and use a target detection model to detect whether a person and a mobile phone exist in the target image, and obtain a human body detection frame and a mobile phone detection frame; Using the Coco-17 model to determine the key point features of the human body according to the human body detection frame; Determine the positional relationship between the mobile phone and the human body according to the mobile phone detection frame and the key point features of the human body; Using the Face Mesh model to determine the state of both eyes according to the human body detection frame; the state of both eyes includes: at least one eye is open and both eyes are closed; Determine the use of the mobile phone according to the positional relationship between the mobile phone and the human body and the state of the eyes; the use of the mobile phone includes: not using the mobile phone, using the mobile phone to make calls, using the mobile phone for entertainment, and using the mobile phone to take photos; Determining the positional relationship between the mobile phone and the human body according to the mobile phone detection frame and the key point features of the human body specifically includes: Determine whether the position of the mobile phone coincides with the position of at least one wrist of the human body according to the positions of the left and right wrists in the mobile phone detection frame and the key point features of the human body; If the positions of the mobile phone and the two wrists of the human body do not overlap, the position relationship between the mobile phone and the human body is the first position relationship; If the position of the mobile phone coincides with at least one wrist of the human body, judging whether the wrist holding the phone is within a circle with the center of both ears as the center and the distance between both shoulders as the diameter according to the positions of the left and right ears, left and right shoulders and left and right wrists in the key features of the human body; If the wrist holding the phone is within a circle with the centers of the two ears as the center and the distance between the two shoulders as the diameter, the positional relationship between the phone and the human body is the second positional relationship; If the wrist holding the phone is not in a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, then determine whether the center points of the two eyes and the mobile phone detection frame are equidistant according to the positions of the left and right eyes in the key point features of the human body and the center point of the mobile phone detection frame, and calculate the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes; whether the center points of the two eyes and the mobile phone detection frame are equidistant is whether the ratio of the difference in length between dis_l_eye and dis_r_eye to the sum of the lengths is less than or equal to 5%; dis_l_eye is the distance between the left eye and the center point of the mobile phone detection frame, and dis_r_eye is the distance between the right eye and the center point of the mobile phone detection frame; the angle calculation formula is: horizontal_distance=ABS((left_eye_x+right_eye_x) / 2-phone_center_x); vertical_distance=ABS((left_eye_y+right_eye_y) / 2-phone_center_y); α=math.agrees(math.atan(vertical_distance / horizontal_distance)); Among them, horizontal_distance is the horizontal distance, vertical_distance is the vertical distance, (left_eye_x, left_eye_y) is the coordinate of the left eye key point, (right_eye_x, right_eye_y) is the coordinate of the right eye key point, (phone_center_x, phone_center_y) is the coordinate of the center point of the phone detection frame, and α is the angle between the center point of the phone detection frame and the line connecting the center points of both eyes and the line connecting both eyes; If the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the center point of the mobile phone detection frame and the line connecting the two eyes and the line connecting the two eyes is greater than the set angle, then the position relationship between the mobile phone and the human body is the third position relationship; the set angle is 15°; If the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the center point of the mobile phone detection frame and the line connecting the two eyes and the line connecting the two eyes is less than or equal to the set angle, then the position relationship between the mobile phone and the human body is the fourth position relationship; If the distances between the eyes and the center point of the mobile phone detection frame are not equal, the position relationship between the mobile phone and the human body is the fifth position relationship; The Face Mesh model is used to determine the state of both eyes according to the human body detection frame, specifically including: Using the Face Mesh model to determine the key features of the eyes according to the human body detection frame; the key features of the eyes include: the positions of the inner and outer eye corners and the vertices of the rectangle tangent to the eyeball; The eye aspect ratio is calculated according to the eye key point features, and the specific formula is: Wherein, EAR represents the aspect ratio of the eye, p1 represents the position of the inner corner of the eye, p4 represents the position of the outer corner of the eye, p2 represents the position of the first vertex of the rectangle tangent to the eyeball, p3 represents the position of the second vertex of the rectangle tangent to the eyeball, p5 represents the position of the third vertex of the rectangle tangent to the eyeball, p6 represents the position of the fourth vertex of the rectangle tangent to the eyeball, and |||| represents the L2 norm; If the eye aspect ratio of at least one eye is greater than a first setting value, the state of the two eyes is that at least one eye is open; the first setting value is greater than zero; the first setting value is 0.3; If the difference between the eye aspect ratio of both eyes and zero is less than a second setting value, the state of both eyes is that both eyes are closed; the second setting value is greater than zero and less than the first setting value; the second setting value is 0.005; Determining the usage of the mobile phone according to the positional relationship between the mobile phone and the human body and the state of the eyes specifically includes: If the position relationship between the mobile phone and the human body is the first position relationship, then the mobile phone usage status is that the mobile phone is not used; If the position relationship between the mobile phone and the human body is the second position relationship, then the mobile phone usage condition is using the mobile phone to make a call; If the position relationship between the mobile phone and the human body is the third position relationship, and the state of the two eyes is that at least one eye is open, then the mobile phone usage is using the mobile phone for entertainment; If the position relationship between the mobile phone and the human body is the third position relationship, and the state of the two eyes is that both eyes are closed, then the mobile phone usage status is that the mobile phone is not used; If the position relationship between the mobile phone and the human body is the fourth position relationship, and the state of the two eyes is that at least one eye is open, then the mobile phone usage is taking photos with the mobile phone; If the position relationship between the mobile phone and the human body is the fourth position relationship, and the state of the two eyes is that both eyes are closed, then the mobile phone usage status is that the mobile phone is not used; If the position relationship between the mobile phone and the human body is the fifth position relationship, the mobile phone usage status is that the mobile phone is not used.

2. The method for detecting mobile phone usage according to claim 1, characterized in that: The key features of the human body include: the positions of the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

3. The method for detecting mobile phone usage according to claim 2, characterized in that: The position relationship between the mobile phone and the human body includes: a first position relationship, a second position relationship, a third position relationship, a fourth position relationship and a fifth position relationship; The first position relationship is that the positions of the mobile phone and the two wrists of the human body do not overlap; The second position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter; the wrist holding the phone is the wrist that coincides with the position of the mobile phone; The third position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, and the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes is greater than the set angle; The fourth position relationship is that the position of the mobile phone coincides with at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of the two ears as the center and the distance between the two shoulders as the diameter, and the two eyes are equidistant from the center point of the mobile phone detection frame, and the angle between the line connecting the center point of the mobile phone detection frame and the center point of the two eyes and the line connecting the two eyes is less than or equal to the set angle; The fifth position relationship is that the position of the mobile phone coincides with that of at least one wrist of the human body, and the wrist holding the phone is not within a circle with the center of both ears as the center and the distance between the shoulders as the diameter, and the two eyes are not equidistant from the center point of the mobile phone detection frame.

4. The method for detecting mobile phone usage according to claim 1, characterized in that: Also includes: A judgment result message is generated according to the usage of the mobile phone and synchronized to the unified monitoring and alarm platform.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for detecting mobile phone usage as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting the usage of a mobile phone as described in any one of claims 1 to 4 are implemented.

Images