Mobile phone playing behavior detection method and device

Abstract

The invention relates to the technical field of use monitoring of mobile communication equipment, and discloses a mobile phone playing behavior detection method and device, and the method comprises the following steps: S1, information collection: collecting video, image and sound information of a real-time scene; s2, data preprocessing: preprocessing the video, image and sound information; s3, face recognition: acquiring face data according to the processed video and image information; s4, attitude analysis; s5, analyzing the image; s6, sound analysis; s7, comprehensive analysis: integrating video, picture and sound information to judge whether the mobile phone is played or not; and S8, outputting a result. The system is reasonable in design, the mobile phone playing behavior is automatically detected by collecting image, video and sound information, the mobile phone using behavior can be efficiently and accurately monitored and managed on the premise that the privacy of a user is not invaded, the system is suitable for various environments such as schools, enterprises and vehicles, and public safety and working efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile communication device usage monitoring, and in particular to a method and device for detecting mobile phone playing behavior. Background Art

[0002] With the rapid development of information technology, people are becoming more and more dependent on mobile phones. It is not uncommon for people to be passive and even cause accidents due to playing with mobile phones. Over-reliance on and improper use of mobile phones may lead to distraction, reduced work efficiency and even safety hazards. Especially in situations that require concentration, such as driving, meetings, and classes, playing with mobile phones may have serious consequences. Therefore, problems such as slowdowns and safety risks caused by playing with mobile phones need to be taken seriously. If personnel are monitored through human supervision, the labor cost will increase; therefore, we propose a method and device for detecting mobile phone playing behavior to solve this problem. Summary of the invention

[0003] The purpose of the present invention is to solve the disadvantage that monitoring personnel through human supervision will increase labor costs, and to propose a method and device for detecting mobile phone playing behavior.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for detecting mobile phone playing behavior comprises the following steps:

[0006] S1, information collection: video, image and sound information of real-time scenes;

[0007] S2, data preprocessing: preprocessing video, image, and sound information;

[0008] S3, face recognition: obtaining face data based on the processed video and image information;

[0009] S4, posture analysis: Combined with the posture estimation algorithm, the human body movements are analyzed to determine whether the mobile phone is being used;

[0010] S5, image analysis: judging whether a mobile phone is being used by distinguishing light and dark and color distribution in the scene image;

[0011] S6, sound analysis: by capturing the sound of touching the mobile phone screen or the sound of the mobile phone speaker in the sound information, it can be judged whether the mobile phone is in use;

[0012] S7, comprehensive analysis: Comprehensive video, picture and sound information to determine whether the user is playing with the phone;

[0013] S8. Result output: output the result of determining whether the user is playing with the mobile phone.

[0014] Preferably, in S4, when analyzing human body movements, firstly, 2D key point monitoring and skeleton positioning of the human body are performed, and the spatial posture relationship of specific joints of the human body is constructed, and posture estimation is performed to determine whether the movements of the arms and hands are consistent with the posture of holding a mobile phone, and the judgment is made based on this information and the preset two state models of "playing with the mobile phone" and "not playing with the mobile phone".

[0015] Preferably, S2 includes the following steps:

[0016] S201, video processing: eliminating irrelevant information in the video, restoring and enhancing effective information, and adjusting the frame rate and resolution of the video to meet the requirements of the analysis algorithm;

[0017] S202, image processing: adjust brightness, contrast and clarity to make the image more suitable for analysis, improve image quality, highlight key features, resize the image as needed, crop out irrelevant areas, focus on the area where the potential mobile phone is held, convert the color space of the image to make it easier to detect mobile phone screens of specific colors or brightness, remove noise from the image to avoid interfering with the subsequent image recognition process, and extract key features of the image, such as shape, texture, color, etc., for further analysis;

[0018] S203, sound processing: eliminating noise and cutting out invalid sound clips.

[0019] Preferably, in S5, when performing image analysis, the following steps are included:

[0020] S501, obtaining images of multiple people in the same area, and comparing the images of faces of different people, and distinguishing the possible mobile phone playing behaviors based on the images of different people;

[0021] S502, obtaining and comparing image information of the same person at different time periods, and eliminating the influence of ambient light to determine whether the person is playing with the mobile phone.

[0022] Preferably, in S6, when performing sound analysis, the following steps are included:

[0023] S601, key press sound: detect the frequency and combination of user key presses. High frequency key press sound may mean that the user is typing or playing games on the mobile phone.

[0024] S602, application prompt sound: Many applications will emit specific prompt sounds, such as message notifications, game sound effects, etc. If these sounds appear frequently, it may indicate that the user is using the phone to perform certain activities;

[0025] S603, sound of switching between call and non-call state: If you hear the mobile phone frequently make sounds of connecting or hanging up a call when the phone is not in a call state, this may indicate that the user is answering a call or ending a call.

[0026] The present invention also provides a mobile phone playing behavior detection device, comprising:

[0027] Information collection module, used to collect video, image and sound information;

[0028] A data processing module, used for processing the acquired information, the data processing module is connected to the information acquisition module;

[0029] A data analysis module, used to analyze the processed information to determine whether there is mobile phone playing behavior, and the data analysis module is connected to the information collection module;

[0030] The result output module is used to output the judgment result, and the result output module is connected to the data analysis module.

[0031] Preferably, the information acquisition module includes a high-definition camera, a high-definition camera and a recorder, the high-definition camera is used to obtain scene video information, the high-definition camera is used to obtain image information, and the recorder is used to obtain sound information.

[0032] Preferably, the data processing module includes a video processing unit, an image processing unit and a sound processing unit.

[0033] Preferably, the data analysis module includes:

[0034] A posture analysis unit is used to analyze mobile phone playing behavior by combining video and posture estimation algorithms;

[0035] An image analysis unit, used to analyze image information to determine mobile phone playing behavior;

[0036] A sound analysis unit, used to analyze sound information to determine mobile phone playing behavior;

[0037] Comprehensive analysis unit, used to integrate audio, video and image information, match and fuse them, and check the accuracy of judgment on mobile phone playing behavior;

[0038] The result integration unit is used to integrate the analysis results.

[0039] Compared with the prior art, the present invention provides a method and device for detecting mobile phone playing behavior, which has the following beneficial effects:

[0040] (1) obtaining video, image and sound information of real-time scenes through high-definition cameras, high-definition cameras and recorders, and processing the obtained information through a data processing module, wherein the video processing unit eliminates irrelevant information in the video, restores and enhances effective information, and adjusts the frame rate and resolution of the video to meet the requirements of the analysis algorithm; the image processing unit adjusts the image brightness, contrast and clarity to make the image more suitable for analysis, improves the image quality, highlights key features, adjusts the image size as needed, cuts out irrelevant areas, focuses on the area where the potential mobile phone is held, converts the color space of the image to make it easier to detect mobile phone screens of specific colors or brightness, removes noise in the image to avoid interfering with the subsequent image recognition process, and extracts key features of the image, such as shape, texture, color, etc., for further analysis; the sound processing unit eliminates noise and cuts out invalid sound clips; the sound processing unit;

[0041] (2) The processed information is analyzed by the data analysis module to determine whether there is any mobile phone playing behavior. The posture analysis unit first performs 2D key point monitoring and skeleton positioning of the human body, and constructs the spatial posture relationship of specific joints of the human body, and performs posture estimation to determine whether the movements of the arms and hands are consistent with the posture of holding the mobile phone. Based on this information and the preset "playing with the mobile phone" and "not playing with the mobile phone" state models, the image analysis unit obtains multiple images of the same area, and compares the images of different people's faces. Based on the images of different people, possible mobile phone playing behavior is judged, and image information of the same person in different time periods is obtained and compared. The influence of ambient light is excluded to determine whether there is mobile phone playing behavior. The sound analysis unit detects the frequency and combination of user key presses. High-frequency key press sounds may mean that the user is typing or playing games on the mobile phone. Many applications will emit specific prompt sounds, such as message notifications, game sound effects, etc. If these sounds appear frequently, it may indicate that the user is using the mobile phone to perform certain activities. If the mobile phone is heard to frequently make connecting or hanging up sounds when not in a call state, it may indicate that the user is answering or ending a call.

[0042] (3) By collecting images, videos and sound information, the system can automatically detect mobile phone use behavior, efficiently and accurately monitor and manage mobile phone use behavior without infringing user privacy. It is suitable for a variety of environments such as schools, enterprises, and vehicles, and helps to improve public safety and work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a method for detecting mobile phone playing behavior proposed by the present invention;

[0044] Figure 2 This is a system block diagram of a mobile phone playing behavior detection device proposed by the present invention.

[0045] In the figure: 1. Information acquisition module; 2. Data processing module; 3. Data analysis module; 4. Result output module. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0047] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0048] Reference Figure 1 , a method for detecting mobile phone playing behavior, comprising the following steps:

[0049] S1, information collection: video, image and sound information of real-time scenes;

[0050] S2, data preprocessing: preprocessing video, image, and sound information;

[0051] S3, face recognition: obtaining face data based on the processed video and image information;

[0052] S4, posture analysis: Combined with the posture estimation algorithm, the human body movements are analyzed to determine whether the mobile phone is being used;

[0053] S5, image analysis: judging whether a mobile phone is being used by distinguishing light and dark and color distribution in the scene image;

[0054] S6, sound analysis: by capturing the sound of touching the mobile phone screen or the sound of the mobile phone speaker in the sound information, it can be judged whether the mobile phone is in use;

[0055] S7, comprehensive analysis: Comprehensive video, picture and sound information to determine whether the user is playing with the phone;

[0056] S8. Result output: output the result of determining whether the user is playing with the mobile phone.

[0057] In this embodiment, in S4, when analyzing human body movements, firstly, 2D key point monitoring and skeleton positioning of the human body are performed, and the spatial posture relationship of specific joints of the human body is constructed, and posture estimation is performed to determine whether the movements of the arms and hands are consistent with the posture of holding a mobile phone, and the judgment is made based on this information and the preset "playing with mobile phone" and "not playing with mobile phone" state models.

[0058] In this embodiment, S2 includes the following steps:

[0059] S201, video processing: eliminating irrelevant information in the video, restoring and enhancing effective information, and adjusting the frame rate and resolution of the video to meet the requirements of the analysis algorithm;

[0060] S202, image processing: adjust brightness, contrast and clarity to make the image more suitable for analysis, improve image quality, highlight key features, resize the image as needed, crop out irrelevant areas, focus on the area where the potential mobile phone is held, convert the color space of the image to make it easier to detect mobile phone screens of specific colors or brightness, remove noise from the image to avoid interfering with the subsequent image recognition process, and extract key features of the image, such as shape, texture, color, etc., for further analysis;

[0061] S203, sound processing: eliminating noise and cutting out invalid sound clips.

[0062] In this embodiment, in S5, when performing image analysis, the following steps are included:

[0063] S501, obtaining images of multiple people in the same area, and comparing the images of faces of different people, and distinguishing the possible mobile phone playing behaviors based on the images of different people;

[0064] S502, obtaining and comparing image information of the same person at different time periods, and eliminating the influence of ambient light to determine whether the person is playing with the mobile phone.

[0065] In this embodiment, in S6, when performing sound analysis, the following steps are included:

[0066] S601, key press sound: detect the frequency and combination of user key presses. High frequency key press sound may mean that the user is typing or playing games on the mobile phone.

[0067] S602, application prompt sound: Many applications will emit specific prompt sounds, such as message notifications, game sound effects, etc. If these sounds appear frequently, it may indicate that the user is using the phone to perform certain activities;

[0068] S603, sound of switching between call and non-call state: If you hear the mobile phone frequently make sounds of connecting or hanging up a call when the phone is not in a call state, this may indicate that the user is answering a call or ending a call.

[0069] Reference Figure 2 The present invention also provides a mobile phone playing behavior detection device, comprising:

[0070] Information collection module 1, used to collect video, image and sound information;

[0071] The data processing module 2 is used to process the acquired information, and the data processing module 2 is connected to the information acquisition module 1;

[0072] Data analysis module 3, used for analyzing the processed information to determine whether there is mobile phone playing behavior, the data analysis module 3 is connected to the information collection module 1;

[0073] The result output module 4 is used to output the judgment result, and the result output module 4 is connected to the data analysis module 3.

[0074] In this embodiment, the information collection module 1 includes a high-definition camera, a high-definition camera and a recorder. The high-definition camera is used to obtain scene video information, the high-definition camera is used to obtain image information, and the recorder is used to obtain sound information.

[0075] In this embodiment, the data processing module 2 includes a video processing unit, an image processing unit and a sound processing unit.

[0076] In this embodiment, the data analysis module 3 includes:

[0077] A posture analysis unit is used to analyze mobile phone playing behavior by combining video and posture estimation algorithms;

[0078] An image analysis unit, used to analyze image information to determine mobile phone playing behavior;

[0079] A sound analysis unit, used to analyze sound information to determine mobile phone playing behavior;

[0080] Comprehensive analysis unit, used to integrate audio, video and image information, match and fuse them, and check the accuracy of judgment on mobile phone playing behavior;

[0081] The result integration unit is used to integrate the analysis results.

[0082] In this embodiment, when in use, video, image and sound information of real-time scenes are obtained through high-definition cameras, high-definition cameras and recorders, and the obtained information is processed through a data processing module, wherein the video processing unit eliminates irrelevant information in the video, replies and enhances effective information, and adjusts the frame rate and resolution of the video to meet the requirements of the analysis algorithm. The image processing unit adjusts the image brightness, contrast and clarity to make the image more suitable for analysis, improves the image quality, highlights key features, adjusts the image size as needed, crops out irrelevant areas, focuses on the area of ​​potential handheld mobile phones, converts the color space of the image to make it easier to detect mobile phone screens of specific colors or brightness, removes noise in the image to avoid interfering with the subsequent image recognition process, and extracts key features of the image, such as shape, texture, color, etc., for further analysis. The sound processing unit eliminates noise and crops invalid sound clips. The sound processing unit;

[0083] The processed information is analyzed by the data analysis module to determine whether there is a behavior of playing with the phone. The posture analysis unit first performs 2D key point monitoring and skeleton positioning of the human body, and constructs the spatial posture relationship of specific joints of the human body, and performs posture estimation to determine whether the movements of the arms and hands are consistent with the posture of holding the mobile phone. According to this information and the preset "playing with the phone" and "not playing with the phone" state models, the image analysis unit obtains images of multiple people in the same area, and compares the images of different people's faces. According to the difference in the images of different people, possible mobile phone playing behaviors are judged, and image information of the same person in different time periods is obtained and compared, and the influence of ambient light is excluded to determine whether there is a behavior of playing with the phone. The sound analysis unit detects the frequency and combination of user keystrokes. High-frequency keystroke sounds may mean that the user is typing or playing games on the phone. Many applications will emit specific prompt sounds, such as message notifications, game sound effects, etc. If these sounds appear frequently, it may indicate that the user is using the phone to perform certain activities. If the phone is heard to frequently make a sound of connecting or hanging up a call when not in a call state, it may indicate that the user is answering or ending a call.

[0084] The result integration unit integrates the analysis results and the result output unit outputs the result of whether there is the mobile phone playing behavior.

[0085] The standard parts used in the present invention can all be purchased from the market, and special-shaped parts can be customized according to the instructions and the drawings. The specific connection methods of each part adopt conventional means such as mature bolts, rivets, welding, etc. in the prior art. Machinery, parts and equipment all adopt conventional models in the prior art, and the circuit connection adopts the conventional connection method in the prior art, which will not be described in detail here.

Claims

1. A method for detecting mobile phone playing behavior, characterized in that: The steps include: S1, information collection: video, image and sound information of real-time scenes; S2, data preprocessing: preprocessing video, image, and sound information; S3, face recognition: obtaining face data based on the processed video and image information; S4, posture analysis: Combined with the posture estimation algorithm, the human body movements are analyzed to determine whether the mobile phone is being used; S5, image analysis: judging whether a mobile phone is being used by distinguishing light and dark and color distribution in the scene image; S6, sound analysis: by capturing the sound of touching the mobile phone screen or the sound of the mobile phone speaker in the sound information, it can be judged whether the mobile phone is in use; S7, comprehensive analysis: Comprehensive video, picture and sound information to determine whether the user is playing with the phone; S8. Result output: output the result of determining whether the user is playing with the mobile phone.

2. A method for detecting mobile phone playing behavior according to claim 1, characterized in that: In S4, when analyzing human body movements, firstly, 2D key point monitoring and skeleton positioning of the human body are performed, and the spatial posture relationship of specific joints of the human body is constructed, and posture estimation is performed to determine whether the movements of the arms and hands are consistent with the posture of holding a mobile phone, and the judgment is made based on this information and the preset "playing with mobile phone" and "not playing with mobile phone" state models.

3. A method for detecting mobile phone playing behavior according to claim 1, characterized in that: The S2 includes the following steps: S201, video processing: eliminating irrelevant information in the video, restoring and enhancing effective information, and adjusting the frame rate and resolution of the video to meet the requirements of the analysis algorithm; S202, image processing: adjust brightness, contrast and clarity to make the image more suitable for analysis, improve image quality, highlight key features, resize the image as needed, crop out irrelevant areas, focus on the area where the potential mobile phone is held, convert the color space of the image to make it easier to detect mobile phone screens of specific colors or brightness, remove noise from the image to avoid interfering with the subsequent image recognition process, and extract key features of the image, such as shape, texture, color, etc., for further analysis; S203, sound processing: eliminating noise and cutting out invalid sound clips.

4. A method for detecting mobile phone playing behavior according to claim 1, characterized in that: In the S5, when performing image analysis, the following steps are included: S501, obtaining images of multiple people in the same area, and comparing the images of faces of different people, and distinguishing the possible mobile phone playing behaviors based on the images of different people; S502, obtaining and comparing image information of the same person at different time periods, and eliminating the influence of ambient light to determine whether the person is playing with the mobile phone.

5. A method for detecting mobile phone playing behavior according to claim 1, characterized in that: In the above S6, when performing sound analysis, the following steps are included: S601, key press sound: detect the frequency and combination of user key presses. High frequency key press sound may mean that the user is typing or playing games on the mobile phone. S602, application prompt sound: Many applications will emit specific prompt sounds, such as message notifications, game sound effects, etc. If these sounds appear frequently, it may indicate that the user is using the phone to perform certain activities; S603, sound of switching between call and non-call state: If you hear the mobile phone frequently make sounds of connecting or hanging up a call when the phone is not in a call state, this may indicate that the user is answering a call or ending a call.

6. A mobile phone playing behavior detection device, characterized in that: include: An information collection module (1) is used to collect video, image and sound information; A data processing module (2), used for processing the acquired information, the data processing module (2) being connected to the information acquisition module (1); A data analysis module (3) is used to analyze the processed information to determine whether there is any mobile phone playing behavior, and the data analysis module (3) is connected to the information collection module (1); A result output module (4) is used to output the judgment result, and the result output module (4) is connected to the data analysis module (3).

7. A mobile phone playing behavior detection device according to claim 6, characterized in that: The information acquisition module (1) comprises a high-definition camera, a high-definition camera and a recorder, wherein the high-definition camera is used to obtain scene video information, the high-definition camera is used to obtain image information, and the recorder is used to obtain sound information.

8. The mobile phone playing behavior detection device according to claim 6, characterized in that: The data processing module (2) comprises a video processing unit, an image processing unit and a sound processing unit.

9. The mobile phone playing behavior detection device according to claim 6, characterized in that: The data analysis module (3) comprises: A posture analysis unit is used to analyze mobile phone playing behavior by combining video and posture estimation algorithms; An image analysis unit, used to analyze image information to determine mobile phone playing behavior; A sound analysis unit, used to analyze sound information to determine mobile phone playing behavior; The comprehensive analysis unit is used to integrate audio, video and image information, match and fuse them, and check the accuracy of judgments on mobile phone playing behavior.

Images