Mobile phone playing behavior detection method and device, readable storage medium and terminal equipment

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a mobile phone playing behavior detection method and device, a computer readable storage medium and terminal equipment. The method comprises the following steps: acquiring a to-be-detected target image frame; detecting whether a mobile phone exists in the target image frame; if the mobile phone exists in the target image frame, detecting whether a human hand exists in the target image frame; if the human hand exists in the target image frame, detecting whether the human hand holds a mobile phone; and if the hand holds the mobile phone, determining that a mobile phone playing behavior is detected in the target image frame. Through the method, whether the mobile phone playing behavior exists or not can be determined according to whether the mobile phone, the hand and the condition that the hand holds the mobile phone exist in the target image frame or not, the multi-dimensional features of the mobile phone, the hand and the handheld mobile phone are comprehensively considered, the false detection probability is reduced, efficient detection of the mobile phone playing behavior is achieved, and the user experience is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method, device, computer-readable storage medium and terminal device for detecting mobile phone playing behavior. Background Art

[0002] With the development of smart technology, more and more smart terminals are entering the user's field of vision. During the user's work or study, smart terminals can be used for supervision and companionship, and can detect mobile phone playing behaviors in time and give reminders to improve the user's work or study efficiency. However, the existing methods for detecting mobile phone playing behaviors are inefficient, resulting in poor user experience. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, apparatus, computer-readable storage medium and terminal device for detecting mobile phone playing behavior to solve the problem that the existing mobile phone playing behavior detection methods are inefficient and result in poor user experience.

[0004] A first aspect of an embodiment of the present application provides a method for detecting mobile phone playing behavior, which may include:

[0005] Acquire the target image frame to be detected;

[0006] Detecting whether there is a mobile phone in the target image frame;

[0007] If there is a mobile phone in the target image frame, detecting whether there is a human hand in the target image frame;

[0008] If there is a human hand in the target image frame, detecting whether the human hand is holding the mobile phone;

[0009] If the person's hand is holding the mobile phone, it is determined that the behavior of playing with the mobile phone is detected in the target image frame.

[0010] In a specific implementation manner of the first aspect, after detecting whether there is a human hand in the target image frame, the method may further include:

[0011] If there is no human hand in the target image frame, detecting whether the mobile phone is in a screen-on state;

[0012] If the mobile phone is in a screen-on state, it is determined that the mobile phone playing behavior is detected in the target image frame.

[0013] In a specific implementation of the first aspect, the detecting whether the human hand holds the mobile phone may include:

[0014] Get the mobile phone detection frame and the hand detection frame;

[0015] Determine whether there is an overlapping area between the mobile phone detection frame and the human hand detection frame;

[0016] If there is an overlapping area between the mobile phone detection frame and the human hand detection frame, a target sub-image is extracted from the target image frame according to the mobile phone detection frame and the human hand detection frame; wherein the target sub-image is a sub-image that includes both the mobile phone and the human hand;

[0017] Detect whether the human hand in the target sub-image is holding the mobile phone.

[0018] In a specific implementation of the first aspect, detecting whether a mobile phone exists in the target image frame may include:

[0019] Use a preset mobile phone detection model to detect whether there is a mobile phone in the target image frame;

[0020] The training process of the mobile phone detection model may include:

[0021] Obtain a preset mobile phone detection training sample set; wherein the mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding mobile phone detection frame;

[0022] The sample image of each training sample in the mobile phone detection training sample set is used as input, and the mobile phone detection frame of each training sample in the mobile phone detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the mobile phone detection model.

[0023] In a specific implementation of the first aspect, detecting whether a human hand exists in the target image frame may include:

[0024] Use a preset human hand detection model to detect whether there is a human hand in the target image frame;

[0025] The training process of the hand detection model may include:

[0026] Obtain a preset human hand detection training sample set; wherein the human hand detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding human hand detection frame;

[0027] The sample image of each training sample in the hand detection training sample set is used as input, and the hand detection frame of each training sample in the hand detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the hand detection model.

[0028] In a specific implementation of the first aspect, the detecting whether the human hand holds the mobile phone may include:

[0029] Using a preset handheld mobile phone detection model to detect whether the human hand is holding the mobile phone;

[0030] The training process of the handheld mobile phone detection model may include:

[0031] Obtaining a preset handheld mobile phone detection training sample set; wherein the handheld mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label;

[0032] The sample image of each training sample in the handheld mobile phone detection training sample set is used as input, and the state label of each training sample in the handheld mobile phone detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the handheld mobile phone detection model.

[0033] In a specific implementation of the first aspect, after determining that the mobile phone playing behavior is detected in the target image frame, the method may further include:

[0034] Execute a prompt operation corresponding to the mobile phone playing behavior.

[0035] A second aspect of an embodiment of the present application provides a mobile phone playing behavior detection device, which may include:

[0036] An image frame acquisition module, used to acquire a target image frame to be detected;

[0037] A mobile phone detection module, used to detect whether there is a mobile phone in the target image frame;

[0038] A human hand detection module, used to detect whether there is a human hand in the target image frame if there is a mobile phone in the target image frame;

[0039] A hand-held detection module, used for detecting whether a human hand is holding the mobile phone if there is a human hand in the target image frame;

[0040] The behavior determination module is used to determine that the behavior of playing with the mobile phone is detected in the target image frame if the person is holding the mobile phone.

[0041] In a specific implementation of the second aspect, the mobile phone playing behavior detection device may further include:

[0042] A screen-on detection module, used to detect whether the mobile phone is in a screen-on state if there is no human hand in the target image frame;

[0043] The behavior determination module is used to determine that the behavior of playing with the mobile phone is detected in the target image frame if the mobile phone is in the screen-on state.

[0044] In a specific implementation of the second aspect, the handheld detection module may include:

[0045] A detection frame acquisition unit, used to acquire a mobile phone detection frame and a human hand detection frame;

[0046] An overlapping area determination unit, used to determine whether there is an overlapping area between the mobile phone detection frame and the human hand detection frame;

[0047] A target sub-image extraction unit, configured to extract a target sub-image from the target image frame according to the mobile phone detection frame and the human hand detection frame if there is an overlapping area between the mobile phone detection frame and the human hand detection frame; wherein the target sub-image is a sub-image including both the mobile phone and the human hand;

[0048] A hand-held detection unit is used to detect whether the human hand in the target sub-image is holding the mobile phone.

[0049] In a specific implementation of the second aspect, the mobile phone detection module may include:

[0050] A model detection unit, used to detect whether a mobile phone exists in the target image frame using a preset mobile phone detection model;

[0051] The model detection unit may include:

[0052] The sample set acquisition subunit is used to acquire a preset mobile phone detection training sample set; wherein the mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding mobile phone detection frame;

[0053] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the mobile phone detection training sample set as input and the mobile phone detection frame of each training sample in the mobile phone detection training sample set as expected output to obtain the mobile phone detection model.

[0054] In a specific implementation of the second aspect, the human hand detection module may include:

[0055] A model detection unit, used to detect whether there is a human hand in the target image frame using a preset human hand detection model;

[0056] The model detection unit may include:

[0057] The sample set acquisition subunit is used to acquire a preset human hand detection training sample set; wherein the human hand detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding human hand detection frame;

[0058] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the hand detection training sample set as input and the hand detection frame of each training sample in the hand detection training sample set as expected output to obtain the hand detection model.

[0059] In a specific implementation of the second aspect, the handheld detection module may include:

[0060] A model detection unit, used to detect whether the human hand is holding the mobile phone using a preset handheld mobile phone detection model;

[0061] The model detection unit may include:

[0062] A sample set acquisition subunit is used to acquire a preset handheld mobile phone detection training sample set; wherein the handheld mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label;

[0063] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the handheld mobile phone detection training sample set as input and the state label of each training sample in the handheld mobile phone detection training sample set as expected output to obtain the handheld mobile phone detection model.

[0064] In a specific implementation of the second aspect, the mobile phone playing behavior detection device may further include:

[0065] The prompt operation execution module is used to execute the prompt operation corresponding to the mobile phone playing behavior.

[0066] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods for detecting mobile phone playing behavior are implemented.

[0067] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned methods for detecting mobile phone playing behavior when executing the computer program.

[0068] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any one of the above-mentioned mobile phone playing behavior detection methods.

[0069] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain the target image frame to be detected; detect whether there is a mobile phone in the target image frame; if there is a mobile phone in the target image frame, detect whether there is a human hand in the target image frame; if there is a human hand in the target image frame, detect whether the human hand is holding the mobile phone; if the human hand is holding the mobile phone, determine that the behavior of playing with the mobile phone is detected in the target image frame. Through the embodiments of the present application, it is possible to determine whether there is the behavior of playing with the mobile phone based on whether there is a mobile phone, a human hand, and the situation of the human hand holding the mobile phone in the target image frame. The multi-dimensional characteristics of the mobile phone, the human hand, and the hand-held mobile phone are comprehensively considered, which can reduce the probability of false detection, achieve efficient detection of the behavior of playing with the mobile phone, and help improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 This is a flow chart of an embodiment of a method for detecting mobile phone playing behavior in an embodiment of the present application;

[0072] Figure 2 This is a schematic diagram of the mobile phone detection frame;

[0073] Figure 3 A schematic diagram of the overlapping area between the mobile phone detection frame and the human hand detection frame;

[0074] Figure 4 Schematic diagram of the process of determining the target subgraph;

[0075] Figure 5 This is a structural diagram of an embodiment of a mobile phone playing behavior detection device in an embodiment of the present application;

[0076] Figure 6 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0079] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0080] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0081] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0082] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0083] With the development of smart technology, more and more smart terminals are entering the user's field of vision. During the user's work or study, smart terminals can be used for supervision and companionship, and can detect mobile phone playing behaviors in time and give reminders to improve the user's work or study efficiency. However, the existing methods for detecting mobile phone playing behaviors are inefficient, resulting in poor user experience.

[0084] In view of this, the embodiments of the present application provide a method, device, computer-readable storage medium and terminal device for detecting mobile phone playing behavior, so as to solve the problem that the existing methods for detecting mobile phone playing behavior are inefficient, resulting in poor user experience. Through the embodiments of the present application, it is possible to determine whether there is mobile phone playing behavior based on whether there is a mobile phone, a human hand and a human hand holding a mobile phone in the target image frame, and comprehensively consider the multi-dimensional characteristics of the mobile phone, the human hand and the hand-held mobile phone, which can reduce the probability of false detection, realize efficient detection of mobile phone playing behavior, and help improve user experience.

[0085] It should be noted that the executor of the method of the present application is a terminal device, specifically, it can be a common computing device such as a learning machine, a tablet computer, a smart desk lamp, a desktop computer, a notebook, a PDA, or other computing devices.

[0086] See also Figure 1 In the embodiment of the present application, an embodiment of a method for detecting mobile phone playing behavior may include:

[0087] Step S101: Acquire a target image frame to be detected.

[0088] In an embodiment of the present application, one or more image frames may be collected and formed into an image sequence, and then the image frames in the image sequence may be analyzed frame by frame to detect whether there is any mobile phone playing behavior.

[0089] For the convenience of description, the method of the present application will be described below by taking any image frame in the image sequence (referred to as the target image frame) as an example.

[0090] It is understandable that a pre-stored image sequence can be obtained from a preset memory module or server, and any frame in the image sequence can be used as the target image frame; or, a camera acquisition device pre-installed on the terminal device can be used to acquire the image sequence, and any frame therein can be used as the target image frame.

[0091] Step S102: Detect whether there is a mobile phone in the target image frame.

[0092] In an embodiment of the present application, a preset mobile phone detection model can be used to detect whether a mobile phone exists in a target image frame, wherein the mobile phone detection model can be a pre-trained artificial intelligence model for performing mobile phone detection.

[0093] It can be understood that the artificial intelligence models used in the embodiments of the present application can all be artificial intelligence models commonly used in the prior art for image detection, including but not limited to convolutional neural network (CNN), recurrent neural network (RNN), YOLO, Fast R-CNN, Faster R-CNN and other common artificial intelligence models in the prior art.

[0094] It is understandable that before using the mobile phone detection model to detect whether there is a mobile phone in the target image frame, the mobile phone detection model can be trained. Specifically, a preset mobile phone detection training sample set can be obtained, each of which can include a sample image and a corresponding mobile phone detection frame; then, the sample image of each training sample in the mobile phone detection training sample set is used as input, and the mobile phone detection frame of each training sample in the mobile phone detection training sample set is used as the expected output, and the initial artificial intelligence model is trained to obtain the mobile phone detection model.

[0095] It should be noted that in order to make the detection results of the mobile phone detection model more robust, the mobile phone detection training sample set may include a preset number of positive sample images and negative sample images in various application scenarios, where the positive sample images are sample images with mobile phones, and the negative sample images are sample images without mobile phones. In addition, considering that in actual mobile phone usage scenarios, there may be situations where the mobile phone is blocked by human body parts or clothing, sample images where the mobile phone is partially blocked can also be obtained, such as Figure 2 As shown in (a), at this time, the mobile phone detection frame corresponding to the sample image can be fully marked with the mobile phone based on prior knowledge, as shown in Figure 2 (b) shows the mobile phone detection frame.

[0096] In a possible embodiment, when training the mobile phone detection model, the mobile phone detection training sample set may be divided into a training set and a test set according to a preset ratio, and the training set may be used for model training, and the test set may be used for model testing.

[0097] In another possible embodiment, the mobile phone detection model can also be updated according to the detection result of the mobile phone detection model to improve the detection accuracy of the mobile phone detection model. For example, if there is a mobile phone in the target image frame, the target image frame can be used as a positive sample image in the mobile phone detection training sample set to update the mobile phone detection model; if there is no mobile phone in the target image frame, the target image frame can be used as a negative sample image in the mobile phone detection training sample set to update the mobile phone detection model.

[0098] In an embodiment of the present application, if there is no mobile phone in the target image frame, step S107 can be executed; if there is a mobile phone in the target image frame, a mobile phone detection frame can be output and step S103 can be executed to further detect whether there is a human hand in the target image frame.

[0099] Step S103: Detect whether there is a human hand in the target image frame.

[0100] In an embodiment of the present application, a preset hand detection model can be used to detect whether there is a hand in the target image frame, wherein the hand detection model can be a pre-trained artificial intelligence model for hand detection.

[0101] It is understandable that before using the hand detection model to detect whether there is a hand in the target image frame, the hand detection model can be trained. Specifically, a preset hand detection training sample set can be obtained, each of which can include a sample image and a corresponding hand detection frame; then, the sample image of each training sample in the hand detection training sample set is used as input, and the hand detection frame of each training sample in the hand detection training sample set is used as the expected output, and the initial artificial intelligence model is trained to obtain the hand detection model.

[0102] It should be noted that in order to make the detection results of the hand detection model more robust, the hand detection training sample set may include a preset number of positive sample images and negative sample images under various application scenarios, wherein the positive sample images are sample images with human hands, and the negative sample images are sample images without human hands.

[0103] In a possible embodiment, when training a hand detection model, the hand detection training sample set may be divided into a training set and a test set according to a preset ratio, and the training set may be used for model training, and the test set may be used for model testing.

[0104] In another possible embodiment, the hand detection model can also be updated according to the detection result of the hand detection model to improve the detection accuracy of the hand detection model. For example, if there is a hand in the target image frame, the target image frame can be used as a positive sample image in the hand detection training sample set to update the hand detection model; if there is no hand in the target image frame, the target image frame can be used as a negative sample image in the hand detection training sample set to update the hand detection model.

[0105] In an embodiment of the present application, if there is a human hand in the target image frame, a human hand detection frame can be output and step S104 can be executed; if there is no human hand in the target image frame, step S105 can be executed to detect whether the mobile phone is in a bright screen state to assist in determining whether there is any behavior of playing with the mobile phone.

[0106] Step S104: Detect whether the person is holding a mobile phone.

[0107] It is understandable that even if a mobile phone and a human hand exist in the target image frame at the same time, it does not necessarily mean that there is a behavior of playing with the mobile phone. For example, the user can just place the mobile phone in the work area, but there is no behavior of playing with the mobile phone. For another example, a human hand can be suspended above the mobile phone, but there is no behavior of playing with the mobile phone. Therefore, in an embodiment of the present application, after detecting that a mobile phone and a human hand exist in the target image frame at the same time, a preset handheld mobile phone detection model can be used to detect whether the human hand in the target image frame is holding a mobile phone, wherein the handheld mobile phone detection model can be a pre-trained artificial intelligence model for handheld mobile phone detection.

[0108] It is understandable that before using the handheld mobile phone detection model to detect whether a person's hand in the target image frame is holding a mobile phone, the handheld mobile phone detection model can be trained. Specifically, a preset handheld mobile phone detection training sample set can be obtained, each of which can include a sample image and a corresponding state label; wherein the state label can include a handheld state and a non-handheld state, the handheld state can indicate that the person's hand is holding a mobile phone, and the non-handheld state can indicate that the person's hand is not holding a mobile phone; then, the sample image of each training sample in the handheld mobile phone detection training sample set is used as input, and the state label of each training sample in the handheld mobile phone detection training sample set is used as the expected output, and the initial artificial intelligence model is trained to obtain a handheld mobile phone detection model.

[0109] It should be noted that in order to make the detection results of the handheld mobile phone detection model more robust, the handheld mobile phone detection training sample set may include a preset number of positive sample images and negative sample images under various application scenarios, where the positive sample images are sample images of hands holding mobile phones, and the negative sample images are sample images of hands not holding mobile phones.

[0110] In a possible embodiment, when training a handheld mobile phone detection model, a handheld mobile phone detection training sample set may be divided into a training set and a test set according to a preset ratio, and the training set may be used for model training, and the test set may be used for model testing.

[0111] In another possible embodiment, the handheld mobile phone detection model can also be updated according to the detection result of the handheld mobile phone detection model to improve the accuracy of the handheld mobile phone detection model. For example, if the target image frame shows a human hand holding a mobile phone, the target image frame can be used as a positive sample image in the handheld mobile phone detection training sample set to update the handheld mobile phone detection model; if the target image frame shows a human hand not holding a mobile phone, the target image frame can be used as a negative sample image in the handheld mobile phone detection training sample set to update the handheld mobile phone detection model.

[0112] In another embodiment provided by the present application, it is possible to first determine whether there is an overlapping area between the mobile phone detection frame and the human hand detection frame. If there is an overlapping area, a handheld mobile phone detection model is used to detect whether the human hand is holding a mobile phone; if there is no overlapping area, it can be directly considered that the human hand is not holding a mobile phone. This embodiment is described in detail below:

[0113] In this embodiment, the mobile phone detection frame output by the mobile phone detection model and the hand detection frame output by the hand detection model can be obtained, and it can be determined whether there is an overlapping area between the mobile phone detection frame and the hand detection frame.

[0114] It is understandable that if Figure 3 As shown, if a person holds a mobile phone, there will be an overlapping area between the mobile phone detection frame and the human hand detection frame. Therefore, if there is an overlapping area between the mobile phone detection frame and the human hand detection frame, it can be directly determined that the mobile phone playing behavior is detected in the target image frame; if there is no overlapping area between the mobile phone detection frame and the human hand detection frame, the target sub-image can be extracted from the target image frame, and the preset hand-held mobile phone detection model can be used to detect whether the person is holding the mobile phone, wherein the target sub-image is a sub-image that includes both the mobile phone and the human hand.

[0115] In a possible embodiment, a target rectangular frame including both the mobile phone and the human hand can be determined according to the mobile phone detection frame and the human hand detection frame, such as Figure 4 As shown in (a), the length and width of the target rectangular box are then expanded by a preset distance to obtain Figure 4 (b) The target subgraph shown.

[0116] In another possible embodiment, the circumscribed rectangle of the mobile phone detection frame and the human hand detection frame can be determined based on the mobile phone detection frame and the human hand detection frame, and then the length and width of the circumscribed rectangle can be expanded by a preset distance to determine the target sub-image.

[0117] After obtaining the target sub-image, a preset handheld mobile phone detection model can be used to detect whether the person in the target sub-image is holding a mobile phone. If the person in the target sub-image is holding a mobile phone, it can be directly determined that the behavior of playing with the mobile phone is detected in the target image frame; if the person in the target sub-image is not holding a mobile phone, it can be directly determined that the behavior of playing with the mobile phone is not detected in the target image frame.

[0118] After detection using the handheld mobile phone detection model, if it is detected that the human hand is holding the mobile phone, step S106 can be executed; if it is detected that the human hand is not holding the mobile phone, step S107 can be executed.

[0119] Step S105: Detect whether the mobile phone is in the screen-on state.

[0120] It is understandable that in actual mobile phone playing scenarios, users may place their mobile phones on a desktop or a mobile phone stand to watch videos and other mobile phone playing behaviors. At this time, when using the hand detection model to detect the hand of the target image frame, the hand may not be detected. Therefore, in the embodiment of the present application, a preset screen-on detection model can also be used to detect whether the mobile phone is in the screen-on state, wherein the screen-on detection model can be a pre-trained artificial intelligence model for mobile phone screen-on detection.

[0121] It is understandable that before using the screen detection model to detect whether the mobile phone screen is on, the screen detection model can be trained. Specifically, a preset screen detection training sample set can be obtained, each of which can include a sample image and a corresponding state label, wherein the state label can include a screen-on state and a screen-off state, the screen-on state can indicate that the mobile phone screen is on, and the screen-off state can indicate that the mobile phone screen is off; then, the sample image of each training sample in the screen detection training sample set is used as input, and the state label of each training sample in the screen detection training sample set is used as the expected output, and the initial artificial intelligence model is trained to obtain the screen detection model.

[0122] It should be noted that in order to make the detection results of the screen-on detection model more robust, the screen-on detection training sample set may include a preset number of positive sample images and negative sample images under various application scenarios, where the positive sample images are sample images when the phone is in the screen-on state, and the negative sample images are sample images when the phone is in the screen-off state.

[0123] In a possible embodiment, when training the bright screen detection model, the bright screen detection training sample set can be divided into a training set and a test set according to a preset ratio, and the training set is used for model training, and the test set is used for model testing.

[0124] In the embodiment of the present application, if the mobile phone is in the screen-on state, step S106 can be executed; if the mobile phone is in the screen-off state, step S107 can be executed.

[0125] Step S106: Determine whether the mobile phone playing behavior is detected in the target image frame.

[0126] In an embodiment of the present application, if a mobile phone and a human hand exist in the target image frame at the same time, and the human hand is holding the mobile phone, or, if a mobile phone exists in the target image frame but no human hand, and the mobile phone is in a bright screen state, it can be determined that the behavior of playing with the mobile phone is detected in the target image frame.

[0127] It can be understood that in order to improve the user's work efficiency, if it is determined that the behavior of playing with the mobile phone is detected in the target image frame, a prompt operation corresponding to the behavior of playing with the mobile phone can be executed, which may include but is not limited to one or more prompt operations such as emitting a prompt sound or prompt voice and displaying prompt text to remind the user to concentrate on work.

[0128] In a possible embodiment, when the prompt operation is performed, an accompanying auxiliary perception operation may also be performed. For example, a preset light may be emitted to perform an auxiliary prompt. For another example, a vibration motor may be controlled to vibrate at a preset frequency to perform an auxiliary prompt.

[0129] In another possible embodiment, the user can set the prompt operation on the client that establishes a communication connection with the terminal device. For example, the user can choose to use only a preset light as a prompt operation. For another example, the user can choose to emit a prompt sound effect and display a prompt text as a prompt operation.

[0130] Step S107: Determine whether the mobile phone playing behavior is not detected in the target image frame.

[0131] In an embodiment of the present application, if there is no mobile phone in the target image frame, or if there is a mobile phone but no human hand in the target image frame, and the mobile phone is in the screen-off state, or if there are both a mobile phone and a human hand in the target image frame but the human hand is not holding the mobile phone, it can be determined that no mobile phone playing behavior is detected in the target image frame.

[0132] It can be understood that, in a possible embodiment, after completing the detection of the target image frame, the above method can be repeatedly executed on the next image frame of the target image frame to achieve continuous detection of the user's mobile phone playing behavior.

[0133] In summary, the embodiment of the present application obtains a target image frame to be detected; detects whether there is a mobile phone in the target image frame; if there is a mobile phone in the target image frame, detects whether there is a human hand in the target image frame; if there is a human hand in the target image frame, detects whether the human hand is holding the mobile phone; if the human hand is holding the mobile phone, determines that the behavior of playing with the mobile phone is detected in the target image frame. Through the embodiment of the present application, it is possible to determine whether there is a behavior of playing with the mobile phone based on whether there is a mobile phone, a human hand, and the situation of the human hand holding the mobile phone in the target image frame. The multi-dimensional characteristics of the mobile phone, the human hand, and the hand-held mobile phone are comprehensively considered, which can reduce the probability of false detection, achieve efficient detection of the behavior of playing with the mobile phone, and help improve the user experience.

[0134] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0135] Corresponding to the method for detecting mobile phone playing behavior described in the above embodiment, Figure 5 A structural diagram of an embodiment of a mobile phone playing behavior detection device provided in an embodiment of the present application is shown.

[0136] In an embodiment of the present application, a mobile phone playing behavior detection device may include:

[0137] The image frame acquisition module 501 is used to acquire the target image frame to be detected;

[0138] A mobile phone detection module 502, used to detect whether there is a mobile phone in the target image frame;

[0139] A human hand detection module 503 is used to detect whether there is a human hand in the target image frame if there is a mobile phone in the target image frame;

[0140] A hand-held detection module 504 is used to detect whether a human hand is holding the mobile phone if there is a human hand in the target image frame;

[0141] The behavior determination module 505 is used to determine that the behavior of playing with the mobile phone is detected in the target image frame if the person is holding the mobile phone.

[0142] In a specific implementation of the embodiment of the present application, the mobile phone playing behavior detection device may also include:

[0143] A screen-on detection module, used to detect whether the mobile phone is in a screen-on state if there is no human hand in the target image frame;

[0144] The behavior determination module is used to determine that the behavior of playing with the mobile phone is detected in the target image frame if the mobile phone is in the screen-on state.

[0145] In a specific implementation of the embodiment of the present application, the handheld detection module may include:

[0146] A detection frame acquisition unit, used to acquire a mobile phone detection frame and a human hand detection frame;

[0147] An overlapping area determination unit, used to determine whether there is an overlapping area between the mobile phone detection frame and the human hand detection frame;

[0148] A target sub-image extraction unit, configured to extract a target sub-image from the target image frame according to the mobile phone detection frame and the human hand detection frame if there is an overlapping area between the mobile phone detection frame and the human hand detection frame; wherein the target sub-image is a sub-image including both the mobile phone and the human hand;

[0149] A hand-held detection unit is used to detect whether the human hand in the target sub-image is holding the mobile phone.

[0150] In a specific implementation of the embodiment of the present application, the mobile phone detection module may include:

[0151] A model detection unit, used to detect whether a mobile phone exists in the target image frame using a preset mobile phone detection model;

[0152] The model detection unit may include:

[0153] The sample set acquisition subunit is used to acquire a preset mobile phone detection training sample set; wherein the mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding mobile phone detection frame;

[0154] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the mobile phone detection training sample set as input and the mobile phone detection frame of each training sample in the mobile phone detection training sample set as expected output to obtain the mobile phone detection model.

[0155] In a specific implementation of the embodiment of the present application, the hand detection module may include:

[0156] A model detection unit, used to detect whether there is a human hand in the target image frame using a preset human hand detection model;

[0157] The model detection unit may include:

[0158] The sample set acquisition subunit is used to acquire a preset human hand detection training sample set; wherein the human hand detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding human hand detection frame;

[0159] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the hand detection training sample set as input and the hand detection frame of each training sample in the hand detection training sample set as expected output to obtain the hand detection model.

[0160] In a specific implementation of the embodiment of the present application, the handheld detection module may include:

[0161] A model detection unit, used to detect whether the human hand is holding the mobile phone using a preset handheld mobile phone detection model;

[0162] The model detection unit may include:

[0163] A sample set acquisition subunit is used to acquire a preset handheld mobile phone detection training sample set; wherein the handheld mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label;

[0164] The model training subunit is used to train the initial artificial intelligence model with the sample image of each training sample in the handheld mobile phone detection training sample set as input and the state label of each training sample in the handheld mobile phone detection training sample set as expected output to obtain the handheld mobile phone detection model.

[0165] In a specific implementation of the embodiment of the present application, the mobile phone playing behavior detection device may also include:

[0166] The prompt operation execution module is used to execute the prompt operation corresponding to the mobile phone playing behavior.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] Figure 6 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0170] like Figure 6 As shown, the terminal device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned mobile phone playing behavior detection method embodiments are implemented, for example Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of modules 501 to 505 are shown.

[0171] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the terminal device 6.

[0172] The terminal device 6 can be a computing device such as a learning machine, a tablet computer, a smart desk lamp, a desktop computer, a notebook, a PDA, etc. It can be understood by those skilled in the art that Figure 6 It is only an example of the terminal device 6 and does not constitute a limitation of the terminal device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 6 may also include input and output devices, network access devices, buses, etc.

[0173] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0174] The memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 6. Further, the memory 61 may also include both an internal storage unit of the terminal device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0175] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0178] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0182] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting mobile phone playing behavior, characterized in that: include: Acquire the target image frame to be detected; Detecting whether there is a mobile phone in the target image frame; If there is a mobile phone in the target image frame, detecting whether there is a human hand in the target image frame; If there is a human hand in the target image frame, detecting whether the human hand is holding the mobile phone; If the person's hand is holding the mobile phone, it is determined that the behavior of playing with the mobile phone is detected in the target image frame.

2. The method for detecting mobile phone playing behavior according to claim 1, characterized in that: After detecting whether there is a human hand in the target image frame, the method further includes: If there is no human hand in the target image frame, detecting whether the mobile phone is in a screen-on state; If the mobile phone is in a screen-on state, it is determined that the mobile phone playing behavior is detected in the target image frame.

3. The method for detecting mobile phone playing behavior according to claim 1, characterized in that: The detecting whether the human hand holds the mobile phone comprises: Get the mobile phone detection frame and the hand detection frame; Determine whether there is an overlapping area between the mobile phone detection frame and the human hand detection frame; If there is an overlapping area between the mobile phone detection frame and the human hand detection frame, a target sub-image is extracted from the target image frame according to the mobile phone detection frame and the human hand detection frame; wherein the target sub-image is a sub-image that includes both the mobile phone and the human hand; Detect whether the human hand in the target sub-image is holding the mobile phone.

4. The method for detecting mobile phone playing behavior according to claim 1, characterized in that: The detecting whether there is a mobile phone in the target image frame comprises: Use a preset mobile phone detection model to detect whether there is a mobile phone in the target image frame; The training process of the mobile phone detection model includes: Obtain a preset mobile phone detection training sample set; wherein the mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding mobile phone detection frame; The sample image of each training sample in the mobile phone detection training sample set is used as input, and the mobile phone detection frame of each training sample in the mobile phone detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the mobile phone detection model.

5. The method for detecting mobile phone playing behavior according to claim 1, characterized in that: The detecting whether there is a human hand in the target image frame comprises: Use a preset human hand detection model to detect whether there is a human hand in the target image frame; The training process of the hand detection model includes: Obtain a preset human hand detection training sample set; wherein the human hand detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding human hand detection frame; The sample image of each training sample in the hand detection training sample set is used as input, and the hand detection frame of each training sample in the hand detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the hand detection model.

6. The method for detecting mobile phone playing behavior according to claim 1, characterized in that: The detecting whether the human hand holds the mobile phone comprises: Using a preset handheld mobile phone detection model to detect whether the human hand is holding the mobile phone; The training process of the handheld mobile phone detection model includes: Obtaining a preset handheld mobile phone detection training sample set; wherein the handheld mobile phone detection training sample set includes a preset number of training samples, each training sample includes a sample image and a corresponding state label; The sample image of each training sample in the handheld mobile phone detection training sample set is used as input, and the state label of each training sample in the handheld mobile phone detection training sample set is used as expected output, and the initial artificial intelligence model is trained to obtain the handheld mobile phone detection model.

7. The method for detecting mobile phone playing behavior according to any one of claims 1 to 6, characterized in that: After determining that the mobile phone playing behavior is detected in the target image frame, the method further includes: Execute a prompt operation corresponding to the mobile phone playing behavior.

8. A mobile phone playing behavior detection device, characterized in that: include: An image frame acquisition module, used to acquire a target image frame to be detected; A mobile phone detection module, used to detect whether there is a mobile phone in the target image frame; A human hand detection module, used to detect whether there is a human hand in the target image frame if there is a mobile phone in the target image frame; A hand-held detection module, used for detecting whether a human hand is holding the mobile phone if there is a human hand in the target image frame; The behavior determination module is used to determine that the behavior of playing with the mobile phone is detected in the target image frame if the person is holding the mobile phone.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting mobile phone playing behavior as described in any one of claims 1 to 7 are implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for detecting mobile phone playing behavior as described in any one of claims 1 to 7 are implemented.

Images