Screen state control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a screen state control method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first target image collected by a camera when a screen of terminal equipment is in a screen-off state; when it is detected that the target user exists in the first target image, detecting whether a target object exists in the first target image; and when the target object exists and the state of the target user satisfies a preset screen-on condition, controlling the screen of the terminal device to be in a screen-on state. By implementing the embodiment of the invention, the screen state of the terminal equipment can be controlled by detecting the target user, the target object and the state of the target user, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of electronic devices, including but not limited to a method, device, electronic device, and storage medium for controlling the screen state. Background Art

[0002] In today's digital learning era, intelligent devices are increasingly widely used in learning scenarios. Intelligent terminal devices such as tablet computers, laptops, and learning machines have become important tools for learners to acquire knowledge and carry out learning activities. In the related art, the screen state control method of terminal devices is relatively single, usually based on manual operations by users, such as pressing the power button or touching the screen to turn on and off the screen. In the case of no operation by users, the terminal device generally turns off the screen at a fixed time or does not turn off the screen, resulting in a poor user experience. Summary of the Invention

[0003] In view of this, the screen state control method, device, electronic device, and storage medium provided by the embodiments of the present application can control the screen state of the terminal device by detecting the target user, target object, and the state of the target user, improving the user experience.

[0004] The first aspect of the embodiments of the present application discloses a method for controlling the screen state, which is applied to a terminal device. The terminal device includes a camera, and the method includes:

[0005] When the screen of the terminal device is in the off state, obtain a first target image collected by the camera;

[0006] When it is detected that there is a target user in the first target image, detect whether there is a target object in the first target image;

[0007] When the target object exists and the state of the target user meets the preset screen-on condition, control the screen of the terminal device to be in the on state.

[0008] As an optional implementation manner, in the first aspect of the embodiments of the present application, the target object is a pen. When the target object exists and the state of the target user meets the preset screen-on condition, controlling the screen of the terminal device to be in the on state includes:

[0009] When it is detected that the hand of the target user is in a pen-holding state, control the screen of the terminal device to be in the on state.

[0010] As an alternative implementation, in the first aspect of the embodiments of the present application, when the target object is a book, controlling the screen of the terminal device to be in the lit state when the target object exists and the state of the target user meets the preset lit screen condition includes:

[0011] When it is detected that the line of sight of the target user is directed towards the book, controlling the screen of the terminal device to be in the lit state.

[0012] As an alternative implementation, in the first aspect of the embodiments of the present application, when the target objects are a book and a pen, controlling the screen of the terminal device to be in the lit state when the target objects exist and the state of the target user meets the preset lit screen condition includes:

[0013] When it is detected that the hand of the target user is not in the pen - holding state and the line of sight of the target user is directed towards the book, controlling the screen of the terminal device to be in the lit state.

[0014] As an alternative implementation, in the first aspect of the embodiments of the present application, the method further includes:

[0015] Obtaining a second target image within a preset period, where the second target image is obtained when the terminal device is in the lit state;

[0016] When it is detected that the target user does not exist in the second target image, controlling the screen of the terminal device to be in the screen - off state.

[0017] As an alternative implementation, in the first aspect of the embodiments of the present application, detecting whether a target object exists in the first target image includes:

[0018] Inputting the first target image into a pre - trained target detection model to obtain the detection result of the target object, where the target detection model is obtained by training an initial model with sample images containing the target object.

[0019] As an alternative implementation, in the first aspect of the embodiments of the present application, controlling the screen of the terminal device to be in the lit state when the target object exists and the state of the target user meets the preset lit screen condition includes:

[0020] When the target object exists and it is detected that the line of sight of the target user is directed towards the terminal device, controlling the screen of the terminal device to be in the lit state.

[0021] In a second aspect of the embodiments of the present application, a screen state control device is disclosed, which is applied to a terminal device. The terminal device includes a camera. The device includes:

[0022] A target image acquisition module, configured to acquire a first target image collected by the camera when the screen of the terminal device is in the screen-off state;

[0023] A target object detection module, configured to detect whether there is a target object in the first target image when it is detected that there is a target user in the first target image;

[0024] A screen state control module, configured to control the screen of the terminal device to be in the screen-on state when the target object exists and the state of the target user meets a preset screen-on condition.

[0025] In a third aspect of the embodiments of the present application, an electronic device is disclosed, including a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor is caused to implement the method as described above.

[0026] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the method as described above is implemented.

[0027] Compared with the related art, the embodiments of the present application at least include the following beneficial effects:

[0028] A screen state control method, device, electronic device, and storage medium disclosed in the embodiments of the present application acquire a first target image collected by a camera when the screen of the terminal device is in the screen-off state, detect whether there is a target object in the first target image when it is detected that there is a target user in the first target image, and control the screen of the terminal device to be in the screen-on state when the target object exists and the state of the target user meets a preset screen-on condition. It can control the screen state of the terminal device by detecting the target user, target object, and the state of the target user, improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings herein are incorporated into the specification and form a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to explain the technical solutions of the present application.

[0030] Figure 1 It is a flowchart of a screen state control method in an embodiment;

[0031] Figure 2 It is a scenario diagram of a screen state control method in an embodiment;

[0032] Figure 3 It is a scenario diagram of the screen state control method in another embodiment;

[0033] Figure 4 It is a scenario diagram of the screen state control method in yet another embodiment;

[0034] Figure 5 It is a schematic flow diagram of the screen state control method in another embodiment;

[0035] Figure 6 It is a schematic flow diagram of the screen state control method in yet another embodiment;

[0036] Figure 7 It is a block diagram of the screen state control device in one embodiment;

[0037] Figure 8 It is a block diagram of the structure of an electronic device in one embodiment. Detailed implementation manners

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will further describe the specific technical solutions of the present application in detail with reference to the accompanying drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0040] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0041] It should be noted that the terms "first / second / third" involved in the embodiments of the present application are used to distinguish similar or different objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged in a specific order or sequence when allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] At a time when digital learning is becoming more and more popular, smart terminal devices such as tablet computers, learning machines, and laptops have become indispensable tools for students in their learning process. These terminal devices carry a wealth of learning resources and have greatly expanded the boundaries and methods of learning. However, in the related technologies, the screens of terminal devices mostly rely on a preset fixed time to automatically turn off the screen, or rely entirely on the user to manually press the power button or touch the screen to switch between the on and off states. Manual operation to turn on the screen may cause the device to be unable to be turned on and used in time due to user negligence, busyness, or failure to notice that the screen is off. In addition, when using the timed screen-off mode, it is easy for the screen to turn off for no reason, requiring the user to frequently turn on the screen, resulting in a poor user experience of the terminal device during the learning process.

[0043] In view of this, the embodiments of the present application disclose a screen state control method, device, electronic device and storage medium, which can control the screen state of the terminal device by detecting the target user, target object and the state of the target user, thereby improving the user experience. Detailed descriptions are given below.

[0044] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a screen state control method in an embodiment. Figure 1 As shown, the method may include the following steps:

[0045] Step 101: When the screen of the terminal device is in a screen-off state, a first target image captured by a camera is obtained.

[0046] The screen state control method provided in the embodiment of the present application is applied to a terminal device. In practical applications, the terminal device may include but is not limited to a learning machine, a tutoring machine, a learning tablet, a smart phone, a watch, a tablet computer, a desktop computer, a laptop computer, a portable terminal, a personal computer (PC), etc. The computer system in the terminal device may be Windows, Linux, IOS or Unix. The embodiment of the present application does not specifically limit the type of the terminal device.

[0047] When the screen of the terminal device is in the screen-off state, the first target image of the environment where the terminal device is currently located is collected through the built-in camera of the terminal device. Optionally, when the screen of the terminal device is in the screen-off state, the camera enters a low-power standby mode. At preset time intervals (for example, every few seconds or minutes), the camera is automatically activated and an image is collected, that is, the first target image. In the embodiments of the present application, the first target image may include various information of the user in the learning scenario, such as the user's state, items in the learning environment (such as books, pens, learning tools, etc.), and the interaction between the user and these items. By analyzing the first target image, it can be determined whether the user is in a learning state, so as to determine whether to switch the screen from the screen-off state to the screen-on state.

[0048] In some embodiments, the camera can collect the first target image by being triggered at time intervals, or can be assisted by other sensors such as a proximity sensor or a light sensor. For example, when the terminal device detects that the user is approaching, the camera is automatically activated and an image is collected.

[0049] To reduce power consumption, the activation frequency of the camera can be optimized according to the actual usage scenario. For example, in a night or a dim environment, the acquisition interval can be appropriately extended. The resolution of the collected image can be adjusted according to actual needs to balance power consumption and recognition accuracy. For example, a lower resolution image can be used to reduce the processing burden. In addition, in the embodiments of the present application, as long as the terminal device is in the running state, whether it is in the screen-off state or the screen-on state, the camera is always running in the background.

[0050] Step 102, when it is detected that there is a target user in the first target image, detect whether there is a target object in the first target image.

[0051] In the embodiments of the present application, after the terminal device obtains the first target image collected by the camera, the processor can detect the first target image. For example, through a face detection algorithm or a pre-trained face detection model, it can be identified whether there is a face in the first target image. If a face is detected, it is determined that there is a target user in the first target image.

[0052] In some embodiments, the target user may also be a person preset in the terminal device. For example, the face image of the target user can be pre-stored in the terminal device. When it is detected that there is a face in the first target image, the terminal device can compare the face in the first target image with the pre-stored face image. If the similarity of the comparison result is higher than the preset threshold, it is determined that there is a target user in the first target image, and it can be further detected whether there is a target object in the first target image; if the similarity of the comparison result is lower than the preset threshold, it is determined that there is no target user in the first target image. When there is no target user in the first target image, the terminal device re-acquires the first target image captured by the camera and maintains the screen-off state.

[0053] In some application scenarios, it may occur that multiple people appear within the shooting range of the camera at the same time. At this time, the terminal device can adopt a multi-target face detection algorithm to simultaneously identify multiple facial features and compare them with the pre-set target user face database.

[0054] In some embodiments, detecting whether there is a target object in the first target image includes:

[0055] Input the first target image into a pre-trained target detection model to obtain the detection result of the target object. The target detection model is obtained by training an initial model with sample images containing the target object.

[0056] In the embodiments of the present application, the target object may be books, pens, and other learning supplies in a learning scenario, which are not specifically limited in the embodiments of the present application. The sample image is an image containing the target object. The sample image should cover the states of the target object under different scenarios, angles, and lighting conditions to ensure the comprehensiveness of model learning. Then, the sample image is labeled, and the labeling content includes the category of the target object, such as "book", "pen", etc., and the bounding box of the target object, that is, the position of the target object in the image. The initial model is trained with the sample image and the corresponding labeling information of the sample image. During the training process, the initial model calculates the difference between the prediction result and the true label, adjusts the model parameters to minimize the loss, and uses an optimization algorithm to update the model parameters to gradually improve the detection accuracy of the model.

[0057] In some embodiments, the initial model may be a YOLO (You Only Look Once) model, an SSD (SingleShot MultiBox Detector) model, a Faster R-CNN (Faster Region-based Convolutional Neural Network) model, an HRNet (High-Resolution Network) model, a RetinaNet (Retina Network) model, etc., and the embodiments of the present application are not limited thereto. The initial model after training is the target detection model. It can be understood that the training process of the initial model and the trained target detection model can be deployed in the terminal device or in the server. The terminal device can upload the first target image to the server and receive the detection result of the target object sent by the server.

[0058] The terminal device inputs the first target image into the trained target detection model. The target detection model can identify whether there is a target object in the first target image, the category of the target object, and the position of the target object in the first target image. When the detection result of the target object is that there is a target object, the terminal device can further detect the state of the target user; when the detection result of the target object is that there is no target object, the terminal device re-acquires the first target image collected by the camera and maintains the screen-off state.

[0059] By adopting the above embodiments, the detection result of the target object is obtained through the pre-trained target detection model, which can accurately identify whether there is a target object, the category of the target object, and the position of the target object in the first target image, and can reduce the situations of misjudgment and missed judgment, providing a more reliable basis for the subsequent detection of the user state.

[0060] Step 103: When the target object exists and the state of the target user meets the preset screen-on condition, control the screen of the terminal device to be in the screen-on state.

[0061] In some possible embodiments, step 103, when the target object exists and the state of the target user meets the preset screen-on condition, controlling the screen of the terminal device to be in the screen-on state may include at least the following three exemplary methods.

[0062] The first example:

[0063] Please refer to Figure 2 , Figure 2It is a scenario diagram of a screen state control method in an embodiment. In some embodiments, the target object is a pen. When the target object exists and the state of the target user meets the preset screen-on condition, controlling the screen of the terminal device to be in the screen-on state includes:

[0064] When it is detected that the hand of the target user is in a pen-holding state, controlling the screen of the terminal device to be in the screen-on state.

[0065] As Figure 2 shown, when the terminal device 203 detects the pen 201, it further detects the state of the target user. In one embodiment, the preset screen-on condition is that the hand 202 of the target user is in a pen-holding state. When the terminal device 203 detects that the hand 202 of the target user is in a pen-holding state, it controls the screen to be in the screen-on state.

[0066] In the embodiments of the present application, the pen can be any type of writing tool, including pencils, ballpoint pens, fountain pens, etc. By inputting the first target image into a pre-trained target detection model, it can be recognized whether there is a pen in the first target image. When there is a pen in the first target image, the first target image can be input into a pre-trained pose detection model. The pose detection model detects the key point positions of the hand of the target user. The key points of the hand include the wrist, finger joints, etc. Based on the detected hand key points, the terminal device can recognize the pose and movement of the hand of the target user, and thus can determine whether the hand of the user is in a pen-holding state. The pose detection model can be deployed in the terminal device or in the server. When the pose detection model is deployed in the server, the terminal device can upload the first target image to the server and receive the detection result of the hand pose of the target user sent by the server.

[0067] As an alternative implementation, the pose detection model can be a pre-trained model based on deep learning, such as a pose detection model based on a convolutional neural network (CNN). The hand image data with the key point positions of finger joints, wrists, etc. marked is input into the pre-trained model for training. The trained pose detection model is used to detect the hand pose of the target user. The terminal device can match the detection result of the target user state with the preset screen-on condition. When the matching is successful, it controls the screen of the terminal device to be in the screen-on state. Exemplarily, when the terminal device detects that there is a pen in the first target image and the hand pose of the target user is in a pen-holding state, it switches the screen from the screen-off state to the screen-on state.

[0068] In some embodiments, some special situations may occur that affect the judgment of the pen-holding state. For example, when the user holds a pen in their hand but is not ready to write, simply detecting the pen-holding action may cause the screen to light up unnecessarily. To avoid this situation, time series analysis can be combined to judge the duration of the user's pen-holding and the stability of the action. If the user holds the pen for a short time and the action changes frequently, the terminal device may not perform the screen lighting operation temporarily; only when the user's pen-holding action remains stable for a period of time, it is considered that the user has the intention to write, and then the screen lighting is controlled. This optimization method can further improve the accuracy of the screen lighting judgment and enhance the user experience.

[0069] The terminal device realizes the screen lighting operation by detecting the postures of the pen and the hand of the target user, enabling the target user to write without manually operating the screen lighting, which can improve the intelligent level of device interaction, reduce manual operation steps, and enhance the user experience.

[0070] The second example:

[0071] Please refer to Figure 3 , Figure 3 which is a scenario diagram of the screen state control method in another embodiment. In some embodiments, the target object is a book. When the target object exists and the state of the target user meets the preset screen lighting conditions, controlling the screen of the terminal device to be in the lit state includes:

[0072] When it is detected that the target user's line of sight is directed at the book, controlling the screen of the terminal device to be in the lit state.

[0073] As Figure 3 shown, when the terminal device 303 detects that there is a book 301 in the first target image through the target detection model, it further detects the state of the target user 302. In one embodiment, the preset screen lighting condition is that the line of sight of the target user 302 is directed at the book 301. When the terminal device 303 detects that the line of sight of the target user 302 is directed at the book 301, it controls the screen to be in the lit state.

[0074] In an embodiment of the present application, by inputting the first target image into a pre-trained target detection model, it can be identified whether there is a book in the first target image. When there is a book in the first target image, the first target image can be input into a pre-trained line of sight detection model to obtain the line of sight detection result of the target user. The line of sight detection model can be deployed in a terminal device or a server. Exemplarily, the face can be located by a face detection algorithm, and then the eye area can be segmented from the face area, and the extracted eye area image can be input into a pre-trained line of sight detection model. The line of sight detection model calculates the user's line of sight direction vector based on eye features (such as pupil center, corneal reflection point, iris texture, etc.) and related algorithms (such as calculations based on geometric relationships, deep learning algorithms, etc.), and the vector represents the direction of the human eye's line of sight in three-dimensional space.

[0075] Based on the line of sight direction vector of the human eye obtained by the line of sight detection model and the book position information obtained by the target detection model, a geometric relationship between the two is established. For example, the intersection of the line of sight direction vector and the plane where the book is located can be calculated to determine whether the intersection is within the bounding box of the book. If the intersection is within the bounding box of the book and the direction of the line of sight direction vector points to the book, it can be determined that the target user's line of sight is directed to the book; otherwise, it is considered that the target user's line of sight is not directed to the book. In order to improve the accuracy and robustness of the judgment, some constraints can be added, such as setting the angle threshold between the line of sight and the plane of the book, the duration of judging the line of sight direction, etc. When the angle threshold and / or the duration are within the preset range, it can be determined that the target user's line of sight is directed to the book.

[0076] When the target detection model detects the book and the line of sight detection model detects that the target user's line of sight is directed toward the book, the terminal device switches the screen from the off state to the on state.

[0077] The terminal device realizes the screen light-up operation by detecting whether the target user's eyes are directed towards the book. When it is detected that the target user is in a learning state, the screen of the terminal device can automatically switch to the screen light-up state, which can improve the intelligence level of device interaction, reduce manual operation steps, and improve user experience.

[0078] The third example:

[0079] See also Figure 4 , Figure 4 is a scene diagram of a screen state control method in another embodiment. In some embodiments, when the target object exists and the state of the target user satisfies a preset screen-on condition, controlling the screen of the terminal device to be in a screen-on state includes:

[0080] When the target object exists and it is detected that the target user's line of sight is directed at the terminal device, control the screen of the terminal device to be in the lit state.

[0081] As Figure 4 shown, when it is detected that the target object exists, but it is not detected that the target user 401 interacts with the target object, it is possible to further detect whether the line of sight of the target user 401 is directed at the terminal device 402. When it is detected that the line of sight of the target user 401 is directed at the terminal device 402, control the screen of the terminal device 402 to be in the lit state.

[0082] In the embodiments of the present application, the target object can be books, pens, and other learning supplies in a learning scenario, etc. When the terminal device detects that there is a book or a pen in the first target image, or any other learning supply exists, detect whether the target user's hand is in a pen-holding state, and detect whether the target user's line of sight is directed at the book. It can be understood that when there is a pen in the first target image and it is detected that the target user's hand is in a pen-holding state, or when there is a book in the first target image and it is detected that the target user's line of sight is directed at the book, the terminal device controls the screen to be in the lit state.

[0083] In some embodiments, when the terminal device detects that the target object exists, but it is detected that the target user's hand is not in a pen-holding state and it is detected that the target user's line of sight is not directed at the book, it is possible to further detect whether the target user's line of sight is directed at the terminal device. The preset lit condition is that the target user's line of sight is directed at the terminal device. Similarly, the first target image can be input into a pre-trained line-of-sight detection model to detect whether the target user's line of sight is directed at the terminal device.

[0084] In some embodiments, the line-of-sight detection model can output the line-of-sight direction vector of the target user, and calculate the angle or projection relationship between this vector and the screen. By judging whether the angle between the vector and the screen normal vector is within a certain threshold range, or whether the projection point of the vector on the screen plane falls within the effective area of the screen, to determine whether the target user's line of sight is directed at the terminal device.

[0085] In some other embodiments, the line-of-sight detection model can also output the probability value that the target user's line of sight is directed at the terminal device. When the probability value is greater than the set threshold, it is considered that the target user's line of sight is directed at the terminal device; when the probability value is less than the set threshold, it is considered that the target user's line of sight is not directed at the terminal device.

[0086] The terminal device realizes the screen-on operation by detecting whether the line of sight of the target object and the target user is directed at the terminal device. When it is detected that the target user is in a learning state, the screen of the terminal device can automatically switch to the screen-on state, which can improve the intelligent level of device interaction, reduce the manual operation steps, and can improve the user experience.

[0087] Next, please refer to Figure 5 , Figure 5 which is a schematic flowchart of a screen state control method in another embodiment. As Figure 5 shown, the method may include the following steps:

[0088] Step 501, obtain a first target image captured by a camera.

[0089] When the screen of the terminal device is in the screen-off state, the camera takes pictures of the surrounding environment to obtain a first target image containing scene information. The specific implementation can refer to the above embodiments and will not be elaborated here. After the terminal device obtains the first target image, it executes the method described in step 502.

[0090] Step 502, detect whether there is a target user in the first target image.

[0091] In some embodiments, the terminal device can identify whether there is a face in the first target image through a face detection algorithm or a pre-trained face detection model, so as to detect whether there is a target user in the first target image. The specific implementation can refer to the above embodiments and will not be elaborated here. When the terminal device detects that there is a target user in the first target image, it executes the method described in step 503. It can be understood that when the terminal device detects that there is no target user in the first target image, it executes the method described in step 501 and maintains the screen-off state.

[0092] Step 503, detect whether there is a book and / or a pen in the first target image.

[0093] In some embodiments, the terminal device may input the first target image into a pre-trained object detection model. The object detection model can identify whether there is a book and / or a pen in the first target image, as well as the position information of the book and / or the pen. For the specific implementation manners, reference may be made to the above embodiments and will not be elaborated herein. When the terminal device detects that there is a book and / or a pen in the first target image, it executes the method described in step 504. It can be understood that when the terminal device detects that there is no book and / or pen in the first target image, it executes the method described in step 501 and maintains the screen-off state. In some other embodiments, as long as the terminal device detects that there is no book in the first target image, regardless of whether there is a pen in the first target image, it executes the method described in step 501.

[0094] Step 504: Detect whether the hand of the target user is in a pen-holding state.

[0095] In the embodiments of the present application, the terminal device may input the first target image into a pre-trained gesture detection model to obtain the hand gesture detection result of the target user. When the gesture detection model identifies that the hand of the target user is in a pen-holding state, it controls the screen of the terminal device to switch from the screen-off state to the screen-on state. For the specific implementation manners, reference may be made to the above embodiments and will not be elaborated herein. When the gesture detection model identifies that the hand of the target user is not in a pen-holding state, it executes the method described in step 505.

[0096] Step 505: Detect whether the line of sight of the target user is directed at the book.

[0097] In some embodiments, the target objects are a book and a pen. When the target objects exist and the state of the target user meets the preset screen-on condition, controlling the screen of the terminal device to be in the screen-on state includes:

[0098] When it is detected that the hand of the target user is not in a pen-holding state and the line of sight of the target user is directed at the book, control the screen of the terminal device to be in the screen-on state.

[0099] In an embodiment of the present application, when the terminal device detects that there is a book and a pen in the first target image, but the hand of the target user is not in the pen-holding state, the terminal device may input the first target image into a pre-trained gaze detection model to obtain the gaze detection result of the target user. The gaze detection model is used to detect whether the gaze of the target user is directed at the book. When it is detected that the hand of the target user is not in the pen-holding state, the terminal device controls the screen to be in a corresponding state according to the gaze detection result of the target user. Exemplarily, when the gaze detection result of the target user is that the gaze of the target user is not directed at the book, the terminal device is controlled to maintain the screen in the off state, and the method described in step 501 is executed. When the gaze detection result of the target user is that the gaze of the target user is directed at the book, the terminal device is controlled to turn on the screen.

[0100] By adopting the above embodiment, the terminal device controls the state of the screen by detecting whether the target object exists and whether the state of the target user meets the preset screen-on condition. When the target user is in the learning state, the terminal device can control the screen to be in the on state without the user manually operating to turn on the screen, which can improve the user experience.

[0101] The above embodiments introduce how to implement the screen-on control of the terminal device when the terminal device is in the off state. Next, how to implement the screen-off control of the terminal device when the terminal device is in the on state will be described.

[0102] Please refer to Figure 6 , Figure 6 which is a schematic flowchart of a screen state control method in another embodiment. As Figure 6 shown, the method may include the following steps:

[0103] Step 601, obtain a second target image within a preset period. The second target image is obtained when the terminal device is in the on state.

[0104] In an embodiment of the present application, the second target image is an image of the current environment of the terminal device collected by the camera of the terminal device when the terminal device is in the on state. When the screen of the terminal device is in the on state, the camera of the terminal device may collect the second target image according to a preset time period. If it is detected that there is a target user in the obtained second target image, the camera re-collects the second target image, and the screen of the terminal device remains in the on state until the terminal device detects that there is no target user in the second target image, and then stops collecting the second target image.

[0105] The preset period is a fixed time interval set in advance, which is also the frequency at which the camera of the terminal device captures the second target image. For example, if the preset period is 5 seconds, then after the screen is turned on, the camera will capture the second target image at the 0th second, and then capture it every 5 seconds, and so on, until the terminal device detects that there is no target user in a certain frame of the second target image, and then stops capturing.

[0106] Step 602, when it is detected that there is no target user in the second target image, control the screen of the terminal device to be in the screen-off state.

[0107] After the terminal device obtains the second target image captured by the camera, it can input the second target image into a pre-trained face detection model to detect whether there is a face in the second target image. Exemplarily, the face detection model can be trained according to the sample images containing faces and the corresponding face annotation information of the sample images. In addition, it is also possible to detect whether there is a face in the second target image through a face detection algorithm.

[0108] As an optional implementation manner, the target user can be a person preset in the terminal device, or a user who appears in the second target image. When the target user is a preset person, the terminal device also needs to further detect whether the user in the second target image is the target user. In some embodiments, when no face is detected in the second target image, it can be determined that there is no target user in the second target image. In other embodiments, when a face is detected in the second target image, but the face of the person preset in the terminal device is not detected, it can also be considered that there is no target user in the second target image.

[0109] When the terminal device detects that there is no target user in the second target image, it can switch the screen from the on state to the off state.

[0110] In some possible embodiments, when no target user is detected in the second target image within the preset period, but obvious occlusion traces are detected in the image, at this time, the terminal device does not immediately turn off the screen, but adds a short waiting time, and continues to capture images for detection during the waiting time. If no target user is detected after the waiting time ends, then perform the screen-off operation.

[0111] The terminal device controls the state of the screen by detecting whether there is a target user in the second target image obtained in the on state. When there is no target user in the second target image, it timely controls the screen of the terminal device to be in the off state, avoiding the screen remaining on continuously after the target user leaves, thus effectively saving power and eliminating the need for the user to operate manually, providing a more convenient and intelligent usage experience for the user.

[0112] In some embodiments, when the terminal device detects that the user is in a learning state and automatically turns on the screen, during the screen-on state, the terminal device can also enable a concentration evaluation mode for user learning. Exemplarily, the camera continuously captures the status information of the user in the learning scenario, and the terminal device extracts the action and eye information of the user during the learning process. For actions, features such as head rotation, body posture changes, and hand movement frequency can be identified through a posture detection model; for eye movements, indicators such as eye gaze direction and blink frequency can be tracked through a gaze detection model. For example, if the head rotates by more than a certain threshold within 30 seconds, it is determined that the head rotates frequently; if the number of blinks per minute exceeds the normal range (15 - 20 times per minute) or is lower than a certain value, it is regarded as abnormal blinking; data such as the proportion of time the eyes are fixed on the terminal device screen or the book are also recorded in real time.

[0113] Using machine learning algorithms, with the action and eye data collected in a large number of different learning states as the training set, a learning concentration evaluation model is constructed. For example, support vector machine (SVM), neural network models, etc. can be used. The learning concentration evaluation model takes the action and eye feature data of the user as input, and through training to learn the relationship between different feature combinations and concentration, outputs a concentration score from 0 to 100, where 0 represents completely unfocused and 100 represents highly focused. For example, if it is detected that the user's head rotates frequently, the eyes wander (the proportion of time looking at the learning content is less than 60%), and the blink frequency is abnormal, the learning concentration evaluation model will give a lower concentration score; while when the user's body remains relatively stable, the eyes are continuously focused on the learning content, and the blink frequency is normal, the score is higher.

[0114] The terminal device can display the user's current learning concentration score in real time. When the concentration score is lower than the set warning value (such as 60 points), the terminal device can issue a reminder, such as popping up a prompt box or playing a soft reminder sound, to remind the user to concentrate. At the same time, after the learning is over, the concentration score can be associated with the learning duration to generate a comprehensive learning status report.

[0115] Adopting the above embodiments, when the terminal device is in the screen-on state, by obtaining the action information and eye information of the user in the learning scenario and combining with the learning concentration evaluation model to obtain the concentration score of the user during the learning process, it can help the user improve learning efficiency and concentration, and can further enhance the user's experience of using the terminal device.

[0116] Please refer to Figure 7 , Figure 7 which is a block diagram of a screen state control device in an embodiment, as Figure 7As shown in the figure, the screen state control device 700 includes: a target image acquisition module 710, a target object detection module 720, and a screen state control module 730, where:

[0117] The target image acquisition module 710 is configured to acquire a first target image collected by the camera when the screen of the terminal device is in the screen-off state;

[0118] The target object detection module 720 is configured to detect whether there is a target object in the first target image when it is detected that there is a target user in the first target image;

[0119] The screen state control module 730 is configured to control the screen of the terminal device to be in the screen-on state when the target object exists and the state of the target user meets a preset screen-on condition.

[0120] In some embodiments, the target object is a pen, and the screen state control module 730 is specifically configured to:

[0121] When it is detected that the hand of the target user is in a pen-holding state, control the screen of the terminal device to be in the screen-on state.

[0122] In some embodiments, the target object is a book, and the screen state control module 730 is specifically configured to:

[0123] When it is detected that the line of sight of the target user is directed at the book, control the screen of the terminal device to be in the screen-on state.

[0124] In some embodiments, the target objects are a book and a pen, and the screen state control module 730 is specifically configured to:

[0125] When it is detected that the hand of the target user is not in a pen-holding state and the line of sight of the target user is directed at the book, control the screen of the terminal device to be in the screen-on state.

[0126] In some embodiments, the target image acquisition module 710 is further configured to:

[0127] Acquire a second target image within a preset period, where the second target image is acquired when the terminal device is in the screen-on state.

[0128] The screen state control module 730 is further configured to:

[0129] When it is detected that there is no target user in the second target image, control the screen of the terminal device to be in the screen-off state.

[0130] In some embodiments, the target object detection module 720 is specifically configured to:

[0131] The first target image is input into a pre-trained target detection model to obtain a detection result of the target object, wherein the target detection model is obtained by training an initial model based on a sample image containing the target object.

[0132] In some embodiments, the screen state control module 730 is further configured to:

[0133] When the target object exists and it is detected that the sight line of the target user is directed toward the terminal device, the screen of the terminal device is controlled to be in a bright screen state.

[0134] The embodiments of the present application disclose a screen state control method, device, electronic device and storage medium. When the screen of the terminal device is in the screen-off state, the first target image captured by the camera is obtained. When the target user is detected in the first target image, the target object is detected in the first target image. When the target object exists and the state of the target user meets the preset screen-on condition, the screen of the terminal device is controlled to be in the screen-on state. By detecting the target user, the target object and the state of the target user, the screen state of the terminal device can be controlled, thereby improving the user experience.

[0135] It should be noted that in the embodiments of this application Figure 7 The division of modules in the screen state control device shown is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0136] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0137] Please refer to Figure 8 , Figure 8 , which is a structural block diagram of an electronic device in an embodiment. As Figure 8 shown, the electronic device 800 may include: a processor 810, a memory 820, and a bus 830.

[0138] Among them, the processor 810 calls the executable program code stored in the memory 820 to execute any one of the screen state control methods disclosed in the embodiments of the present application. Those skilled in the art can understand that Figure 8 the structure of the electronic device shown in does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0139] The processor 810 is configured to, when the screen of the terminal device is in the screen-off state, obtain a first target image collected by the camera; when it is detected that there is a target user in the first target image, detect whether there is a target object in the first target image; when the target object exists and the state of the target user meets the preset screen-on condition, control the screen of the terminal device to be in the screen-on state. For specific details, please refer to the detailed description in the method examples and will not be elaborated here.

[0140] In the embodiments of the present application, the processor 810 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by the hardware processor, or executed by a combination of hardware and software modules in the processor.

[0141] The memory 820 can be used to store software programs and modules. The processor 810 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 820. The memory 820 may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device. In addition, the memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0142] In the embodiments of the present application, the processor 810 is connected to the memory 820 through the bus 830. The bus 830 is in Figure 8is represented by a thick line. The connection manners between other components are only for illustrative purposes and are not restrictive. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 8 is only represented by a thick line, but it does not mean that there is only one bus or one type of bus. An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0143] An embodiment of the present application provides a computer program product containing instructions. When it runs on a computer, the computer is made to execute the steps in the method provided in the above method embodiment.

[0144] Those skilled in the art can understand that Figure 8 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0145] In one embodiment, the screen state control method provided by the present application can be implemented in the form of a computer program, and the computer program can run on an electronic device as shown in Figure 8 The memory of the electronic device may store each program module that constitutes the above device. The computer program constituted by each program module causes the processor to execute the steps in the methods of the various embodiments of the present application described in this specification.

[0146] It should be pointed out here that: the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments and have beneficial effects similar to those of the method embodiments. For the technical details not disclosed in the storage medium, storage medium and device embodiments of the present application, please refer to the descriptions of the method embodiments of the present application for understanding.

[0147] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the "in one embodiment" or "in an embodiment" or "in some embodiments" that appear throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the magnitude of the serial numbers of the above processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments. The descriptions of the above embodiments tend to emphasize the differences between the embodiments, and their similarities or similarities can be referred to each other. For the sake of brevity, they will not be elaborated herein.

[0148] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, object A and / or object B can represent: object A exists alone, object A and object B exist simultaneously, and object B exists alone. These three situations.

[0149] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0150] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described embodiments are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of the devices or modules can be electrical, mechanical or other forms.

[0151] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network elements; some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional module in the embodiments of the present application may be fully integrated in a processing unit, or each module may be separately used as a unit, or two or more modules may be integrated in a unit; the above integrated modules may be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0153] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM), magnetic disks, or optical disks and other various media that can store program codes.

[0154] Alternatively, if the above integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence or the part that contributes to the related art, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing an electronic device to execute all or part of the methods described in the various embodiments of the present application. And the foregoing storage medium includes: removable storage devices, ROM, magnetic disks, or optical disks and other various media that can store program codes.

[0155] The methods disclosed in the several method embodiments provided in the present application can be arbitrarily combined without conflict to obtain new method embodiments.

[0156] The features disclosed in the several product embodiments provided in the present application can be arbitrarily combined without conflict to obtain new product embodiments.

[0157] The features disclosed in the several method or device embodiments provided in the present application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0158] As described above, it is only the implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the said claims.

Claims

1. A screen state control method, characterized in that: Applied to a terminal device, the terminal device includes a camera, and the method includes: When the screen of the terminal device is in a screen-off state, acquiring a first target image captured by the camera; In a case where it is detected that a target user exists in the first target image, detecting whether a target object exists in the first target image; When the target object exists and the state of the target user satisfies a preset screen-on condition, the screen of the terminal device is controlled to be in a screen-on state.

2. The method according to claim 1, characterized in that The target object is a pen, and when the target object exists and the state of the target user satisfies a preset screen-on condition, controlling the screen of the terminal device to be in a screen-on state includes: When it is detected that the hand of the target user is in the pen-holding state, the screen of the terminal device is controlled to be in the screen-on state.

3. The method according to claim 1, characterized in that The target object is a book, and when the target object exists and the state of the target user satisfies a preset screen-on condition, controlling the screen of the terminal device to be in a screen-on state includes: When it is detected that the sight line of the target user is directed toward the book, the screen of the terminal device is controlled to be in the bright screen state.

4. The method according to claim 1, characterized in that The target object is a book and a pen, and when the target object exists and the state of the target user satisfies a preset screen-on condition, controlling the screen of the terminal device to be in a screen-on state includes: When it is detected that the hand of the target user is not in the pen-holding state and the sight of the target user is directed toward the book, the screen of the terminal device is controlled to be in the screen-on state.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a second target image within a preset period, where the second target image is acquired when the terminal device is in the bright screen state; When it is detected that the target user does not exist in the second target image, the screen of the terminal device is controlled to be in the screen-off state.

6. The method according to claim 1, characterized in that The detecting whether a target object exists in the first target image includes: The first target image is input into a pre-trained target detection model to obtain a detection result of the target object, wherein the target detection model is obtained by training an initial model based on a sample image containing the target object.

7. The method according to claim 1, characterized in that When the target object exists and the state of the target user satisfies a preset screen-on condition, controlling the screen of the terminal device to be in a screen-on state includes: When the target object exists and it is detected that the sight line of the target user is directed toward the terminal device, the screen of the terminal device is controlled to be in a bright screen state.

8. A screen status control device, characterized in that: Applied to a terminal device, the terminal device includes a camera, and the device includes: A target image acquisition module, used for acquiring a first target image captured by the camera when the screen of the terminal device is in a screen-off state; a target object detection module, configured to detect whether a target object exists in the first target image when a target user is detected in the first target image; The screen status control module is used to control the screen of the terminal device to be in a screen-on state when the target object exists and the state of the target user meets a preset screen-on condition.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images