User status detection method, device, electronic device and storage medium

Abstract

The disclosed embodiment discloses a method, device, electronic device and storage medium for detecting user status, the method comprising: obtaining a video stream of a user learning; identifying the external features of the user based on the video stream to obtain the external features of the user; determining the current status of the user based on the external features, wherein the external features include the features of the user's face and key points of the skeleton. The technical solution of the disclosed embodiment achieves the purpose of accurately detecting the status of the user when learning.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a user status detection method, device, electronic device, and storage medium. Background Art

[0002] Usually primary school students have poor self-control ability, especially in learning. Many of them are more resistant and need parents to check on them one-on-one when they are doing homework or reviewing lessons.

[0003] Nowadays, the pace of life is getting faster and faster, and parents' time is very precious, so many parents are unable to check their children's homework or review their lessons on the spot. The absence of parents leads to many children being unable to study efficiently and develop good study habits. Summary of the invention

[0004] The embodiments of the present disclosure provide a user status detection method, device, electronic device and storage medium to accurately detect the user's learning status.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for detecting a user state, the method comprising:

[0006] Get the video stream of the user while learning;

[0007] Recognize the external features of the user based on the video stream to obtain the external features of the user;

[0008] Determine the current state of the user based on the external features;

[0009] The external features include facial features and skeleton key point features of the user.

[0010] In a second aspect, an embodiment of the present disclosure further provides a user status detection device, the device comprising:

[0011] The acquisition module is used to obtain the video stream when the user is learning;

[0012] A recognition module, used to recognize the external features of the user based on the video stream to obtain the external features of the user;

[0013] A determination module, used to determine the current state of the user based on the external features;

[0014] The external features include facial features and skeleton key point features of the user.

[0015] In a third aspect, an embodiment of the present disclosure further provides a device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the user status detection method as described in any one of the embodiments of the present disclosure.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute a user status detection method as described in any one of the embodiments of the present disclosure.

[0020] The technical solution of the disclosed embodiment is to obtain a video stream of the user while learning; identify the external features of the user based on the video stream, and determine the current state of the user based on the external features; wherein the external features include the facial features and features of the key points of the skeleton of the user, and the current state of the user is determined by jointly determining the facial features and features of the key points of the skeleton of the user, thereby achieving accurate identification of the user's state. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A schematic diagram of a user status detection method provided in the first embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a user status detection method provided in the second embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of the structure of a user status detection device provided in Embodiment 3 of the present disclosure;

[0025] Figure 4 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] Embodiment 1

[0032] Figure 1 A flow chart of a user status detection method provided in the first embodiment of the present disclosure. The user status detection method provided in this embodiment can be applied to the scenario of detecting the behavior or mental state of primary school students when they are studying or taking exams. Specifically, when it is detected that the primary school students are dozing off, sleeping on their stomachs, or looking around while studying or taking exams, reminders are issued to the primary school students in a timely manner to improve the learning efficiency of the primary school students, help them develop good study habits and correct study postures, or help maintain discipline in the examination room during exams. The method can be executed by a user status detection device, which can be implemented in the form of software and / or hardware, and is usually arranged in front of the study table or in a suitable position in the classroom, and the suitable position specifically refers to a position where images of the user studying or taking exams can be captured.

[0033] See Figure 1 As shown, the user status detection method provided in this embodiment includes the following steps:

[0034] Step 110: Obtain the video stream of the user while learning.

[0035] Among them, the users may include primary school students, candidates of various ages, etc.

[0036] Specifically, the video stream of the user's learning is obtained, including:

[0037] The video stream of the user studying is obtained through a camera specially set for the current application scenario. For example, the video stream of a primary school student studying is obtained through a camera set above the primary school student's study table; or the video stream of a candidate taking an exam is obtained through a camera set at a suitable position in the classroom or examination room, where the suitable position refers to a position where the user's complete image can be captured.

[0038] Step 120: Identify external features of the user based on the video stream to obtain external features of the user.

[0039] Among them, the external features include the features of the user's face and the features of the key points of the skeleton. The facial features include, for example, features such as frequent blinking, blurred eyes or closed eyes; the features of the key points of the skeleton are specifically the coordinate values ​​of the key points of the skeleton, or the features of the contour composed of the coordinate values ​​of each key point of the skeleton. The external features include: prone sleeping features, body tilt features or dozing features. The prone sleeping features and body tilt features can be distinguished by the position information of the key points of the skeleton. The key points of the skeleton specifically refer to points that can clearly represent the body posture, such as points that constitute the outline of the shoulders, points that constitute the outline of the head, etc. In the learning scene, the image of the user captured usually only includes the image of the upper body of the user. Therefore, the key points of the skeleton can be located from the shoulders and head of the user. When the user is prone sleeping, the entire shoulder and head have a relatively obvious characteristic contour. By comparing the features identified based on the captured image with the set contour, the external features of the current user can be determined. The set contour is specifically the body contour of the user when learning to maintain the correct posture or standard posture. Its essence is the coordinate values ​​of the points that constitute the body and the positional relationship between the points. Further, in addition to determining the current external features of the user by contour contrast, it can also be determined by the relative distance between a single skeleton key point and a reference point. For example, a point somewhere on the top of the head can be determined as a skeleton key point, or a point somewhere on the chin can be determined as a skeleton key point, and the distance between the skeleton key point and the study table is calculated to determine whether the current external features of the user are prone sleeping features. It is understandable that when the user is prone sleeping, the distance between the top of the head and the chin of the user and the study table is small, and when the user maintains a correct posture or a standard posture to study, the distance between the top of the head and the chin and the study table is large, so it can be determined based on the relative distance between the skeleton key point and the reference point whether the external features of the user are prone sleeping features. Similarly, it can also be determined based on the relative distance between the skeleton key point and the reference point whether the external features of the user are body crooked features.

[0040] It should be noted that the sleepiness characteristics include mild sleepiness and severe sleepiness, wherein the characteristics of mild sleepiness are: yawning frequency increases, blinking frequency decreases or eyes become dull, etc.; the characteristics of severe sleepiness are: the body is crooked or directly sleeping prone, so the sleepiness characteristics in some cases include the prone sleeping characteristics or crooked characteristics. At the same time, by combining the user's facial features (such as yawning, blinking, etc.) and body posture features (i.e., the features of the key points of the skeleton), the accuracy of fatigue recognition is improved.

[0041] Specifically, this example identifies the external features of the user according to the video stream based on deep learning technology to improve the recognition accuracy of the external features.

[0042] Exemplarily, identifying the external features of the user based on the video stream to obtain the external features of the user includes:

[0043] Inputting the video stream into a pre-trained neural network model to obtain external feature parameters of the user;

[0044] Determining the external characteristics of the user according to the external characteristic parameters;

[0045] Wherein, the external feature parameters include at least one of the eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of key points of bones; the external features include: prone sleeping feature, crooked feature or dozing feature. The pre-trained neural network model is obtained based on training samples, and the training samples include video streams marked with external feature parameters. The neural network model includes: CNN (Convolutional Neural Networks). Convolutional neural networks obtain local features of images by adopting local connections. Convolutional neural networks directly use two-dimensional images as inputs, automatically learn image features and implicit relationships within the data, without complex preprocessing of the image. The basic structure of convolutional neural networks includes convolutional layers, downsampling layers and fully connected layers. According to the characteristics of the local receptive field, the convolutional neural network uses local connections instead of full connections between two adjacent layers of neurons, greatly reduces the parameters required for training by weight sharing, and reduces the dimension of network parameters by using downsampling technology.

[0046] Furthermore, the external feature parameters include coordinate information of skeleton key points and yawning frequency, and determining the external features of the user according to the external feature parameters includes:

[0047] If the coordinate information of the skeleton key points meets the set outline and the yawning frequency exceeds the threshold, the user's external feature is determined to be a sleepy feature; by combining the user's facial features (such as yawning, blinking, etc.) and body posture features (i.e., the coordinate information of the skeleton key points), the recognition accuracy of the sleepy feature is improved. The user's body posture outline can be generated by connecting the coordinate information of the skeleton key points.

[0048] Step 130: Determine the current state of the user based on the external features.

[0049] Specifically, when the external feature of the user is a drowsy feature, the current state of the user is drowsy;

[0050] When the external feature of the user is a prone sleeping feature, the current state of the user is prone sleeping;

[0051] When the external feature of the user is a skew feature, the current state of the user is skew.

[0052] Furthermore, when the user is currently in a dozing state, the alarm device is triggered to sound an alarm to remind the user; the alarm can be used to wake up the user, allowing the user to recover from the dozing state to the awake state, thereby improving the user's learning efficiency.

[0053] Alternatively, when the user is currently tilted, a set voice is played to give a voice reminder to the user; for example, when the user's body is detected to be tilted, the voice message "Please maintain a standard sitting posture to protect your eyes" is played to help the user develop a good sitting posture habit. Alternatively, when the user is detected to be tired and inattentive, the voice message "Please cherish your time and study hard" is played to help the user restrain himself and improve learning efficiency.

[0054] Alternatively, when the absence or vacancy of the user is detected, a notification is sent to the associated communication terminal so that the guardian can be informed of the user's status. The associated communication terminal may be a mobile phone of the user's guardian, such as a parent or teacher's mobile phone, so that the guardian can monitor the user offline.

[0055] Furthermore, the method may also include:

[0056] A learning status report is generated based on the real-time external characteristics of primary school students within a set time period, so that the primary school students themselves, teachers or parents can understand the mental state of the primary school students when studying, and then formulate a more suitable learning plan for the primary school students, such as adjusting the study time of the primary school students to a time period with a better learning state.

[0057] The technical solution of the disclosed embodiment obtains the video stream of the user when learning, and recognizes the video stream based on deep learning technology to obtain the user's external features, which include facial features and features of bone key points. The current state of the user is determined by combining the bone key points and facial features, thereby improving the detection accuracy of the user's state and achieving the purpose of automatically detecting the user's mental state when learning.

[0058] Embodiment 2

[0059] Figure 2 A flow chart of a user state detection method provided in the second embodiment of the present disclosure. Based on the schemes of the above embodiments, the present embodiment further optimizes the user state detection method. Specifically, before the acquired video stream is input into the pre-trained neural network model, the video stream is pre-processed to obtain the eye area and mouth area of ​​the user in the video stream, and the video stream of the user's eye area and mouth area is further input into the pre-trained neural network model. By obtaining the eye area and mouth difference of the user in the video stream in advance, the recognition workload of the neural network model is reduced. At this time, the neural network model only needs to identify the specific eye state and mouth state based on the eye area image and the mouth area image, without having to preferentially identify the eye area image and the mouth area image from the overall image, and then identify the specific state, thereby improving the recognition speed and accuracy of the model, especially for users wearing glasses, the eye state of such users (such as blinking frequency, eye closure characteristics or eye opening characteristics) can be accurately identified. Figure 2 As shown, the method includes:

[0060] Step 210: Obtain the video stream of the user while learning.

[0061] Step 220: pre-process the video stream to obtain the eye area and mouth area of ​​the user in the video stream.

[0062] Specifically, the recognition of the eye area and mouth area of ​​the user in the video stream can be based on the learning method: first, a large number of labeled human eye and non-human eye samples are collected from the face image, and then a classifier that can distinguish between human eyes and non-human eyes is trained through different learning methods. In actual recognition, the pre-trained classifier is used to scan the input image area to complete the recognition of the human eye area. Currently, the commonly used learning-based methods include the AdaBoost method, the SVM (Support Vector Machine) method, and the PCA (Principal Component Analysis) method. The recognition of the mouth area is similar to the recognition of the eye area, and will not be repeated.

[0063] The eye and mouth regions of the user in the video stream can also be identified based on a template method, specifically: a unified template based on the shape of the eye is designed, and then the position of the eye in the region of interest is searched using template matching.

[0064] The eye and mouth areas of the user in the video stream can also be identified based on feature methods, specifically: by using prior knowledge, the eyes and mouth are located using features such as position, shape, edge information, color information, and symmetry in the face image.

[0065] Step 230: Input the video streams of the user's eye area and mouth area into a pre-trained neural network model to obtain the user's external feature parameters.

[0066] The external characteristic parameters include at least one of eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of key bone points.

[0067] Step 240: Determine the external characteristics of the user according to the external characteristic parameters.

[0068] Furthermore, in order to improve the recognition accuracy of external features, especially the recognition of drowsiness features, the user's body posture can be further combined. For example, if the body is tilted, or the head is lowered and shaking slowly, it can be determined that the user may be drowsy. At this time, if it is identified that the user's blinking frequency is low, or the yawning frequency is high, or the eyes are closed, it is determined that the user has the external features of drowsiness.

[0069] Step 250: Determine the current state of the user based on the external features.

[0070] The technical solution of the disclosed embodiment improves the recognition accuracy of facial behavior features by preprocessing the acquired video stream before inputting it into a pre-trained neural network model to obtain the eye area and mouth area of ​​the user in the video stream, and further inputting the video stream of the eye area and mouth area of ​​the user into the pre-trained neural network model, thereby accurately identifying the eye state of such users (such as blinking frequency, eye closure features or eye opening features) wearing glasses, and improves the recognition accuracy of drowsiness features by combining the coordinate information of skeletal key points, thereby achieving accurate detection of the user state.

[0071] Embodiment 3

[0072] Figure 3 A user status detection device provided in the third embodiment of the present disclosure includes: an acquisition module 310, an identification module 320 and a determination module 330;

[0073] Among them, the acquisition module 310 is used to obtain the video stream of the user when learning; the recognition module 320 is used to identify the external features of the user based on the video stream to obtain the external features of the user; the determination module 330 is used to determine the current state of the user based on the external features, wherein the external features include the user's facial features and features of bone key points.

[0074] On the basis of the above technical solution, the recognition module 320 includes an input unit for inputting the video stream into a pre-trained neural network model to obtain external feature parameters of the user; a determination unit for determining the external features of the user according to the external feature parameters;

[0075] Wherein, the external characteristic parameters include at least one of eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of key points of skeleton;

[0076] The external features include: a prone sleeping feature, a tilted feature, or a dozing feature.

[0077] On the basis of the above-mentioned technical solutions, the pre-trained neural network model is obtained by training based on training samples, the training samples include video streams marked with external feature parameters, and the neural network model includes: a convolutional neural network CNN.

[0078] Based on the above technical solutions, if the external feature parameters include coordinate information of skeletal key points and yawning frequency, the determination unit is specifically used to: if the coordinate information of the skeletal key points conforms to the set contour and the yawning frequency exceeds a threshold, then determine that the user's external feature is a sleepy feature.

[0079] On the basis of the above technical solutions, the recognition module 320 further includes a preprocessing unit, which is used to preprocess the video stream before inputting the video stream into the pre-trained neural network model to obtain the eye area and mouth area of ​​the user in the video stream;

[0080] Correspondingly, the input unit is specifically used to: input the video stream of the user's eye area and mouth area into a pre-trained neural network model.

[0081] Based on the above technical solutions, the device also includes a reminder module, which is used to trigger an alarm device to alarm when the user's current state is drowsy, or send a notification to an associated communication terminal to inform the guardian of the user's current state; when the user's current state is crooked, a set voice is played.

[0082] On the basis of the above technical solutions, the users include primary school students;

[0083] Correspondingly, the device also includes: a generating module, which is used to generate a learning status report according to the real-time external characteristics of the primary school students within a set time period.

[0084] The technical solution of the disclosed embodiment improves the recognition accuracy of facial behavior features by preprocessing the acquired video stream before inputting it into a pre-trained neural network model to obtain the eye area and mouth area of ​​the user in the video stream, and further inputting the video stream of the eye area and mouth area of ​​the user into the pre-trained neural network model, thereby accurately identifying the eye state of such users (such as blinking frequency, eye closure features or eye opening features) wearing glasses, and improves the recognition accuracy of external features by combining the coordinate information of skeletal key points, thereby achieving accurate detection of the user state.

[0085] The user status detection device provided in the embodiments of the present disclosure can execute the user status detection method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0086] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0087] Embodiment 4

[0088] Reference below Figure 4 , which shows an electronic device (eg, Figure 4 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 406 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0090] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 406 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 406, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0092] The terminal provided in the embodiment of the present disclosure and the user status detection method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.

[0093] Embodiment 5

[0094] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the user status detection method provided by the above embodiment is implemented.

[0095] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0096] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0097] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0098] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0099] Get the video stream of the user while learning;

[0100] Recognize the external features of the user based on the video stream to obtain the external features of the user;

[0101] Determine the current state of the user based on the external features;

[0102] The external features include facial features and skeleton key point features of the user.

[0103] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, an editable content display unit may also be described as an "editing unit".

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] Some example embodiments of the present disclosure are listed below.

[0109] In one aspect, a user status detection method is provided, the method comprising:

[0110] Get the video stream of the user while learning;

[0111] Recognize the external features of the user based on the video stream to obtain the external features of the user;

[0112] Determine the current state of the user based on the external features;

[0113] The external features include facial features and skeleton key point features of the user.

[0114] In some embodiments, identifying the external features of the user based on the video stream to obtain the external features of the user includes:

[0115] Inputting the video stream into a pre-trained neural network model to obtain external feature parameters of the user;

[0116] Determining the external characteristics of the user according to the external characteristic parameters;

[0117] Wherein, the external characteristic parameters include at least one of eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of key points of skeleton;

[0118] The external features include: a prone sleeping feature, a tilted feature, or a dozing feature.

[0119] In some embodiments, the pre-trained neural network model is trained based on training samples, the training samples include video streams marked with external feature parameters, and the neural network model includes: a convolutional neural network CNN.

[0120] In some embodiments, the external feature parameters include coordinate information of skeleton key points and yawning frequency, and determining the external features of the user according to the external feature parameters includes:

[0121] If the posture information meets the set profile and the yawning frequency exceeds the threshold, determining that the user's external feature is a dozing feature;

[0122] When the external feature of the user is a drowsy feature, the current state of the user is drowsy.

[0123] In some embodiments, before inputting the video stream into a pre-trained neural network model, the method further includes:

[0124] Preprocessing the video stream to obtain the eye area and mouth area of ​​the user in the video stream;

[0125] Inputting the video stream into a pre-trained neural network model includes: inputting the video streams of the user's eye area and mouth area into the pre-trained neural network model.

[0126] In some embodiments, the method further includes: when the user is currently in a dozing state, triggering an alarm device to sound an alarm, or sending a notification to an associated communication terminal so that a guardian can be informed of the user's current state;

[0127] When the user's current state is crooked, play the set voice.

[0128] In some embodiments, the user comprises an elementary school student;

[0129] The method further comprises:

[0130] Generate a learning status report based on the real-time external characteristics of primary school students within a set time period.

[0131] In one aspect, a user status detection device is provided, the device comprising:

[0132] The acquisition module is used to obtain the video stream when the user is learning;

[0133] A recognition module, used to recognize the external features of the user based on the video stream to obtain the external features of the user;

[0134] A determination module, used to determine the current state of the user based on the external features;

[0135] The external features include facial features and skeleton key point features of the user.

[0136] In some embodiments, the identification module includes:

[0137] An input unit, used to input the video stream into a pre-trained neural network model to obtain external feature parameters of the user;

[0138] A determination unit, configured to determine the external features of the user according to the external feature parameters;

[0139] Wherein, the external characteristic parameters include at least one of eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of key points of skeleton;

[0140] The external features include: a prone sleeping feature, a tilted feature, or a dozing feature.

[0141] In some embodiments, the pre-trained neural network model is trained based on training samples, the training samples include video streams marked with external feature parameters, and the neural network model includes: a convolutional neural network CNN.

[0142] In some embodiments, the external feature parameters include coordinate information of skeletal key points and yawning frequency, and the determination unit is specifically used to: if the coordinate information of the skeletal key points conforms to the set contour and the yawning frequency exceeds a threshold, determine that the user's external feature is a sleepy feature.

[0143] In some embodiments, the recognition module further includes a preprocessing unit, which is used to preprocess the video stream before inputting the video stream into the pre-trained neural network model to obtain the eye area and mouth area of ​​the user in the video stream;

[0144] The input unit is specifically used to input the video streams of the user's eye area and mouth area into a pre-trained neural network model.

[0145] In some embodiments, the device also includes a reminder module, which is used to trigger an alarm device to sound an alarm when the user's current state is drowsy, or send a notification to an associated communication terminal to inform the guardian of the user's current state; when the user's current state is crooked, play a set voice.

[0146] In some embodiments, the user comprises an elementary school student;

[0147] The device also includes: a generating module, which is used to generate a learning status report according to the real-time external features of the primary school students within a set time period.

[0148] In one aspect, an electronic device is provided, the electronic device comprising:

[0149] one or more processors;

[0150] a storage device for storing one or more programs,

[0151] When the one or more programs are executed by the one or more processors, the one or more processors implement the following user status detection method:

[0152] Get the video stream of the user while learning;

[0153] Recognize the external features of the user based on the video stream to obtain the external features of the user;

[0154] Determine the current state of the user based on the external features;

[0155] The external features include facial features and skeleton key point features of the user.

[0156] In some embodiments, when the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the following user status detection method:

[0157] Inputting the video stream into a pre-trained neural network model to obtain external feature parameters of the user;

[0158] Determining the external characteristics of the user according to the external characteristic parameters;

[0159] Determine the current state of the user based on the external features;

[0160] The external features include the user's facial features and features of skeleton key points; the external feature parameters include at least one of eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of skeleton key points; the external features include: prone sleeping feature, crooked feature or dozing feature;

[0161] The pre-trained neural network model is obtained by training based on training samples, wherein the training samples include video streams marked with external feature parameters, and the neural network model includes: a convolutional neural network (CNN).

[0162] In some embodiments, when the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the following user status detection method:

[0163] The external feature parameters include coordinate information of skeleton key points and yawning frequency, and if the coordinate information of the skeleton key points meets the set profile and the yawning frequency exceeds a threshold, it is determined that the external feature of the user is a sleepy feature;

[0164] When the external feature of the user is a drowsy feature, the current state of the user is drowsy.

[0165] In some embodiments, when the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the following user status detection method:

[0166] Preprocessing the video stream to obtain the eye area and mouth area of ​​the user in the video stream;

[0167] The video streams of the user's eye area and mouth area are input into the pre-trained neural network model.

[0168] In some embodiments, when the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the following user status detection method:

[0169] When the user is currently dozing off, the alarm device is triggered to sound an alarm, or a notification is sent to an associated communication terminal so that the guardian is informed of the user's current state;

[0170] When the user's current state is crooked, play the set voice.

[0171] In some embodiments, when the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the following user status detection method:

[0172] The users include primary school students, and a learning status report is generated according to the real-time external features of the primary school students within a set time period.

[0173] In one aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processing device, the user status detection method described below is implemented:

[0174] Get the video stream of the user while learning;

[0175] Recognize the external features of the user based on the video stream to obtain the external features of the user;

[0176] Determine the current state of the user based on the external features;

[0177] The external features include facial features and skeleton key point features of the user.

[0178] In some embodiments, when the program is executed by the processing device, the user status detection method described below is implemented:

[0179] Inputting the video stream into a pre-trained neural network model to obtain external feature parameters of the user;

[0180] Determining the external characteristics of the user according to the external characteristic parameters;

[0181] Determine the current state of the user based on the external features;

[0182] The external features include the user's facial features and features of skeleton key points; the external feature parameters include at least one of eye closing time, blinking frequency, yawning frequency, mouth opening time or coordinate information of skeleton key points; the external features include: prone sleeping feature, crooked feature or dozing feature;

[0183] The pre-trained neural network model is obtained by training based on training samples, wherein the training samples include video streams marked with external feature parameters, and the neural network model includes: a convolutional neural network (CNN).

[0184] In some embodiments, when the program is executed by the processing device, the user status detection method described below is implemented:

[0185] The external feature parameters include coordinate information of skeleton key points and yawning frequency, and if the coordinate information of the skeleton key points meets the set profile and the yawning frequency exceeds a threshold, it is determined that the external feature of the user is a sleepy feature;

[0186] When the external feature of the user is a drowsy feature, the current state of the user is drowsy.

[0187] In some embodiments, when the program is executed by the processing device, the user status detection method described below is implemented:

[0188] Preprocessing the video stream to obtain the eye area and mouth area of ​​the user in the video stream;

[0189] The video streams of the user's eye area and mouth area are input into the pre-trained neural network model.

[0190] In some embodiments, when the program is executed by the processing device, the user status detection method described below is implemented:

[0191] When the user is currently dozing off, the alarm device is triggered to sound an alarm, or a notification is sent to an associated communication terminal so that the guardian is informed of the user's current state;

[0192] When the user's current state is crooked, play the set voice.

[0193] In some embodiments, when the program is executed by the processing device, the user status detection method described below is implemented:

[0194] The users include primary school students, and a learning status report is generated according to the real-time external features of the primary school students within a set time period.

[0195] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0196] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0197] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A user status detection method, characterized in that: include: Get the video stream of the user while learning; Recognize the external features of the user based on the video stream to obtain the external features of the user; Determine the current state of the user based on the external features; The external features include the user's facial features and features of key skeleton points; the facial features include frequent blinking, blurred eyes or closed eyes; The identifying the external features of the user based on the video stream to obtain the external features of the user includes: Inputting the video stream into a pre-trained neural network model to obtain external feature parameters of the user; the pre-trained neural network model is trained based on training samples, and the training samples include video streams marked with external feature parameters; Determining the external characteristics of the user according to the external characteristic parameters; The external characteristic parameters include eye closing time, blinking frequency, yawning frequency, mouth opening time and coordinate information of key bone points; The external features also include: prone sleeping feature, crooked feature or dozing feature; The prone sleeping feature is determined based on the position information of the skeleton key points of the user; or, based on the relative distance between the skeleton key points and the reference point; The skew feature is determined based on the position information of the skeleton key point of the user; or, is determined based on the relative distance between the skeleton key point and the reference point; The sleepiness feature is determined based on the coordinate information of the skeleton key points and the yawning frequency.

2. The method according to claim 1, characterized in that The neural network model includes: a convolutional neural network CNN.

3. The method according to claim 1, characterized in that The external feature parameters include coordinate information of skeleton key points and yawning frequency, and determining the external features of the user according to the external feature parameters includes: If the coordinate information of the skeleton key points conforms to the set outline and the yawning frequency exceeds the threshold, it is determined that the external feature of the user is a dozing feature; When the external feature of the user is a drowsy feature, the current state of the user is drowsy.

4. The method according to claim 1, characterized in that Before inputting the video stream into the pre-trained neural network model, the method further includes: Preprocessing the video stream to obtain the eye area and mouth area of ​​the user in the video stream; Inputting the video stream into a pre-trained neural network model includes: inputting the video streams of the user's eye area and mouth area into the pre-trained neural network model.

5. The method according to any one of claims 1 to 4, characterized in that it also includes: When the user is currently dozing off, the alarm device is triggered to sound an alarm, or a notification is sent to an associated communication terminal so that the guardian is informed of the user's current state; When the user's current state is crooked, play the set voice.

6. The method according to any one of claims 1 to 4, characterized in that: The users include primary school students; The method further comprises: Generate a learning status report based on the real-time external characteristics of primary school students within a set time period.

7. A user status detection device, characterized in that: include: The acquisition module is used to obtain the video stream of the user when learning; A recognition module, used to recognize the external features of the user based on the video stream to obtain the external features of the user; The identification module comprises: An input unit, used to input the video stream into a pre-trained neural network model to obtain external feature parameters of the user; the pre-trained neural network model is trained based on training samples, and the training samples include video streams marked with external feature parameters; A determination unit, configured to determine the external features of the user according to the external feature parameters; The external characteristic parameters include eye closing time, blinking frequency, yawning frequency, mouth opening time and coordinate information of key bone points; A determination module, used to determine the current state of the user based on the external features; The external features include the user's facial features and features of key skeleton points; the facial features include frequent blinking, blurred eyes or closed eyes; The external features also include: prone sleeping feature, crooked feature or dozing feature; The prone sleeping feature is determined based on the position information of the skeleton key points of the user; or, based on the relative distance between the skeleton key points and the reference point; The skew feature is determined based on the position information of the skeleton key point of the user; or, is determined based on the relative distance between the skeleton key point and the reference point; The sleepiness feature is determined based on the coordinate information of the skeleton key points and the yawning frequency.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the user status detection method as described in any one of claims 1 to 6.

9. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute the user status detection method according to any one of claims 1 to 6 when executed by a computer processor.

Images