User online status identification method, device, server and storage medium

Abstract

The embodiment of the present application is applicable to the field of computer vision technology, and provides a method, device, server and storage medium for identifying the online status of a user, the method comprising: receiving a video stream formed by a camera device at a first user end continuously capturing images of a first user, the video stream being a video stream formed by the camera device continuously capturing images of the first user; extracting multiple image frames from the video stream; for any of the image frames, extracting an image frame containing a binocular image of the first user from the image frame, and classifying the online status of the first user according to the image frame to obtain an online status classification result corresponding to the image frame; determining the current online status of the first user according to multiple online status classification results corresponding to the multiple image frames. The above method can accurately identify the online status of a user and ensure the implementation effect of online activities.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of computer vision technology, and in particular, relate to a method, device, server, and storage medium for identifying a user's online status. Background Art

[0002] The development of information technology has enabled many things in work, life, and study to be done online, such as online education, online meetings, and so on.

[0003] Taking online education as an example, the most common form of online education is that the teacher teaches the course to the students on the other end of the network through the Internet. This form of online education is also called "online class" or "online teaching". Compared with the face-to-face teacher and student in the offline classroom, the teacher can quickly observe the facial expression of each student to judge whether the student is listening carefully. When teaching online, due to the large differences in the video windows of each student, such as different light brightness, different camera resolutions, etc., the online teacher cannot accurately judge the listening status of each student in a short time.

[0004] In the prior art, some online education software can extract students' facial information during class and judge whether they are listening attentively by identifying the student's current facial expression. However, due to individual differences, a single facial expression cannot accurately judge whether a student is listening attentively, and the result has a large error, which seriously affects the effect of online teaching. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, device, server and storage medium for identifying the online status of a user, which can accurately identify the online status of a user and ensure the implementation effect of online activities.

[0006] A first aspect of an embodiment of the present application provides a method for identifying a user's online status, which is applied to a server. The method includes:

[0007] Receiving a video stream formed by a camera device at a first user end continuously capturing images of the first user;

[0008] Extracting a plurality of image frames from the video stream;

[0009] For any of the image frames, extracting an image frame containing the binocular images of the first user from the image frame, and classifying the online status of the first user according to the image frame to obtain an online status classification result corresponding to the image frame;

[0010] The current online status of the first user is determined according to a plurality of online status classification results corresponding to a plurality of image frames.

[0011] A second aspect of an embodiment of the present application provides a device for identifying a user's online status, which is applied to a server, and the device includes:

[0012] A video stream receiving module, used to receive a video stream formed by a camera device at a first user end continuously capturing images of the first user;

[0013] An image frame extraction module, used for extracting multiple image frames from the video stream;

[0014] An image frame extraction module, configured to extract, for any of the image frames, an image frame containing the binocular image of the first user from the image frame;

[0015] a classification result recognition module, configured to classify the online status of the first user according to the image frame, and obtain an online status classification result corresponding to the image frame;

[0016] The online status determination module is used to determine the current online status of the first user according to multiple online status classification results corresponding to multiple image frames.

[0017] A third aspect of an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying the online status of a user as described in the first aspect above is implemented.

[0018] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying the online status of a user as described in the first aspect above is implemented.

[0019] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method for identifying the online status of a user described in the first aspect.

[0020] Compared with the prior art, the embodiments of the present application have the following advantages:

[0021] In the embodiment of the present application, after receiving the video stream formed by the camera device of the first user end continuously capturing images of the first user, by extracting multiple frames of image frames from the video stream, an image frame containing the binocular image of the first user can be extracted from any image frame, thereby classifying the online status of the first user according to the image frame, and obtaining the online status classification result corresponding to the image frame. Then, according to the multiple online status classification results corresponding to the multiple frames of image frames, the current online status of the first user can be accurately determined. The embodiment of the present application identifies the online status of the first user based on the rich status information provided by the eyes, which has a higher accuracy rate than the judgment method using facial expressions in the prior art. Secondly, the embodiment of the present application uses multiple frames of image frames extracted within a certain period of time to comprehensively identify the status, and the result is more convincing and representative than the judgment using a single image frame; the final result of the comprehensive output of multiple frames of image frames also greatly reduces the probability of random errors. Third, in view of the differences between the camera devices used by the first user, directly processing the image frames may affect the accuracy of the recognition results due to blurring and other factors. The embodiment of the present application can restore the true online status of the first student to the greatest extent by using super-resolution reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a schematic diagram of a method for identifying a user's online status provided in an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of a system architecture to which a method for identifying a user online status provided in an embodiment of the present application is applicable;

[0025] Figure 3 It is a schematic diagram of an implementation of S102 in a method for identifying a user online status provided in an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of an implementation of S103 in a method for identifying a user online status provided in an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of an implementation of S105 in a method for identifying a user online status provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of a method for processing a target state in a method for identifying a user online state provided in an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a device for identifying a user's online status provided in an embodiment of the present application;

[0030] Figure 8 It is a schematic diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0032] The technical solution of the present application is described below through specific embodiments.

[0033] Reference Figure 1 , shows a schematic diagram of a method for identifying a user online status provided by an embodiment of the present application, and the method may specifically include the following steps:

[0034] S101: Receive a video stream formed by a camera device at a first user terminal continuously capturing images of a first user.

[0035] This method can be applied to a server, that is, the execution subject of the embodiment of the present application can be a server. Figure 2 FIG. 1 is a schematic diagram of a system architecture to which a method for identifying a user online status provided in an embodiment of the present application is applicable. Figure 2 The system architecture shown may include a first user terminal 201, a second user terminal 202, and a server 203. The first user terminal 201 may include multiple terminal devices, such as Figure 2 The terminal device 201a, the terminal device 201b, ..., the terminal device 201n, etc. are shown in FIG. The second user terminal 202 may generally include only one terminal device, namely Figure 2 Of course, in actual applications, the second user terminal 202 may also include multiple terminal devices in some cases, which is not limited in the embodiment of the present application.

[0036] based on Figure 2The system architecture shown in the figure can be applied to different scenarios. For example, online education scenarios, online conference scenarios, etc. In the online education scenario, the first user terminal 201 can be a student terminal attending a class. Figure 2 The terminal devices 201a, 201b, ..., 201n shown in the figure may be terminal devices used by a plurality of students for listening to a class, and the second user terminal 202 may be a teacher's terminal for giving a class. Figure 2 The terminal device 202a shown in FIG. 2 may be a terminal device used by a teacher when teaching. In an online conference scenario, the first user terminal 201 may be a user terminal participating in a conference. Figure 2 The terminal devices 201a, 201b, ..., 201n shown in the figure may be terminal devices used by multiple users when attending a meeting, and the second user terminal 202 may be a user terminal that hosts the meeting. Figure 2 The terminal device 202a shown in the figure may be a terminal device used by a conference host, and this embodiment of the present application does not limit this.

[0037] For ease of understanding, the embodiment of the present application takes the example in which the second user end 202 includes only a single terminal device 202a and the application scenario is an online education scenario for subsequent explanation.

[0038] In the online education scenario, the first user terminal is the student terminal, and the second user terminal is the teacher terminal. Therefore, the first user in the embodiment of the present application is a student, and the second user is a teacher. When the teacher of the second user teaches the student of the first user, the terminal devices used by both parties need to have a camera device. In this way, the student can see the whole process of the teacher's teaching from the terminal device of the first user terminal, and the teacher can also see the status of each student when listening to the class from the terminal device of the second user terminal.

[0039] In an embodiment of the present application, the video stream of the first user terminal and the video stream of the second user terminal can interact through the server. Among them, the server can distribute the video stream of the second user terminal to each terminal device of the first user terminal, and the video image seen by each terminal device of the first user terminal can be the same, that is, the video image seen by each student can be the same video image of the teacher teaching. When the server sends the video stream of the first user terminal to the terminal device of the second user terminal, since the terminal device of the first user terminal includes multiple terminals, the server can process the video streams collected by the multiple terminal devices of the first user terminal into one video stream, and then display it on the terminal device of the second user terminal. The method for identifying the user online status provided in the embodiment of the present application is a method for the server to process multiple video streams of the first user terminal.

[0040] In the embodiment of the present application, the camera device at the first user end can continuously capture the video of the first user to form a video stream, and the terminal device at the first user end can transmit the video stream to the server for processing. For example, during the course of listening to the class, the student can turn on the camera or other camera device of the terminal device to shoot the video, and the video stream captured by the camera device will be transmitted to the server in real time.

[0041] S102: extract multiple image frames from the video stream.

[0042] Usually, a video stream is formed by continuously acquired image frames. In the embodiment of the present application, in order to identify the online status of the first user, the video stream acquired by the first user terminal can be processed, some image frames can be extracted therefrom, and the online status of the first user in these image frames can be identified. In an online education scenario, the online status of the first user can be the status of a student listening to a class; in an online conference scenario, the online status of the first user can be the status of a participant in the conference.

[0043] In order to ensure the accuracy of online status recognition, the image frame extracted from the video stream should not be just a single frame, but multiple frames of image frames can be extracted; and each image frame should have a certain time interval.

[0044] In a possible implementation of the embodiment of the present application, as Figure 3 As shown, extracting multiple image frames from the video stream in S102 may specifically include the following sub-steps S1021-S1022:

[0045] S1021. Obtain a video stream to be extracted with a preset duration from the video stream.

[0046] In an embodiment of the present application, the preset duration may be a duration that can accurately evaluate the online status of the first user. Generally, the duration should not be too short. The online status evaluated based on a too short duration may not be accurate. Taking the online education scenario as an example, according to the normal habits of students in class, students will not be in a state for a short time. For example, after a student enters a distracted state, this state will basically last for more than 1 second. On the other hand, the duration should not be too long. Over a long period of time, the online status of the first user may have changed.

[0047] Therefore, in the embodiment of the present application, the preset duration may be 5 seconds, and the server may process the video stream every 5 seconds.

[0048] S1022: Continuously extract a plurality of image frames from the video stream to be extracted at a preset time interval.

[0049] In the embodiment of the present application, the preset time interval may be an interval of 200 ms (milliseconds), that is, one image frame is extracted every 200 ms. In this way, within 5 seconds, the server can extract 25 image frames. Based on these 25 image frames, the current online status of the first user can be basically and accurately evaluated.

[0050] Of course, the above example extracts image frames at equal time intervals. In some embodiments, the server may also extract image frames from the video stream at unequal time intervals. For example, the image frames may be extracted at 150ms time intervals in the first 2 seconds and at 300ms time intervals in the last 3 seconds, which is not limited in the present embodiment.

[0051] Usually, due to the large number of first users, the terminal devices used by the multiple first users are different, and the environment in which each first user is located will also affect the image quality of the collected video stream. For example, the terminal devices used by some students can collect relatively clear video streams, while the terminal devices used by some students collect relatively blurry video streams.

[0052] In order to solve the impact of video stream quality differences on online status recognition, after extracting multiple image frames from the video stream, the embodiment of the present application can also perform super-resolution reconstruction on the multiple image frames respectively to obtain reconstructed image frames.

[0053] Super-resolution reconstruction refers to reconstructing a corresponding high-resolution image from an observed low-resolution image. Super-resolution reconstruction can improve the resolution of the original image through hardware or software methods. In this way, the clarity of the reconstructed image frame can be greater than the clarity of the image frame before reconstruction, which helps the server to accurately recognize the image in the image frame.

[0054] S103 . For any of the image frames, extract an image frame containing the binocular images of the first user from the image frame.

[0055] Generally, eyes have rich status information. When identifying the online status of the first user, by analyzing the eyes of the first user, the accuracy is higher than that of analyzing based on the facial status of the first user in the prior art, and the result can be more accurate.

[0056] Therefore, in the embodiment of the present application, each image frame can be processed separately, and an image frame containing the binocular image of the first user can be extracted from each image frame. Specifically, the server can identify the binocular image of the first user in each frame by image recognition, and then use the image frame to frame the binocular image.

[0057] In a possible implementation of the embodiment of the present application, as Figure 4As shown, for any image frame, extracting an image frame containing the binocular image of the first user from the image frame in S103 may specifically include the following sub-steps S1031-S1033:

[0058] S1031. For any reconstructed image frame, extract features of the image frame to obtain a feature map of the image frame.

[0059] In the embodiment of the present application, the server may extract features from the image frame after super-resolution reconstruction of the image frame, that is, the server may extract features from the image frame after super-resolution reconstruction to obtain a corresponding feature map. In this way, compared with directly extracting features from the original image frame, the problem of inaccurate or incomplete feature extraction due to image blur can be avoided.

[0060] In a specific implementation, the server may use a detection module to implement the above feature extraction and subsequent cropping of the binocular image frame.

[0061] In an example of an embodiment of the present application, the server can use the Yolo detection model to extract features in the image frame. The Yolo detection model is an end-to-end target detection module that runs fast and can meet the real-time requirements of various application scenarios in the embodiment of the present application.

[0062] The Resnet50 network structure of the Yolo detection model can be used to extract features from input images. Therefore, the Yolo detection model can be deployed in the server. After completing the super-resolution reconstruction of the extracted image frame, the reconstructed image frame is transmitted to the Yolo detection model, and the Resnet50 network structure in the model extracts features from the image frame to obtain the feature map of the image frame.

[0063] S1032: Perform binocular target frame prediction in the feature map to obtain binocular coordinates of the first user.

[0064] In the embodiment of the present application, for the extracted feature map, the Yolo detection model can perform binocular target frame prediction in the feature map. In this way, binocular coordinates in the image frame can be obtained.

[0065] S1033: Crop an image frame including the binocular images of the first user from the image frame according to the binocular coordinates.

[0066] The Yolo detection model can perform a cutout process in the image frame based on the predicted binocular coordinates, thereby cropping an image frame containing the binocular images of the first user. Compared with directly processing the image frame, the embodiment of the present application can perform subsequent processing only on the binocular images of the first user by cropping the binocular image frame, thereby reducing the impact of other area images on recognition.

[0067] S104: Classify the online status of the first user according to the image frame to obtain an online status classification result corresponding to the image frame.

[0068] In an embodiment of the present application, the server can process the binocular image frame cropped in the previous step based on the rich status information contained in the eyes, and identify the current status of the first user when the image frame is captured, that is, the online status classification result.

[0069] In a possible implementation of the embodiment of the present application, classifying the online status of the first user according to the image frame can be implemented by a multi-classification model. The multi-classification model can be obtained by performing supervised classification training on multiple sample image frames, and the multiple sample image frames are annotated with corresponding online classification results.

[0070] For example, multiple sample image frames may be collected in advance, each of which may include a person's binocular image, and based on the binocular images in the sample image frames, they may be annotated with different online classification results. For example, different states of students when listening to a class, such as distracted state, focused state, sleepy state, etc. Then, the server may use a deep learning method to train multiple sample image frames annotated with online classification results, thereby obtaining the above-mentioned multi-classification model.

[0071] In this way, when classifying the online status of the first user according to the image frame, the server can first convert the image frame cropped in the previous step into a one-dimensional vector in matrix form; then input the one-dimensional vector into a pre-trained multi-classification model, and the multi-classification model outputs the online status classification result corresponding to the image frame.

[0072] In a possible implementation of the embodiment of the present application, the multi-classification model can be a gradient boosting decision tree (GBDT) model composed of multiple decision trees. GBDT is an iterative decision tree algorithm that consists of multiple decision trees, and the conclusions of all trees are accumulated as the final answer.

[0073] Therefore, when the online status classification result corresponding to the image frame is output based on the above-mentioned gradient boosting decision tree model, the one-dimensional vector can be input into the gradient boosting decision tree model to obtain the voting result of each decision tree in the gradient boosting decision tree model for the online status classification result corresponding to the image frame. The process of each decision tree voting for the image frame is the same as the process of each decision tree performing classification training on multiple sample image frames.

[0074] In this way, each decision tree can obtain a voting result, which represents the online state classification result of the image frame predicted by the decision tree. The voting result with the highest vote rate can be used as the online state classification result corresponding to the image frame. For example, if the gradient boosting decision tree model is composed of 500 decision trees, when voting on a certain image frame, 100 decision trees vote that the state of the first user displayed in the image frame is sleepy, and the other 400 decision trees vote that the state of the first user displayed in the image frame is distracted. Since the distracted state has a higher vote rate, the gradient boosting decision tree model can output the distracted state as the online state classification result corresponding to the image frame.

[0075] It should be noted that the image frames extracted by the server from the video stream include multiple frames, so each image frame may correspond to an online status classification result.

[0076] S105: Determine the current online status of the first user according to multiple online status classification results corresponding to the multiple image frames.

[0077] In the embodiment of the present application, the online status classification result corresponding to each image frame only indicates the online status of the first user at the time when the image frame is collected, and cannot accurately reflect the online status of the first user over a period of time. Therefore, the online status of the first user over a period of time, that is, the current online status, can be determined based on multiple online status classification results corresponding to multiple image frames.

[0078] In a possible implementation of the embodiment of the present application, as Figure 5 As shown, in S105, determining the current online status of the first user according to the multiple online status classification results corresponding to the multiple image frames may specifically include the following sub-steps S1051-S1052:

[0079] S1051: Count the number of online status classification results of each type.

[0080] S1052: Determine the online status classification result corresponding to the maximum number as the current online status of the first user.

[0081] In a specific implementation, the current online status of the first user may be determined according to the number of different online status classification results. Therefore, the number of online status classification results of each type may be counted first.

[0082] For example, for the captured video stream, if 25 image frames are extracted from 5 seconds, if there are 25 online status classification results after processing, each online status classification result can have a corresponding type, such as distracted state, focused state, sleepy state, etc., the number of online status classification results of each type can be counted separately, that is, the number of image frames in the distracted state, focused state and sleepy state in the 25 image frames can be counted.

[0083] Then, the online status classification result corresponding to the maximum number may be determined as the current online status of the first user.

[0084] For instance, in the above example, if the number of image frames in the distracted state is 13 frames, the number of image frames in the focused state is 5 frames, and the number of image frames in the sleepy state is 7 frames among the 25 image frames, since the number of image frames in the distracted state is the largest, it can be determined that the current online state of the first user is the distracted state.

[0085] In a possible implementation of the embodiment of the present application, as Figure 6 As shown, after determining the current online status of the first user according to the multiple online status classification results corresponding to the multiple image frames, the method may further include the following steps S106-S107:

[0086] S106: If the current online status of the first user is a target status, push information of the target status of the first user to a terminal device of the second user.

[0087] In an embodiment of the present application, the target state may be an online state that does not match the current online scenario. For example, in an online education scenario, the first user as a student should maintain a state of listening carefully to the class, and his online state should remain focused, and other states that do not match it can be considered as target states. For example, distracted state, sleepy state, etc. If the current online state of the first user is the above target state, it is necessary to inform the teacher as the second user, and the teacher will remind the student to listen carefully to the class.

[0088] Therefore, if the current online status of the first user is the target status, the server can push the information of the target status of the first user to the terminal device of the second user, for example, pushing the online status of a student to the terminal device used by a teacher.

[0089] S107: Highlight the target status of the first user in the display interface of the terminal device.

[0090] Typically, the display interface of the terminal device used by the second user may display online images of each first user.

[0091] For example, the terminal device used by the teacher displays a picture of all or some students attending the class. However, due to the large number of first users, when the first user has an online status that does not match the current online scene, the teacher cannot know the status change of the first user in time.

[0092] To solve this problem, after pushing the information of the first user's target status to the terminal device of the second user, the server can remind the second user by highlighting the target status of the first user in the terminal device of the second user.

[0093] For example, if a student is distracted during class, after the server identifies the student's distracted state and pushes the information of the distracted state to the terminal device used by the teacher, the server can inform the teacher that the student is not listening carefully in class by enlarging the image of the student in the terminal device used by the teacher or other similar methods. In order to ensure the quality of learning, the teacher can remind the student to listen carefully in class based on the above information.

[0094] In an embodiment of the present application, after receiving a video stream transmitted by a camera device at the first user end, by extracting multiple image frames from the video stream, an image frame containing the binocular image of the first user can be extracted from any image frame, thereby classifying the online status of the first user according to the image frame, and obtaining an online status classification result corresponding to the image frame. Then, according to the multiple online status classification results corresponding to the multiple image frames, the current online status of the first user can be accurately determined. The embodiment of the present application identifies the online status of the first user based on the rich status information provided by the eyes, which has a higher accuracy rate than the judgment using facial expressions in the prior art. Secondly, the embodiment of the present application uses multiple image frames extracted within a certain period of time to comprehensively identify the status, which is more convincing and representative than the judgment using a single image frame; the final result of the comprehensive output of multiple image frames also greatly reduces the probability of random errors. Thirdly, in view of the differences between the camera devices used by the first user, directly processing the image frame may affect the accuracy of the recognition result due to blurring and other situations. The embodiment of the present application uses super-resolution reconstruction to restore the real online status of the first student to the greatest extent.

[0095] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0096] Reference Figure 7 , shows a schematic diagram of a user online status recognition device provided by an embodiment of the present application, the device may specifically include a video stream receiving module 701, an image frame extraction module 702, an image frame extraction module 703, a classification result recognition module 704 and an online status determination module 705, wherein:

[0097] The video stream receiving module 701 is used to receive a video stream formed by a camera device at a first user terminal continuously capturing images of the first user;

[0098] An image frame extraction module 702, configured to extract multiple image frames from the video stream;

[0099] An image frame extraction module 703 is used to extract an image frame containing the binocular image of the first user from any of the image frames;

[0100] A classification result identification module 704, configured to classify the online status of the first user according to the image frame to obtain an online status classification result corresponding to the image frame;

[0101] The online status determination module 705 is used to determine the current online status of the first user according to a plurality of online status classification results corresponding to a plurality of image frames.

[0102] In the embodiment of the present application, the image frame extraction module 702 can be specifically used to: obtain a video stream to be extracted with a preset duration from the video stream; and continuously extract multiple frames of the image frames from the video stream to be extracted at preset time intervals.

[0103] In an embodiment of the present application, the device may further include a super-resolution reconstruction module, which may be specifically used to: perform super-resolution reconstruction on multiple frames of the image frames respectively to obtain the reconstructed image frames; wherein the clarity of the reconstructed image frames is greater than the clarity of the image frames before reconstruction.

[0104] In an embodiment of the present application, the image frame extraction module 703 can be specifically used to: for any reconstructed image frame, perform feature extraction on the image frame to obtain a feature map of the image frame; perform binocular target frame prediction in the feature map to obtain binocular coordinates of the first user; and crop an image frame containing a binocular image of the first user from the image frame according to the binocular coordinates.

[0105] In an embodiment of the present application, the classification result recognition module 704 can be specifically used to: convert the image frame into a one-dimensional vector in matrix form; input the one-dimensional vector into a pre-trained multi-classification model to obtain an online state classification result corresponding to the image frame output by the multi-classification model; wherein the multi-classification model can be obtained by performing supervised classification training on multiple frames of sample image frames, and the multiple frames of the sample image frames are annotated with the corresponding online classification results.

[0106] In the embodiment of the present application, the online status determination module 705 can be specifically used to: count the number of the online status classification results of each type; and determine the online status classification result corresponding to the maximum number as the current online status of the first user.

[0107] In the embodiment of the present application, the device may further include a target status push module and a target status display module, wherein:

[0108] The target status push module may be used to: if the current online status of the first user is the target status, push the target status information of the first user to the terminal device of the second user;

[0109] The target state display module may be used to: highlight the target state of the first user in the display interface of the terminal device; wherein the target state is an online state that does not match the current online scenario.

[0110] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0111] Reference Figure 8 , shows a schematic diagram of a server provided in an embodiment of the present application. Figure 8 As shown, the server 800 in the embodiment of the present application includes: a processor 810, a memory 820, and a computer program 821 stored in the memory 820 and executable on the processor 810. When the processor 810 executes the computer program 821, the steps in each embodiment of the above-mentioned method for identifying the online status of a user are implemented, for example Figure 1 Alternatively, when the processor 810 executes the computer program 821, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 7 Functions of modules 701 to 705 are shown.

[0112] Exemplarily, the computer program 821 may be divided into one or more modules / units, which are stored in the memory 820 and executed by the processor 810 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 821 in the server 800. For example, the computer program 821 may be divided into a video stream receiving module, an image frame extraction module, an image frame extraction module, a classification result recognition module, and an online status determination module, and the specific functions of each module are as follows:

[0113] A video stream receiving module, used to receive a video stream formed by a camera device at a first user end continuously capturing images of the first user;

[0114] An image frame extraction module, used for extracting multiple image frames from the video stream;

[0115] An image frame extraction module, configured to extract, for any of the image frames, an image frame containing the binocular image of the first user from the image frame;

[0116] a classification result recognition module, configured to classify the online status of the first user according to the image frame, and obtain an online status classification result corresponding to the image frame;

[0117] The online status determination module is used to determine the current online status of the first user according to multiple online status classification results corresponding to multiple image frames.

[0118] The server 800 may be a server in the aforementioned embodiments, and the server 800 may be a computing device such as a cloud server. The server 800 may include, but is not limited to, a processor 810 and a memory 820. Those skilled in the art will appreciate that Figure 8 It is only an example of the server 800 and does not constitute a limitation of the server 800. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the server 800 may also include input and output devices, network access devices, buses, etc.

[0119] The processor 810 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0120] The memory 820 may be an internal storage unit of the server 800, such as a hard disk or memory of the server 800. The memory 820 may also be an external storage device of the server 800, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 800. Further, the memory 820 may also include both an internal storage unit of the server 800 and an external storage device. The memory 820 is used to store the computer program 821 and other programs and data required by the server 800. The memory 820 may also be used to temporarily store data that has been output or is to be output.

[0121] An embodiment of the present application also discloses a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for identifying the online status of a user as described in the above embodiments is implemented.

[0122] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying the online status of a user as described in the above-mentioned embodiments is implemented.

[0123] The embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method for identifying the online status of a user described in the above-mentioned embodiments.

[0124] If the functions implemented by the server in the embodiment of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / server, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices / servers and methods can be implemented in other ways. For example, the device / server embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for identifying a user's online status, characterized in that: Applied to a server, the method comprises: Receiving a video stream formed by a camera device at a first user end continuously capturing images of the first user; Extracting a plurality of image frames from the video stream, wherein the video stream is a video stream within a preset time length, and time intervals for extracting the plurality of image frames are equal; For any of the image frames, an image frame including the binocular image of the first user is extracted from the image frame, and based on the image frame, a multi-classification model is used to classify the online status of the first user to obtain an online status classification result corresponding to the image frame, wherein the multi-classification model is obtained by performing supervised classification training on multiple sample image frames, each of which includes a binocular image of a person and is respectively annotated with different states of the student when listening to the class based on the binocular images in the sample image frames; Determine the current online state of the first user according to multiple online state classification results corresponding to the multiple image frames, wherein the online state is the state of the first user when attending a class, and the online state is determined by the voting results of multiple decision trees constituting the multi-classification model, and the voting result of each decision tree represents the online state classification result of the image frame predicted by the decision tree; The multi-classification model is a gradient boosting decision tree model composed of multiple decision trees, and the online status of the first user is classified using the multi-classification model according to the image frame to obtain an online status classification result corresponding to the image frame, including: Convert the image frame into a one-dimensional vector in matrix form; Inputting the one-dimensional vector into the gradient boosting decision tree model, obtaining the voting result of each decision tree in the gradient boosting decision tree model for the online status classification result corresponding to the image frame; wherein the process of each decision tree voting on the image frame is the same as the process of each decision tree performing classification training on multiple frames of the sample image frames; The voting result with the highest vote rate is used as the online status classification result corresponding to the image frame.

2. The method according to claim 1, characterized in that After extracting a plurality of image frames from the video stream, the method further comprises: Super-resolution reconstruction is performed on multiple image frames respectively to obtain reconstructed image frames; wherein the clarity of the reconstructed image frames is greater than the clarity of the image frames before reconstruction.

3. The method according to claim 2, characterized in that The step of extracting an image frame including the binocular image of the first user from any of the image frames includes: For any reconstructed image frame, extract features from the image frame to obtain a feature map of the image frame; Perform binocular target frame prediction in the feature map to obtain binocular coordinates of the first user; An image frame including the binocular images of the first user is cropped from the image frame according to the binocular coordinates.

4. The method according to claim 1, characterized in that: The determining the current online status of the first user according to the multiple online status classification results corresponding to the multiple image frames includes: Counting the number of each online status classification result; The online status classification result corresponding to the maximum number is determined as the current online status of the first user.

5. The method according to any one of claims 1 to 4, characterized in that: After determining the current online status of the first user according to the multiple online status classification results corresponding to the multiple image frames, the method further includes: If the current online status of the first user is the target status, pushing the target status information of the first user to the terminal device of the second user; The target state of the first user is highlighted in the display interface of the terminal device; wherein the target state is an online state that does not match the current online scenario.

6. A device for identifying a user's online status, characterized in that: Applied to a server, the device comprises: A video stream receiving module, used to receive a video stream formed by a camera device at a first user end continuously capturing images of the first user; An image frame extraction module, used to extract multiple image frames from the video stream, the video stream is a video stream within a preset time length, and the time intervals for extracting the multiple image frames are equal; An image frame extraction module, configured to extract, for any of the image frames, an image frame containing a binocular image of the first user from the image frame; a classification result recognition module, configured to classify the online status of the first user according to the image frame using a multi-classification model to obtain an online status classification result corresponding to the image frame, wherein the multi-classification model is obtained by performing supervised classification training on multiple sample image frames, each of which includes a binocular image of a person and is respectively annotated with different states of the student when listening to the class based on the binocular images in the sample image frames; An online status determination module, used to determine the current online status of the first user according to multiple online status classification results corresponding to multiple frames of the image frames, wherein the online status is the status of the first user when attending a class, and the online status is determined by the voting results of multiple decision trees constituting the multi-classification model, and the voting result of each decision tree represents the online status classification result of the image frame predicted by the decision tree; The multi-classification model is a gradient boosting decision tree model composed of multiple decision trees, and the classification result recognition module is specifically used for: Convert the image frame into a one-dimensional vector in matrix form; Inputting the one-dimensional vector into the gradient boosting decision tree model, obtaining the voting result of each decision tree in the gradient boosting decision tree model for the online status classification result corresponding to the image frame; wherein the process of each decision tree voting on the image frame is the same as the process of each decision tree performing classification training on multiple frames of the sample image frames; The voting result with the highest vote rate is used as the online status classification result corresponding to the image frame.

7. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for identifying the online status of a user as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying the online status of a user as described in any one of claims 1 to 5 is implemented.

Images