On-hook detection method, device, storage medium and electronic device

Abstract

The present application discloses a hang-up detection method, device, storage medium and electronic device, which relate to the field of data processing technology, and include: identifying each frame of a plurality of consecutive frames of images collected to obtain the center point coordinates of the eyes of a target user in each frame of the image; analyzing the stability of binocular motion in each frame of the image based on the center point coordinates, selecting images whose stability analysis results do not meet a first hang-up condition from the plurality of consecutive frames of images, and splicing the selected images in chronological order to obtain a plurality of spliced ​​images; analyzing the change trend of binocular motion corresponding to the plurality of spliced ​​images based on the center point coordinates, matching the change trend of binocular motion with the standard change trend corresponding to the target user, and analyzing the smoothness of binocular motion based on the obtained matching result; and performing weighted analysis on the obtained smoothness analysis result and stability analysis result to obtain a hang-up detection result of the target user.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a hang-up detection method, device, storage medium and electronic device. Background Art

[0002] In the process of teaching through the Internet, hang-up detection can detect whether the user is truly involved in the learning activities instead of just letting the computer automatically run the course. An effective hang-up detection method helps ensure that the user is truly involved in the learning activities and improves the learning effect.

[0003] At present, the hang-up detection is mainly carried out by collecting user images in real time through the camera to detect whether the user is in the predetermined screen range. If it is determined that the user is not in the predetermined screen range, the timing is started based on the time when the user leaves the predetermined screen range. When the timing reaches the upper limit, it is determined that the user has hung up.

[0004] However, using this AFK detection method, it is impossible to effectively identify the situation where the user has not left the preset screen range but is not studying. For example, the user has been distracted or focused on other electronic devices, etc., and only by monitoring the length of time the user is not in the screen range to determine whether the user has AFK behavior will lead to inaccurate AFK detection results, which in turn will lead to large deviations in the user's learning concentration judged by the AFK detection results, and AFK detection cannot effectively help users improve their learning effects. Summary of the invention

[0005] In view of this, the present application provides a hang-up detection method, device, storage medium and electronic device, the main purpose of which is to improve the technical problem that the current existing technology will lead to inaccurate hang-up detection results, which in turn leads to large deviations in the user's learning concentration judged by the hang-up detection results, and cannot effectively help users improve their learning effects through hang-up detection.

[0006] In a first aspect, the present application provides a method for detecting an on-hook condition, comprising:

[0007] Recognize each frame of the collected continuous multiple frames of images to obtain the center point coordinates of the target user's eyes in each frame of the image, where the center point coordinates are the center point coordinates corresponding to the left eye and the right eye of the target user respectively;

[0008] Analyzing the stability of binocular movement in each frame of the image based on the center point coordinates, selecting images whose stability analysis results do not meet the first hang-up condition from the continuous multiple frames of images, and splicing the selected images in chronological order to obtain multiple spliced ​​images;

[0009] Analyzing a change trend of binocular motion corresponding to the multiple-frame stitched images based on the center point coordinates, matching the change trend of binocular motion with a standard change trend corresponding to the target user, and analyzing the smoothness of binocular motion based on the obtained matching result;

[0010] A weighted analysis is performed on the obtained smoothness analysis result and the stability analysis result to obtain the hang-up detection result of the target user.

[0011] In a second aspect, the present application provides a hang-up detection device, comprising:

[0012] The recognition module is configured to recognize each frame of the collected continuous multiple frames of images to obtain the center point coordinates of the target user's eyes in each frame of the image, wherein the center point coordinates are the center point coordinates corresponding to the left eye and the right eye of the target user respectively;

[0013] an analysis module configured to analyze the stability of binocular movement in each frame of the image based on the center point coordinates, select images whose stability analysis results do not meet the first hang-up condition from the continuous multiple frames of images, and splice the selected images in a chronological order to obtain multiple spliced ​​images;

[0014] The analysis module is further configured to analyze the change trend of binocular motion corresponding to the multiple frames of stitched images based on the center point coordinates, match the change trend of binocular motion with the standard change trend corresponding to the target user, and analyze the smoothness of binocular motion based on the obtained matching result;

[0015] The analysis module is further configured to perform weighted analysis on the obtained smoothness analysis result and the stability analysis result to obtain the hang-up detection result of the target user.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hang-up detection method described in the first aspect.

[0017] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the hang-up detection method described in the first aspect when executing the computer program.

[0018] Through the above technical scheme, the present application provides a hang-up detection method, device, storage medium and electronic device. Compared with the current existing technology, the present application can realize fine-grained recognition of continuous images by identifying each frame of multiple consecutive frames of images to determine the center point coordinates of the user's eyes in each frame of images, and accurately and meticulously detect all the user's binocular behaviors to avoid omissions; the stability of the binocular movement in each frame of images is analyzed through the center point coordinates, so as to judge whether the user has hang-up behavior from the stability level of binocular movement, and further select the images whose stability analysis results do not meet the first hang-up condition from the multiple consecutive frames of images, and splice the selected images in chronological order to obtain a multi-frame spliced ​​image, and the selected images are images for which it is impossible to analyze whether there is hang-up behavior through the stability of binocular movement. For example, by splicing these images in chronological order, we can obtain temporally continuous images again, and then we can analyze the changing trend of binocular movement corresponding to the multi-frame spliced ​​images through the center point coordinates, and then match the changing trend of binocular movement with the standard changing trend corresponding to the target user, and analyze the smoothness of binocular movement based on the matching results. Combining the changing trend of binocular movement with smoothness can not only make accurate analysis based on the changing trend, but also make overall control through smoothness, so as to ensure the accuracy of analysis while ensuring precision, and avoid multiple misjudgments. Finally, through weighted analysis of smoothness analysis results and stability analysis results, we can obtain the hang-up detection situation of the target user in the learning process, and the hang-up detection situation can accurately determine the user's learning concentration, so as to better help users improve their learning effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of a flow chart of a hang-up detection method provided in an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of a flow chart of a hang-up detection method provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of a flow chart of a hang-up detection method provided in an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of a flow chart of a hang-up detection method provided in an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram of a flow chart of a hang-up detection method provided in an embodiment of the present application is shown;

[0026] Figure 6 A schematic diagram of a flow chart of a hang-up detection method provided in an embodiment of the present application is shown;

[0027] Figure 7 A schematic diagram of an example process provided by an embodiment of the present application is shown;

[0028] Figure 8 A schematic structural diagram of a hang-up detection device provided in an embodiment of the present application is shown;

[0029] Fig. 9 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0031] In order to improve the technical problem that the existing technology may lead to inaccurate hang-up detection results, which in turn leads to a large deviation in the user's learning concentration determined by the hang-up detection results, and the hang-up detection cannot effectively help users improve their learning effects. This embodiment provides a hang-up detection method, such as Figure 1 As shown, the method includes:

[0032] Step 101: Identify each frame of the collected continuous multiple frames of images to obtain the center point coordinates of the target user's eyes in each frame of the image.

[0033] The center point coordinates are the center point coordinates corresponding to the left eye and the right eye of the target user respectively.

[0034] In an embodiment of the present application, the continuous multiple-frame images may be acquired in real time through a video file or a camera, and the images may be images containing a human face.

[0035] Furthermore, the target user may be a person who needs to be tested for hang-up, for example, a user who needs to learn through online videos or online live broadcasts.

[0036] Correspondingly, the center point coordinates may be the center point coordinates corresponding to the user's left eye and right eye respectively. The center point coordinates may be used to identify each frame of the image through Dlib's 68-point model, identify the human eye data in each frame of the image, and determine the center point coordinates of the human eye from the human eye coordinate data.

[0037] Specifically, Dlib is an open source deep learning library, which is widely used in the field of facial feature point detection. The 68-point model is mainly used for face alignment and expression analysis, and the position of facial features is determined by detecting key points on the face. These key points include important positions of eyes, eyebrows, nose, mouth, chin and other parts. In the embodiment of the present application, the Dlib library can be used to perform key point detection on the extracted facial image, and 68 different facial key points are detected, including feature points of multiple facial areas such as eyes, eyebrows, nose, mouth and chin. Specifically, in each image, the system uses the Dlib 68-point model to extract 68 key points of the user's face, and determines the coordinates of the center points of the eyes through the key points of the eyes in the facial key points.

[0038] It should be noted that the center point coordinates in this application are composed of three dimensions, namely horizontal position, vertical position and time.

[0039] Step 102: Analyze the stability of binocular movement in each frame of image based on the center point coordinates, select images whose stability analysis results do not meet the first hang-up condition from multiple consecutive frames of images, and splice the selected images in chronological order to obtain multiple frames of spliced ​​images.

[0040] In some examples, the stability of binocular movement in each frame of image may be a stability of a process in which the binocular movement in each frame of image changes from a center point of a corresponding previous frame of image to a current center point.

[0041] Exemplarily, the first on-hook condition may be a condition set for the stability analysis result and capable of determining the presence of an on-hook behavior.

[0042] Optionally, stitching in chronological order can be performed by re-stitching the selected pictures in a continuous time sequence. For example, if the time coordinates of the images to be stitched before stitching are t1, t5 and t6, the time coordinates of the images obtained after stitching in chronological order are t1, t2 and t3, that is, the images of multiple time periods are re-stitched into a chronologically continuous image.

[0043] Step 103: Analyze the change trend of binocular motion corresponding to the multi-frame spliced ​​image based on the center point coordinates, match the change trend of binocular motion with the standard change trend corresponding to the target user, and analyze the smoothness of binocular motion based on the obtained matching result.

[0044] It should be noted that the stability of the binocular movement in step 102 can be used to determine the hang-up situation with obviously abnormal stability. For example, if the user uses a photo instead, it can be determined that the stability of the user's eyes meets the hang-up conditions. For hang-up behaviors that are not very obvious, it is necessary to further judge through the changing trend of the binocular movement combined with the smoothness.

[0045] In an embodiment of the present application, the changing trend of binocular movement can be the changing situation of the user's eyes in a multi-frame stitched image. Matching the changing trend of binocular movement with the standard changing trend can determine that there are abnormal images in the multi-frame stitched images with abnormal changing trends. On this basis, the smoothness of the overall binocular changes of the multi-frame stitched images is analyzed to accurately determine whether there is any hanging behavior in the multi-frame stitched images.

[0046] It should be noted that the analysis object of the changing trend of binocular movement is several consecutive frames in the multi-frame stitched image. Specifically, the changing trend can be analyzed for three consecutive frames. The smoothness of binocular movement is based on the multi-frame stitched image as a whole. Combining the two can not only accurately analyze subtle movements, but also make corrections on the whole to avoid misjudgment. For example, if the user occasionally feels sore eyes and needs relief, the changing trend of binocular movement will be judged as hanging up behavior, but correction will be made in the process of analyzing the overall smoothness, which ensures the accuracy of hanging up detection to a greater extent.

[0047] Step 104: Perform weighted analysis on the obtained smoothness analysis results and stability analysis results to obtain the hang-up detection result of the target user.

[0048] For this embodiment, the on-hook detection result of the target user may include no on-hook behavior, slight on-hook behavior, and severe on-hook behavior.

[0049] Compared with the current existing technology, this embodiment identifies each frame of the continuous multi-frame image and determines the center point coordinates of the user's eyes in each frame, so as to realize fine-grained analysis of the continuous images and accurately and meticulously detect all the user's binocular behaviors to avoid omissions; the stability of the binocular movement in each frame is analyzed through the center point coordinates, so as to preliminarily judge whether the user has hang-up behavior from the stability level of binocular movement, and further selects the images whose stability analysis results do not meet the first hang-up condition from the continuous multi-frame image, and splices the selected images in chronological order to obtain a multi-frame spliced ​​image, the selected images are images for which it is impossible to analyze whether there is hang-up behavior through the stability of binocular movement, and these images are spliced ​​in chronological order. Stitching can obtain temporally continuous images again, and then the changing trend of binocular movement corresponding to the multi-frame stitched images can be analyzed through the center point coordinates, and then the changing trend of binocular movement is matched with the standard changing trend corresponding to the target user, and the smoothness of binocular movement is analyzed based on the matching result. Combining the changing trend of binocular movement with smoothness can not only perform accurate analysis based on the changing trend, but also perform overall control through smoothness, ensuring the accuracy of the analysis while ensuring accuracy, avoiding multiple misjudgments, and finally, through weighted analysis of the smoothness analysis results and the stability analysis results, the hang-up detection situation of the target user in the learning process is obtained. The hang-up detection situation can accurately determine the user's learning concentration, and better help users improve their learning effects.

[0050] As a refinement and extension of the above embodiment, when analyzing the stability of binocular motion in each frame of image based on the center point coordinates, the following methods may be used but are not limited to: Figure 2 As shown, the method includes:

[0051] Step 201: Based on the horizontal coordinate, the vertical coordinate and the time coordinate in the center point coordinate, determine the first binocular motion distance between each frame image and its corresponding previous frame image, and the second binocular motion distance between each frame image and its corresponding next frame image.

[0052] In the embodiment of the present application, the time coordinate in the center point coordinate can be the timestamp corresponding to each frame of the image. Specifically, when processing a video stream or an image sequence, the timestamp corresponding to the image is very important information, which can help us understand the specific time point of the image capture. Timestamps are usually used to synchronize video streams, perform time series analysis, and record the time when an event occurs.

[0053] For example, after capturing the user's facial image, the eye center point data is first obtained through Dlib's 68-point model, and a three-dimensional coordinate list is generated through different timestamps. The eye coordinates in each frame are composed of three dimensions: horizontal position (X), vertical position (Y) and time (T), indicating the movement of the eye at different time points.

[0054] For example, four consecutive frames of images are image A, image B, image C, and image D. The coordinates of the center points corresponding to image A, image B, image C, and image D can be:

[0055] Image A: Left eye center coordinates: left_eye_center = [322.5, 715.5, 1.00], right eye center coordinates: right_eye_center = [372.5, 714.5, 1.00]

[0056] Image B: Left eye center coordinates: left_eye_center = [422.5, 725.5, 1.05], right eye center coordinates: right_eye_center = [462.5, 725.5, 1.05]

[0057] Image C: Left eye center coordinates: left_eye_center = [472.5, 723.5, 1.10], right eye center coordinates: right_eye_center = [522.5, 724.5, 1.10]

[0058] Image D: Left eye center coordinates: left_eye_center = [522.5, 729.5, 1.15], right eye center coordinates: right_eye_center = [585.0, 730.0, 1.15]

[0059] In Image A, Image B, Image C, and Image D, each coordinate point consists of (X, Y, T), where T represents the timestamp when the image is captured.

[0060] In some examples, the first binocular motion distance is the distance between the center point of each frame image and its corresponding previous frame image, and the second binocular distance is the distance between the center point of each frame image and its corresponding next frame image.

[0061] For example, four consecutive frames of images are image A, image B, image C and image D. For image B, the first binocular distance is the distance between the center point of image B and the center point of image A, and the second binocular distance is the distance between the center point of image B and the center point of image C.

[0062] Specifically, the first binocular distance and the second binocular distance can be determined by Formula 1, and Formula 1 is specifically as follows:

[0063] (Formula 1)

[0064] In Formula 1, distance represents the first binocular distance or the second binocular distance, wherein if distance represents the first binocular distance, x2 and x1 represent the horizontal coordinate of the center point of each frame image and the horizontal coordinate of the center point of the corresponding previous frame image, y2 and y1 represent the vertical coordinate of the center point of each frame image and the vertical coordinate of the center point of the corresponding previous frame image, t2 and t1 represent the time coordinate of the center point of each frame image and the time coordinate of the center point of the corresponding previous frame image, respectively;

[0065] If distance represents the second binocular distance, x2 and x1 represent the horizontal coordinate of the center point of each frame image and the horizontal coordinate of the center point of the next frame image corresponding thereto, y2 and y1 represent the vertical coordinate of the center point of each frame image and the vertical coordinate of the center point of the next frame image corresponding thereto, and t2 and t1 represent the time coordinate of the center point of each frame image and the time coordinate of the center point of the next frame image corresponding thereto.

[0066] Exemplarily, based on the center point coordinates corresponding to the above-mentioned images A, B, C and D, it can be determined that the left eye distance between image A and image B is 100.49, and the right eye distance is 90.66; the left eye distance between image B and image C is 50.04, and the right eye distance is 60.01; the left eye distance between image C and image D is 50.36, and the right eye distance is 62.74.

[0067] Step 202: Determine the instantaneous binocular movement speed corresponding to each frame of image based on the difference between the first binocular movement distance and the second binocular movement distance and the time difference between each frame of image and its corresponding previous frame of image.

[0068] The instantaneous speed is used to indicate the speed change of each frame of image compared with its corresponding previous frame of image.

[0069] Specifically, the instantaneous speed of each frame of image can be determined by Formula 2, which is specifically as follows:

[0070] (Formula 2)

[0071] In formula 2, X represents the instantaneous speed of each frame of image. Indicates the second binocular motion distance corresponding to each frame image, Indicates the first binocular motion distance corresponding to each frame image, Indicates the time coordinate corresponding to each frame of image, Indicates the time coordinate of the previous frame of image corresponding to each frame of image.

[0072] It should be noted that if the instantaneous speed obtained is a negative value, it can be determined that the speed of each frame image is smaller than that of the corresponding previous frame image. If the instantaneous speed obtained is a positive value, it can be determined that the speed of each frame image is larger than that of the corresponding previous frame image.

[0073] Step 203: Determine the instantaneous speed variance corresponding to each frame of image based on the instantaneous speed of binocular movement corresponding to each frame of image and the instantaneous speed mean, and analyze the stability of binocular movement in each frame of image through the instantaneous speed variance to obtain the stability value corresponding to each frame of image.

[0074] In the embodiment of the present application, in order to judge the stability of the user's eye movement, we use variance to measure the fluctuation of instantaneous speed. The larger the variance, the more unstable the eye movement is, and there may be a situation where the user is hanging up or deceiving the system.

[0075] Specifically, the variance can be used to analyze the stability of binocular motion in each frame of the image through Formula 3, which is as follows:

[0076] (Formula 3)

[0077] In formula 3, Indicates the stability value of binocular movement, represents the instantaneous speed corresponding to the i-th frame image, It represents the average instantaneous speed of all frames, and N represents the number of consecutive multi-frame images.

[0078] Further, when selecting images whose stability analysis results do not meet the first hang-up condition from the continuous multiple frames of images, and splicing the selected images in chronological order to obtain multiple frames of spliced ​​images, the following steps may be adopted but are not limited to, such as Figure 3 As shown, including:

[0079] Step 301: Identify at least one group of continuous candidate images whose stability values ​​exceed a predetermined stability threshold from a plurality of continuous frame images, and determine the number of continuous frames corresponding to the at least one group of continuous candidate images.

[0080] It should be noted that the stability threshold can be set based on demand or experience, and the specific value of the stability threshold is not limited in the embodiments of the present application.

[0081] Optionally, if the stability exceeds a preset threshold, it means that the eye movement is unstable, and the user may not be operating seriously or using static photos instead.

[0082] For example, if the time coordinates of multiple consecutive frames of images correspond to t1, t2, ...ti, there may be consecutive candidate images corresponding to time ta to tb that exceed the stability threshold and consecutive candidate images corresponding to time tc to td that exceed the stability threshold, and so on.

[0083] Step 302: Determine the continuous candidate images whose continuous frame number is greater than or equal to the predetermined frame number threshold in at least one group of continuous candidate images as target continuous images with hang-up behavior, and determine the first hang-up number of the target user based on the number of groups of target continuous images.

[0084] Exemplarily, based on step 301, if all the continuous frames corresponding to the time ta to tb exceed the threshold, and (ba) is greater than the preset number of times c (i.e., the predetermined frame number threshold in step 302), then the continuous candidate images corresponding to the time ta to tb are determined as the target continuous images, which means that the eye movement in the target continuous image is unstable, and the number of hanging-up behaviors is counted as +1.

[0085] Step 303: splice the remaining images after removing the target continuous image from the continuous multiple frame images in chronological order to obtain a multi-frame spliced ​​image.

[0086] Exemplarily, based on step 302, the continuous multi-frame image is filtered out from ta to tb frames to take the remaining frames, and the ta-1 moment is connected with the tb+1 moment, that is, the time coordinate of the image corresponding to the tb+1 moment is replaced by tb+1 to the new ta. This processing can transform the spliced ​​image into a continuous image in time sequence, that is, the subsequent smoothness analysis of human eye movement can be performed based on the continuous image (that is, the multi-frame spliced ​​image in step 303).

[0087] Optionally, when analyzing the changing trend of binocular movements corresponding to the multi-frame spliced ​​images based on the center point coordinates, the following methods may be used but are not limited to: Figure 4 As shown, the method includes:

[0088] Step 401: Based on the coordinates of the center point, a first coordinate graph showing changes in the horizontal coordinates of both eyes with the time coordinates and a second coordinate graph showing changes in the vertical coordinates with the time coordinates are constructed.

[0089] In the embodiment of the present application, both the first coordinate diagram and the second coordinate diagram are two-dimensional coordinate diagrams. The first coordinate diagram only includes the horizontal coordinates and time coordinates of both eyes, and the second coordinate diagram only includes the vertical coordinates and time coordinates of the binocular movements.

[0090] Exemplarily, the coordinates in the first coordinate graph may be: left left eye (x1, t1), (x2, t2), (x3, t3), (x4, t4) ..... (xn, tn), and correspondingly, the coordinates in the second coordinate graph may be left left eye (y1, t1), (y2, t2), (y3, t3), (y4, t4) ..... (yn, tn).

[0091] Step 402: In the first coordinate graph, every three consecutive coordinates are formed into a first triangle, and the changing trend of the horizontal coordinate in the changing trend of the binocular movement is determined by the areas of the multiple first triangles formed in the first coordinate graph; and in the second coordinate graph, every three consecutive coordinates are formed into a second triangle, and the changing trend of the vertical coordinate in the changing trend of the binocular movement is determined by the areas of the multiple second triangles formed in the second coordinate graph.

[0092] For this embodiment, the first coordinate graph and the second coordinate graph both include multiple coordinate points, wherein every three consecutive coordinate points can form a triangle. For example, if the multiple coordinate points included in the first coordinate graph are (x1, t1), (x2, t2), (x3, t3), (x4, t4) ..... (xn, tn), then (x1, t1), (x2, t2), (x3, t3) can form a first triangle, (x2, t2), (x3, t3), (x4, t4) can form a second triangle, and so on. Examples are not given one by one here.

[0093] Correspondingly, if the multiple coordinate points included in the second coordinate graph are (y1, t1), (y2, t2), (y3, t3), (y4, t4) ..... (yn, tn), then (y1, t1), (y2, t2), (y3, t3) can form the first triangle, (y2, t2), (y3, t3), (y4, t4) can form the second triangle, and so on, and examples are not given one by one here.

[0094] Furthermore, in the first coordinate diagram and the second coordinate diagram, if there are n coordinate points, that is, n-2 triangles can be formed, the area of ​​the n-2 triangles formed in the first coordinate diagram can represent the changing trend of the horizontal coordinates in the changing trend of the binocular movement, and the area of ​​the n-2 triangles formed in the second coordinate diagram can represent the changing trend of the vertical coordinates in the changing trend of the binocular movement.

[0095] For example, if the first coordinate graph includes multiple coordinate points P1, P2, ..., P n , then every three consecutive points form a triangle, forming a total of n-2 triangles: (P1, P2, P3), (P2, P3, P4),..., (Pn-2 , P n-1 , P n ).

[0096] Specifically, for each of the three consecutive points P n-2 (x n-2 , t n-2) , P n-1 (x n-1 , t n-1 ), P n (x n , t n ) can be represented by the vector as shown in the following formulas 4 and 5:

[0097] (Formula 4)

[0098] (Formula 5)

[0099] In Formula 4 and Formula 5, Expressed as coordinate P n-2 and coordinates P n-1 The vector between Expressed as coordinate P n-1 and coordinates P n The vector between .

[0100] Based on Formula 4 and Formula 5, we can combine Formula 6 to determine the continuous three points P n-2 (x n-2 , t n-2) , P n-1 (x n-1 , t n-1 ), P n (x n , t n ) is the area of ​​the triangle formed by the equation 6 as follows:

[0101] X=1 / 2( ) (Formula 6)

[0102] In formula 6, X represents the continuous three points P n-2 (x n-2 , t n-2) , P n-1 (x n-1 , t n-1 ), P n (x n , t n ) is the area of ​​the triangle formed by .

[0103] Further, when matching the change trend of binocular movement with the standard change trend corresponding to the target user, the following steps may be adopted but are not limited to: Figure 5 As shown, including:

[0104] Step 501: Determine a plurality of first area differences between the areas of the first triangles and a plurality of first area differences between the horizontal standard triangles, and a plurality of second area differences between the areas of the second triangles and a plurality of second area differences between the vertical standard triangles.

[0105] Among them, the horizontal standard triangle is used to determine the horizontal standard change trend in the standard change trend, and the vertical standard triangle is used to determine the vertical standard change trend in the standard change trend. The horizontal standard change trend and the vertical standard change trend are determined based on the center point coordinates of the target user's eyes in the standard continuous image.

[0106] In an embodiment of the present application, the horizontal standard triangle and the vertical standard triangle are obtained based on the standard center point coordinates of the user's eyes in the recorded learning video data watched by the user under standard conditions. For example, the standard conditions may be that the user should not hang up or the user's concentration in watching the video meets the conditions, etc.

[0107] Furthermore, for the recorded learning video data watched by users under standard conditions, it is necessary to obtain the face image of the course learning video at T (distance) intervals, and display it in the same fixed ratio as the continuous multi-frame image currently undergoing hang-up detection, and then locate the facial feature points of the above-mentioned Dlib 68-point model to obtain the standard human eye center point coordinates extracted based on the standard learning video, and based on the standard human eye center point coordinates, respectively construct a first standard coordinate map containing only horizontal coordinates and time coordinates and a second standard coordinate map containing only vertical coordinates and time coordinates, and then determine the horizontal standard triangle based on the first standard coordinate map, and determine the vertical standard triangle based on the second standard coordinate map.

[0108] Step 502: Based on the multiple first area difference values ​​and the multiple second area difference values, determine multiple area difference values ​​representing the matching results of the change trend of binocular movement and the standard change trend.

[0109] In the embodiment of the present application, the multiple area differences may be the sum of the first area difference and the second area difference, which may represent the overall difference between the changing trend of binocular movement and the standard changing trend.

[0110] Further, when analyzing the smoothness of binocular movement based on the obtained matching results, the following steps may be adopted but are not limited to: Figure 6 As shown, including:

[0111] Step 601: Determine a standard deviation value representing the smoothness of binocular motion in a plurality of consecutive image frames according to a plurality of area difference values, the number of the plurality of area difference values, and the average of the plurality of area difference values.

[0112] In the embodiment of the present application, the standard deviation value representing the smoothness of binocular motion in the continuous multiple frames of images is determined according to the multiple area differences, the number of the multiple area differences and the average of the multiple area differences, which can be calculated by the following formula 7:

[0113] (Formula 7)

[0114] In Formula 7, The sample standard deviation can represent the smoothness of binocular motion in multiple consecutive frames of images. is the sample capacity (i.e., the number of area differences in consecutive multi-frame images), is the area difference of each sample, Represents the mean of multiple area differences.

[0115] Step 602: If there is an area difference value exceeding a predetermined difference threshold value among the multiple area difference values ​​and the standard deviation value exceeds a predetermined standard deviation threshold value, the number of area difference values ​​exceeding the predetermined difference threshold value is determined as the second hang-up number of the target user.

[0116] In an embodiment of the present application, by combining judgments based on multiple area differences and smoothness, it is possible to accurately analyze subtle movements and make corrections overall to avoid misjudgments. For example, if a user occasionally feels sore eyes and needs relief, the changing trend of binocular movement will determine that there is an on-hook behavior, but corrections will be made in the process of analyzing the overall smoothness, thereby ensuring the accuracy of on-hook detection to a greater extent.

[0117] Specifically, the area difference corresponds to three frames of images. The three frames of images with abnormal area difference correspond to situations where the user may take a short break for the eyes. If the analysis is not combined with the smoothness of the overall binocular movement, it will lead to many misjudgments of the hang-up behavior. After analysis combined with the smoothness of the overall binocular movement, correction can be made as a whole to avoid misjudgments, thereby ensuring the accuracy of hang-up detection to a greater extent.

[0118] It should be noted that the predetermined difference threshold and the predetermined standard deviation threshold can be set according to experience or demand, and the specific values ​​of the predetermined difference threshold and the predetermined standard deviation threshold are not limited in the embodiments of the present application.

[0119] Optionally, the second on-hook number may be the on-hook number detected by area difference and smoothness analysis.

[0120] Optionally, when performing weighted analysis on the obtained smoothness analysis results, stability analysis results, and matching results to obtain the on-hook detection result of the target user, the following steps may be adopted but are not limited to, including:

[0121] Step 11: Obtain the sum of the first hanging-up times and the second hanging-up times corresponding to the multiple groups of continuous multi-frame images included in the complete learning process of the user, and the third hanging-up times detected through the user operation behavior.

[0122] For this embodiment, the user operation behavior may be a browser tab switching behavior, a mouse movement behavior, a keyboard typing behavior, etc., and the third hang-up number may be the hang-up number detected together through the user operation behaviors.

[0123] Step 12: Perform weighted analysis on the total of the first hang-up times, the second hang-up times, and the third hang-up times to obtain the hang-up detection result of the target user in the complete learning process.

[0124] In some examples, the user's learning effectiveness score can be obtained by performing a weighted analysis on the total of the first hang-up times, the total of the second hang-up times, and the third hang-up times, and the hang-up detection results of the user in the complete learning process can be judged by the learning effectiveness score.

[0125] Specifically, the sum of the first hang-up times, the second hang-up times, and the third hang-up times can be weighted analyzed by formula 8, and formula 8 is specifically as follows:

[0126] (Formula 8)

[0127] In Formula 8, R represents the learning effectiveness score used to determine the on-hook detection status of the target user, C1, C2 C i Indicates the number of different types of on-hook behaviors, specifically the sum of the first on-hook number, the second on-hook number, and the third on-hook number, K1, K2 Ki represents the weights of different types of hang-up detection methods, and specifically may be the weights corresponding to binocular movement smoothness analysis detection, binocular movement stability analysis, and user operation behavior detection, respectively.

[0128] Exemplarily, determining the hang-up detection result through the learning effectiveness score may include: if the R value is 0, it is determined that the user has no hang-up behavior; if the R value is greater than the preset (minor) threshold, the user has slight hang-up behavior; if the R value is greater than the preset (serious) threshold, the user has severe hang-up behavior, wherein the values ​​of the preset (minor) threshold and the preset (serious) threshold can be set according to experience or demand, and the values ​​of the preset (minor) threshold and the preset (serious) threshold are not limited in the embodiments of the present application.

[0129] Compared with the current existing technology, this embodiment identifies each frame of the continuous multi-frame image and determines the center point coordinates of the user's eyes in each frame, so as to realize fine-grained analysis of the continuous images and accurately and meticulously detect all the user's binocular behaviors to avoid omissions; the stability of the binocular movement in each frame is analyzed through the center point coordinates, so as to preliminarily judge whether the user has hang-up behavior from the stability level of binocular movement, and further selects the images whose stability analysis results do not meet the first hang-up condition from the continuous multi-frame image, and splices the selected images in chronological order to obtain a multi-frame spliced ​​image, the selected images are images for which it is impossible to analyze whether there is hang-up behavior through the stability of binocular movement, and these images are spliced ​​in chronological order. Stitching can obtain temporally continuous images again, and then the changing trend of binocular movement corresponding to the multi-frame stitched images can be analyzed through the center point coordinates, and then the changing trend of binocular movement is matched with the standard changing trend corresponding to the target user, and the smoothness of binocular movement is analyzed based on the matching result. Combining the changing trend of binocular movement with smoothness can not only perform accurate analysis based on the changing trend, but also perform overall control through smoothness, ensuring the accuracy of the analysis while ensuring accuracy, avoiding multiple misjudgments, and finally, through weighted analysis of the smoothness analysis results and the stability analysis results, the hang-up detection situation of the target user in the learning process is obtained. The hang-up detection situation can accurately determine the user's learning concentration, and better help users improve their learning effects.

[0130] In order to illustrate the specific implementation process of this embodiment, the following specific application examples are given: Figure 7 As shown, but not limited to:

[0131] First, when the user starts to learn the video course, create an array for detecting the user's left and right eye behavior and the eye position array ID. Get the user's face data every ts, and get the position of the eyes based on the face data (if there is no eye data, set it to [0,0]). Calculate the center points of the two eyes and store them in the preset data group. If there is historical collection data, calculate the movement amount dis1 of the left and right eye center points of the last collection, calculate the instantaneous speed between the distance dis0 of the left and right eye center points of the previous frame, and if there is a historical instantaneous speed, calculate the instantaneous speed of the left and right eyes and the square sum of the speed deviations, find the variance, cut off the data that exceeds the threshold time period, get the user's historical course data (if there is no historical course data, you can get the face data stored in other applications in the device), generate the current user's left and right eye (x, t) (y, t) map, calculate the area difference of the historical data at the same time, calculate the smoothness, and judge the user status by comparing the smoothness threshold to get the hang-up detection result.

[0132] Compared with the current existing technology, this embodiment identifies each frame of the continuous multi-frame image and determines the center point coordinates of the user's eyes in each frame, so as to realize fine-grained analysis of the continuous images and accurately and meticulously detect all the user's binocular behaviors to avoid omissions; the stability of the binocular movement in each frame is analyzed through the center point coordinates, so as to preliminarily judge whether the user has hang-up behavior from the stability level of binocular movement, and further selects the images whose stability analysis results do not meet the first hang-up condition from the continuous multi-frame image, and splices the selected images in chronological order to obtain a multi-frame spliced ​​image, the selected images are images for which it is impossible to analyze whether there is hang-up behavior through the stability of binocular movement, and these images are spliced ​​in chronological order. Stitching can obtain temporally continuous images again, and then the changing trend of binocular movement corresponding to the multi-frame stitched images can be analyzed through the center point coordinates, and then the changing trend of binocular movement is matched with the standard changing trend corresponding to the target user, and the smoothness of binocular movement is analyzed based on the matching result. Combining the changing trend of binocular movement with smoothness can not only perform accurate analysis based on the changing trend, but also perform overall control through smoothness, ensuring the accuracy of the analysis while ensuring accuracy, avoiding multiple misjudgments, and finally, through weighted analysis of the smoothness analysis results and the stability analysis results, the hang-up detection situation of the target user in the learning process is obtained. The hang-up detection situation can accurately determine the user's learning concentration, and better help users improve their learning effects.

[0133] Further, as Figures 1 to 6 The specific implementation of the method shown in the embodiment provides a hang-up detection device, such as Figure 8As shown, the device includes: an identification module 71 and an analysis module 72.

[0134] The recognition module 71 is configured to recognize each frame of the collected continuous multiple frames of images, and obtain the center point coordinates of the target user's eyes in each frame of the image, wherein the center point coordinates are the center point coordinates corresponding to the left eye and the right eye of the target user respectively;

[0135] The analysis module 72 is configured to analyze the stability of binocular movement in each frame of the image based on the center point coordinates, select images whose stability analysis results do not meet the first hang-up condition from the continuous multiple frames of images, and splice the selected images in a chronological order to obtain multiple spliced ​​images;

[0136] The analysis module 72 is further configured to analyze the change trend of binocular motion corresponding to the multiple frames of stitched images based on the center point coordinates, match the change trend of binocular motion with the standard change trend corresponding to the target user, and analyze the smoothness of binocular motion based on the obtained matching result;

[0137] The analysis module 72 is further configured to perform weighted analysis on the obtained smoothness analysis result and the stability analysis result to obtain the hang-up detection result of the target user.

[0138] In some examples of the present embodiment, the analysis module 72 is specifically configured to determine a first binocular motion distance between each frame image and its corresponding previous frame image, and a second binocular motion distance between each frame image and its corresponding next frame image based on the horizontal coordinate, vertical coordinate and time coordinate in the center point coordinates; determine the instantaneous speed of the binocular motion corresponding to each frame image according to the difference between the first binocular motion distance and the second binocular motion distance and the time difference between each frame image and its corresponding previous frame image, and the instantaneous speed is used to represent the speed change of each frame image compared with its corresponding previous frame image; determine the instantaneous speed variance corresponding to each frame image according to the instantaneous speed of the binocular motion corresponding to each frame image and the instantaneous speed mean, and analyze the stability of the binocular motion in each frame image through the instantaneous speed variance to obtain the stability value corresponding to each frame image.

[0139] In some examples of the present embodiment, the analysis module 72 is further configured to identify at least one group of continuous candidate images whose stability values ​​exceed a predetermined stability threshold from the continuous multi-frame images, and determine the number of continuous frames corresponding to the at least one group of continuous candidate images; determine the continuous candidate images whose continuous frame numbers are greater than or equal to the predetermined frame number threshold in the at least one group of continuous candidate images as target continuous images with hanging-up behavior, and determine the first hanging-up number of the target user based on the number of groups of the target continuous images; and splice the remaining images in the continuous multi-frame images after removing the target continuous images in chronological order to obtain the multi-frame spliced ​​image.

[0140] In some examples of the present embodiment, the analysis module 72 is further specifically configured to construct a first coordinate graph of the horizontal coordinates of the two eyes changing with the time coordinates, and a second coordinate graph of the vertical coordinates changing with the time coordinates based on the center point coordinates; in the first coordinate graph, every three consecutive coordinates are formed into a first triangle, and the changing trend of the horizontal coordinates in the changing trend of the binocular movement is determined by the areas of the multiple first triangles formed in the first coordinate graph; and in the second coordinate graph, every three consecutive coordinates are formed into a second triangle, and the changing trend of the vertical coordinates in the changing trend of the binocular movement is determined by the areas of the multiple second triangles formed in the second coordinate graph.

[0141] In some examples of this embodiment, the analysis module 72 is further configured to determine a plurality of first area differences between the areas of the plurality of first triangles and a plurality of first area differences between the horizontal standard triangles, and a plurality of second area differences between the areas of the plurality of second triangles and a plurality of second area differences between the vertical standard triangles, the horizontal standard triangles being used to determine the horizontal standard change trend in the standard change trend, the vertical standard triangles being used to determine the vertical standard change trend in the standard change trend, the horizontal standard change trend and the vertical standard change trend being determined based on the center point coordinates of both eyes of the target user in the standard continuous image; based on the plurality of first area differences and the plurality of second area differences, a plurality of area differences representing the matching results of the change trend of the binocular movement and the standard change trend are determined.

[0142] In some examples of the present embodiment, the analysis module 72 is further specifically configured to determine a standard deviation value representing the smoothness of the binocular movement in the continuous multiple frames of images based on the multiple area difference values, the number of the multiple area difference values ​​and the mean of the multiple area difference values; if there are area difference values ​​among the multiple area difference values ​​that exceed a predetermined difference threshold and the standard deviation value exceeds the predetermined standard deviation threshold, then the number of area difference values ​​that exceed the predetermined difference threshold is determined as the second hang-up number of the target user.

[0143] In some examples of this embodiment, the analysis module 72 is further specifically configured to obtain the sum of the first hanging-up times and the second hanging-up times corresponding to multiple groups of continuous multi-frame images included in the user's complete learning process, as well as the third hanging-up times detected through user operation behavior; perform weighted analysis on the first hanging-up times, the second hanging-up times and the third hanging-up times to obtain the hanging-up detection result of the target user in the complete learning process.

[0144] It should be noted that for other corresponding descriptions of the functional units involved in the on-hook detection device provided in this embodiment, reference can be made to Figures 1 to 6 The corresponding description in will not be repeated here.

[0145] Based on the above Figures 1 to 6 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figures 1 to 6 The method shown.

[0146] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0147] like Fig. 9 The figure shows a hardware structure diagram of an electronic device of the present invention, including:

[0148] at least one processor 801; and,

[0149] A memory 802 is communicatively connected to at least one of the processors 801; wherein,

[0150] The memory 802 stores instructions that can be executed by at least one of the processors. The instructions are executed by at least one of the processors so that the at least one of the processors can execute the above-mentioned on-hook detection method.

[0151] Fig. 9 A processor 801 is taken as an example.

[0152] The electronic device may further include: an input device 803 and a display device 804 .

[0153] The processor 801, the memory 802, the input device 803 and the display device 804 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0154] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the hang-up detection method in the embodiment of the present application, for example, Figures 1 to 6 The processor 801 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 802, that is, the on-hook detection method in the above embodiment is implemented.

[0155] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the hang-up detection method, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the device for executing the hang-up detection method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The input device 803 can receive user clicks and generate signal inputs related to user settings and function control of the on-hook detection method. The display device 804 can include display devices such as a display screen.

[0157] The one or more modules are stored in the memory 802, and when executed by the one or more processors 801, the on-hook detection method in any of the above method embodiments is executed.

[0158] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0159] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0160] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0161] Through the description of the above implementation methods, the technical personnel in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or it can be implemented by hardware. By applying the scheme of this embodiment, compared with the current existing technology, this embodiment can realize fine-grained analysis of continuous images by identifying each frame of continuous multiple frames of images and determining the center point coordinates of the user's eyes in each frame of images, and accurately and meticulously detect all binocular behaviors of the user to avoid omissions; by analyzing the stability of binocular movements in each frame of images through the center point coordinates, it is possible to preliminarily judge whether the user has hang-up behavior from the stability level of binocular movements, and further by selecting the images whose stability analysis results do not meet the first hang-up condition from the continuous multiple frames of images, and splicing the selected images in chronological order to obtain multiple frames of spliced ​​images, the selected images are images for which it is impossible to analyze whether there is hang-up behavior through the stability of binocular movements, and these images are spliced ​​in chronological order. Stitching together the images sequentially can obtain a time-series continuous image again, and then the changing trend of binocular movement corresponding to the multi-frame stitched image can be analyzed through the center point coordinates, and then the changing trend of binocular movement is matched with the standard changing trend corresponding to the target user, and the smoothness of binocular movement is analyzed based on the matching result. Combining the changing trend of binocular movement with smoothness can not only make accurate analysis based on the changing trend, but also make overall control through smoothness, so as to ensure the accuracy of analysis while ensuring precision, and avoid multiple misjudgments. Finally, through weighted analysis of the smoothness analysis results and the stability analysis results, the hang-up detection situation of the target user in the learning process is obtained. The hang-up detection situation can accurately determine the user's learning concentration, so as to better help users improve their learning effects.

[0162] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0163] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A method for detecting a hang-up, characterized in that: include: Recognize each frame of the collected continuous multiple frames of images to obtain the center point coordinates of the target user's eyes in each frame of the image, where the center point coordinates are the center point coordinates corresponding to the left eye and the right eye of the target user respectively; The stability of binocular motion in each frame image is analyzed based on the center point coordinates, images whose stability analysis results do not meet the first hang-up condition are selected from the continuous multiple frames of images, and the selected images are spliced ​​in chronological order to obtain multiple spliced ​​images, including: based on the horizontal coordinates, vertical coordinates and time coordinates in the center point coordinates, a first binocular motion distance between each frame image and its corresponding previous frame image, and a second binocular motion distance between each frame image and its corresponding next frame image are determined; based on the difference between the first binocular motion distance and the second binocular motion distance, and the time difference between each frame image and its corresponding previous frame image, the instantaneous speed of the binocular motion corresponding to each frame image is determined, and the instantaneous speed is used to represent the speed change of each frame image compared with the corresponding previous frame image; based on the instantaneous speed of the binocular motion corresponding to each frame image and the instantaneous speed mean, the instantaneous speed variance corresponding to each frame image is determined, and the stability of the binocular motion in each frame image is analyzed by the instantaneous speed variance to obtain the stability value corresponding to each frame image; Based on the center point coordinates, a change trend of binocular motion corresponding to the multi-frame spliced ​​image is analyzed, the change trend of binocular motion is matched with the standard change trend corresponding to the target user, and the smoothness of binocular motion is analyzed based on the obtained matching result, including: based on the center point coordinates, constructing a first coordinate graph of changes in the horizontal coordinates of the binoculars with the time coordinates, and a second coordinate graph of changes in the vertical coordinates with the time coordinates; in the first coordinate graph, every three consecutive coordinates are formed into a first triangle, and the change trend of the horizontal coordinates in the change trend of the binocular motion is determined by the areas of the multiple first triangles formed in the first coordinate graph, and in the second coordinate graph, every three consecutive coordinates are formed into a second triangle, and the change trend of the vertical coordinates in the change trend of the binocular motion is determined by the areas of the multiple second triangles formed in the second coordinate graph; determining the difference between the areas of the multiple first triangles and the multiple first areas of the horizontal standard triangles, and the multiple second triangles The area of ​​the target user's eye movement is determined based on the coordinates of the center points of the two eyes of the target user in the standard continuous image; based on the multiple first area differences and the multiple second area differences, multiple area differences representing the matching results of the change trend of the eye movement and the standard change trend are determined; based on the multiple area differences, the number of the multiple area differences and the mean of the multiple area differences, a standard deviation value representing the smoothness of the eye movement in the continuous multi-frame image is determined; if there is an area difference value exceeding a predetermined difference threshold value among the multiple area differences and the standard deviation value exceeds the predetermined standard deviation threshold value, the number of area differences exceeding the predetermined difference threshold value is determined as the second hang-up number of the target user; A weighted analysis is performed on the obtained smoothness analysis result and the stability analysis result to obtain the hang-up detection result of the target user.

2. The method according to claim 1, characterized in that: The step of selecting images whose stability analysis results do not satisfy the first hang-up condition from the continuous multiple frames of images, and splicing the selected images in chronological order to obtain multiple spliced ​​images includes: Identify at least one group of continuous candidate images whose stability values ​​exceed a predetermined stability threshold from the continuous multiple frames of images, and determine the number of continuous frames corresponding to the at least one group of continuous candidate images; Determine the continuous candidate images whose continuous frame number is greater than or equal to a predetermined frame number threshold in the at least one group of continuous candidate images as target continuous images with hang-up behavior, and determine the first hang-up number of the target user based on the number of groups of the target continuous images; The remaining images after removing the target continuous image from the continuous multiple frame images are spliced ​​in chronological order to obtain the multiple frame spliced ​​image.

3. The method according to claim 2, characterized in that The weighted analysis of the obtained smoothness analysis result, the stability analysis result and the matching result to obtain the hang-up detection result of the target user includes: Obtaining the sum of the first hanging-up times and the second hanging-up times corresponding to the multiple groups of continuous multi-frame images included in the complete learning process of the user, and the third hanging-up times detected through the user operation behavior; A weighted analysis is performed on the total of the first hang-up times, the second hang-up times, and the third hang-up times to obtain a hang-up detection result of the target user in the complete learning process.

4. A hang-up detection device, characterized in that: include: The recognition module is configured to recognize each frame of the collected continuous multiple frames of images to obtain the center point coordinates of the target user's eyes in each frame of the image, wherein the center point coordinates are the center point coordinates corresponding to the left eye and the right eye of the target user respectively; The analysis module is configured to analyze the stability of binocular motion in each frame of image based on the center point coordinates, select images whose stability analysis results do not meet the first hang-up condition from the continuous multiple frames of image, and splice the selected images in chronological order to obtain multiple spliced ​​images, including: determining a first binocular motion distance between each frame of image and its corresponding previous frame of image, and a second binocular motion distance between each frame of image and its corresponding next frame of image based on the horizontal coordinates, vertical coordinates and time coordinates in the center point coordinates; determining an instantaneous speed of binocular motion corresponding to each frame of image based on the difference between the first binocular motion distance and the second binocular motion distance and the time difference between each frame of image and its corresponding previous frame of image, wherein the instantaneous speed is used to represent the speed change of each frame of image compared with the corresponding previous frame of image; determining an instantaneous speed variance corresponding to each frame of image based on the instantaneous speed of binocular motion corresponding to each frame of image and the instantaneous speed mean, and analyzing the stability of binocular motion in each frame of image through the instantaneous speed variance to obtain a stability value corresponding to each frame of image; The analysis module is further configured to analyze the changing trend of binocular motion corresponding to the multi-frame spliced ​​image based on the center point coordinates, match the changing trend of binocular motion with the standard changing trend corresponding to the target user, and analyze the smoothness of binocular motion based on the obtained matching result, including: constructing a first coordinate graph of the change of the horizontal coordinates of the binoculars with the time coordinates and a second coordinate graph of the change of the vertical coordinates with the time coordinates based on the center point coordinates; in the first coordinate graph, forming a first triangle with every three consecutive coordinates, and determining the changing trend of the horizontal coordinates in the changing trend of the binocular motion through the areas of the multiple first triangles formed in the first coordinate graph, and forming a second triangle with every three consecutive coordinates in the second coordinate graph, and determining the changing trend of the vertical coordinates in the changing trend of the binocular motion through the areas of the multiple second triangles formed in the second coordinate graph; determining the difference between the areas of the multiple first triangles and the multiple first areas of the horizontal standard triangles, and the multiple The area of ​​the second triangle is determined by a plurality of second area differences between the area of ​​the second triangle and the longitudinal standard triangle, the transverse standard triangle is used to determine the transverse standard change trend in the standard change trend, and the longitudinal standard triangle is used to determine the longitudinal standard change trend in the standard change trend, the transverse standard change trend and the longitudinal standard change trend are determined based on the center point coordinates of the eyes of the target user in the standard continuous image; based on the plurality of first area differences and the plurality of second area differences, a plurality of area differences representing the matching result of the change trend of the binocular movement and the standard change trend are determined; based on the plurality of area differences, the number of the plurality of area differences and the mean of the plurality of area differences, a standard deviation value representing the smoothness of the binocular movement in the continuous multi-frame images is determined; if there is an area difference value exceeding a predetermined difference threshold value among the plurality of area differences and the standard deviation value exceeds the predetermined standard deviation threshold value, the number of the area difference values ​​exceeding the predetermined difference threshold value is determined as the second hang-up number of the target user; The analysis module is further configured to perform weighted analysis on the obtained smoothness analysis result and the stability analysis result to obtain the hang-up detection result of the target user.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

Images