Target behavior detection method, device and electronic equipment

Abstract

The present invention belongs to the field of computer information processing, and provides a target behavior detection method, device, electronic device and computer-readable medium, the method comprising: acquiring video data of a target object in real time; extracting a target video frame from the video data; generating a three-dimensional model of the target object based on the target video frame, a target object detection model and a key point detection model; determining the posture fluctuation rate of the target object based on the three-dimensional model of the target object; and determining that the target object has the target behavior when the posture fluctuation rate is greater than a posture threshold. The present invention can detect the behavior of a user who is conducting real-time online learning through the network, and determine whether the user has the target behavior, so as to evaluate the user's learning effect and learning situation, and when the user has bad behavior, remind the user to help improve the learning effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer information processing, and is particularly applicable to the field of video recognition, and more specifically relates to a target behavior detection method, device, electronic device and computer-readable medium. Background Art

[0002] With the development of technology and the needs of daily life, more and more users have shifted their focus to the Internet, and various network applications have emerged. There are also more and more video learning applications that can assist users in learning. Among the applications that assist users in learning, there is a type that assists users in learning by playing video courses. In this type of application, the content of the teacher's open class is played in the video, and users watch and learn through their respective mobile terminals.

[0003] This type of assisted learning application has greatly improved the user's learning convenience and efficiency. However, since there are a large number of users studying together at the same time and each user is in a different position, while online learning methods provide convenience, they cannot supervise the user's learning situation like learning in school. In this case, the user's learning efficiency and concentration can only be achieved through user consciousness or parental supervision. In these two methods, for younger users, when relying on user consciousness to achieve self-supervision, unsupervised online learning often does not achieve satisfactory results; when relying on parental supervision, it often takes up a lot of parents' time and energy, and user satisfaction is low.

[0004] Therefore, a new target behavior detection method, device, electronic device and computer-readable medium are needed.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The present invention aims to solve the drawbacks of online learning methods in the prior art. As mentioned above, when conducting online learning, for younger users, when relying on users to consciously realize self-supervision, unsupervised online learning often does not achieve satisfactory results; when relying on parents for supervision, it often takes up a lot of parents' time and energy, and user satisfaction is low. The present invention obtains the video data of the target object in real time to determine whether the target object has the target behavior. It can detect the behavior of users who are conducting real-time online learning through the network, and determine whether the user has the target behavior, so as to evaluate the user's learning effect and learning situation, and when the user has bad behavior, remind the user to help improve the learning effect.

[0007] According to one aspect of the present invention, a target behavior detection method is proposed, which includes: acquiring video data of a target object in real time; extracting a target video frame from the video data; generating a three-dimensional model of the target object based on the target video frame, a target object detection model and a key point detection model; determining a posture fluctuation rate of the target object based on the three-dimensional model of the target object; and determining that the target object has the target behavior when the posture fluctuation rate is greater than a posture threshold.

[0008] According to a preferred embodiment of the present invention, it also includes: training a face detection model based on a video frame with a target object to generate the target object detection model; training a face key point detection model based on a video frame with key point annotations to generate the key point detection model.

[0009] According to a preferred embodiment of the present invention, the method further includes: generating a plurality of posture fluctuation rates through a plurality of video data with labels; comparing the plurality of posture fluctuation rates with a plurality of thresholds to determine the posture threshold value from a collection of the plurality of thresholds.

[0010] According to a preferred embodiment of the present invention, acquiring video data of a target object in real time includes: acquiring video data of a head of the target object in real time by a camera device at a mobile terminal.

[0011] According to a preferred embodiment of the present invention, extracting the target video frame from the video data includes: extracting the target video frame from the video data at a preset time point.

[0012] According to a preferred embodiment of the present invention, the target video frame is extracted from the video data at a preset time point, comprising: extracting a current video frame corresponding to the current time point from the video data at the preset time point; extracting a front video frame corresponding to a time point before the current time point; extracting a rear video frame corresponding to a time point after the current time point; and generating the target video frame through the front video frame, the current video frame, and the rear video frame.

[0013] According to a preferred embodiment of the present invention, a three-dimensional model of the target object is generated based on the target video frame, the target object detection model and the key point detection model, including: inputting the target video frame into the target object detection model to generate a face detection frame and head posture angle; inputting the target video frame into the key point detection model to generate face key points; and constructing a three-dimensional model of the target object's head based on the face detection frame, head posture angles and face key points.

[0014] According to a preferred embodiment of the present invention, the posture fluctuation rate of the target object is determined based on the three-dimensional model of the target object, including: obtaining front angle parameters of the front video frame, current angle parameters of the current video frame, and rear angle parameters of the rear video frame based on the front video frame, the current video frame, the rear video frame, and the three-dimensional model of the head; and determining the posture fluctuation rate of the three-dimensional model of the head based on the front angle parameters, the current angle parameters, and the current angle parameters.

[0015] According to a preferred embodiment of the present invention, multiple posture fluctuation rates are generated through multiple labeled video data, including: generating test data from multiple labeled video data; inputting the test data into the target object detection model and the key point detection model to generate multiple three-dimensional models of target objects; and generating multiple posture fluctuation rates of the target objects based on the three-dimensional models of the multiple target objects.

[0016] According to a preferred embodiment of the present invention, the multiple posture fluctuation rates are compared with multiple thresholds to determine the posture threshold from the set of the multiple thresholds, including: comparing the multiple posture fluctuation rates with each threshold respectively, and determining whether the target object has the target behavior according to the comparison result; comparing the comparison result with the label of the target object to determine the accuracy corresponding to each threshold; and taking the threshold with the largest accuracy as the posture threshold.

[0017] The second aspect of the present invention proposes a target behavior detection device, which includes: a real-time video module, used to obtain video data of a target object in real time; a video frame module, used to extract target video frames from the video data; a three-dimensional model module, used to generate a three-dimensional model of the target object based on the target video frame, a target object detection model and a key point detection model; a fluctuation rate module, used to determine the posture fluctuation rate of the target object based on the three-dimensional model of the target object; and a judgment module, used to determine that the target object has the target behavior when the posture fluctuation rate is greater than a posture threshold.

[0018] According to a preferred embodiment of the present invention, it also includes: a model generation module, which is used to train a face detection model based on a video frame with a target object to generate the target object detection model; and is also used to train a face key point detection model based on a video frame with key point annotations to generate the key point detection model.

[0019] According to a preferred embodiment of the present invention, it is characterized in that it also includes: a threshold generation module, used to generate multiple posture fluctuations through multiple labeled video data; compare the multiple posture fluctuations with multiple thresholds to determine the posture threshold from the set of the multiple thresholds.

[0020] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the described method.

[0021] A fourth aspect of the present invention further proposes a computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method described is implemented.

[0022] According to the target behavior detection method of the present invention, by extracting the target video frame from the video data of the target object acquired in real time; and generating a three-dimensional model of the target object based on the target video frame, the target object detection model and the key point detection model; and determining the posture fluctuation rate of the target object based on the three-dimensional model of the target object; and determining whether the target object has the target behavior based on the posture fluctuation rate, the behavior of the user who is conducting real-time online learning through the network can be detected, and it can be determined whether the user has the target behavior, so that the learning effect and learning situation of the user can be evaluated, and when the user has bad behavior, reminders can be given to assist the user in improving the learning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a system block diagram of a target behavior detection method and device according to an embodiment of the present invention.

[0024] Figure 2 It is a flow chart of a target behavior detection method according to an embodiment of the present invention.

[0025] Figure 3 It is a flow chart of a target behavior detection method according to an embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of a target behavior detection method according to an embodiment of the present invention.

[0027] Figure 5 It is a flow chart of a target behavior detection method according to an embodiment of the present invention.

[0028] Figure 6 It is a block diagram of a target behavior detection device according to an embodiment of the present invention.

[0029] Figure 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0032] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0033] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0034] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0036] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0037] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0038] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0039] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0040] Figure 1 It is a system block diagram showing a method and device for detecting target behavior according to an exemplary embodiment.

[0041] like Figure 1 As shown, the system architecture 10 may include user terminal devices 101, 102, 103, a network 104, a server 105, and a teacher terminal device 106. The network 104 is used to provide a medium for communication links between the user terminal devices 101, 102, 103 and the server 105 and the teacher terminal device 106. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0042] Student users can use user terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Teacher users can use teacher terminal device 106 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on user terminal devices 101, 102, 103 and teacher terminal device 106, such as video learning applications, instant messaging tools, email clients, social platform software, etc.

[0043] The user terminal devices 101, 102, 103 and the teacher terminal device 106 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0044] The server 105 may be a server that provides various services, such as a background management server that provides support for video learning websites browsed by users using the user terminal devices 101, 102, and 103. The background management server may analyze and process the received real-time user videos, and feed back the processing results (such as whether the user has the target behavior) to the administrator and / or teacher user.

[0045] It is worth mentioning that in this solution, the teacher terminal device 106 is not necessary. It is understandable that in video learning applications, real-time live broadcast of teacher lectures can be used to assist users in learning, and other teaching content pre-stored in the server can also be used to assist users in learning. The present invention is not limited to this.

[0046] In some application scenarios, user terminal devices 101, 102, 103 can, for example, obtain video data of the target object in real time; user terminal devices 101, 102, 103 can, for example, extract target video frames from the video data; user terminal devices 101, 102, 103 can, for example, generate a three-dimensional model of the target object based on the target video frame, the target object detection model and the key point detection model; user terminal devices 101, 102, 103 can, for example, determine the posture fluctuation rate of the target object based on the three-dimensional model of the target object; user terminal devices 101, 102, 103 can, for example, determine that the target object has the target behavior when the posture fluctuation rate is greater than a posture threshold.

[0047] In other application scenarios, the user terminal devices 101, 102, 103 can, for example, obtain video data of the target object in real time; the user terminal devices 101, 102, 103 or the server 105 can, for example, extract target video frames from the video data; the server 105 can, for example, generate a three-dimensional model of the target object based on the target video frames, the target object detection model and the key point detection model; the server 105 can, for example, determine the posture fluctuation rate of the target object based on the three-dimensional model of the target object; the server 105 can, for example, determine that the target object has the target behavior when the posture fluctuation rate is greater than a posture threshold.

[0048] The server 105 may also, for example, train a face detection model based on a video frame with a target object to generate the target object detection model; the server 105 may also, for example, train a face key point detection model based on a video frame with key point annotations to generate the key point detection model.

[0049] The server 105 may also generate a plurality of posture fluctuation rates, for example, from a plurality of video data with labels; the server 105 may also compare the plurality of posture fluctuation rates with a plurality of thresholds to determine the posture threshold value from the plurality of thresholds.

[0050] The server 105 can be a physical server, or can be composed of multiple servers, for example. A part of the server 105 can be used as a model training system in the present invention, for example, to train a face detection model based on a video frame with a target object to generate the target object detection model; to train a face key point detection model based on a video frame with key point annotations to generate the key point detection model; and a part of the server 105 can also be used as a threshold generation system in the present invention, for example, to generate multiple posture fluctuations through multiple labeled video data; and to compare the multiple posture fluctuations with multiple thresholds to determine the posture threshold from the set of the multiple thresholds.

[0051] It should be noted that the target behavior detection method provided in the embodiment of the present invention can be executed by the server 105 or the user terminal devices 101, 102, 103, and accordingly, the target behavior detection device can be set in the server 105 or the user terminal devices 101, 102, 103. The application end provided to the user for browsing the video learning application is generally located in the user terminal devices 101, 102, 103.

[0052] Figure 2 2 is a flow chart showing a target behavior detection method according to an exemplary embodiment. The target behavior detection method 20 at least includes steps S202 to S210.

[0053] like Figure 2 As shown, in S202, video data of the target object is obtained in real time. For example, the video data of the head of the target object can be obtained in real time through a camera device on a mobile terminal. In the application scenario described above, a student user can watch a video course through a video learning application set on a mobile terminal. In this process, the camera on the mobile terminal obtains the video data of the student user in real time.

[0054] In S204, the target video frame is extracted from the video data; after the camera on the mobile terminal obtains the video data of the student user, the video data can be preliminarily processed locally, and the video data can also be directly forwarded to a remote server for processing, but the present invention is not limited to this.

[0055] In one embodiment, the mobile terminal or the remote server may extract the target video frame from the video data at a preset time point, for example. The preset time may be 60 seconds or 100 seconds or other time.

[0056] In one embodiment, extracting the target video frame from the video data at a preset time point includes: extracting a current video frame corresponding to the current time point from the video data at the preset time point; extracting a previous video frame corresponding to a time point before the current time point; extracting a subsequent video frame corresponding to a time point after the current time point; and generating the target video frame through the previous video frame, the current video frame, and the subsequent video frame.

[0057] In one embodiment, when the preset time interval is a cycle of 60 seconds, the image corresponding to the first second of the initial cycle can be counted as the first frame, and so on. The image of the 60th frame is extracted as the current frame data, and the image of the 59th frame is also extracted as the previous video frame, and the image of the 61st frame (the first frame of the next cycle) is extracted as the next video frame.

[0058] In S206, a three-dimensional model of the target object is generated based on the target video frame, the target object detection model and the key point detection model, including: inputting the target video frame into the target object detection model to generate a face detection frame and a head posture angle; inputting the target video frame into the key point detection model to generate face key points; and constructing a three-dimensional model of the head of the target object based on the face detection frame, the head posture angle and the face key points.

[0059] After the target video frame is input into the target object detection model, a face detection frame and a head posture angle can be generated. After the target video frame is input into the key point detection model, a face key point can be generated. Based on the face key point, an auxiliary head posture angle can be generated. The head posture angle and the auxiliary head posture angle are jointly judged to determine the head posture angle used to construct the three-dimensional model.

[0060] A 3D model of the target object is generated according to the head posture angle and the standard user 3D model initial model.

[0061] In S208, the posture fluctuation rate of the target object is determined based on the three-dimensional model of the target object. A front angle parameter of the front video frame, a current angle parameter of the current video frame, and a rear angle parameter of the rear video frame are respectively obtained based on the front video frame, the current video frame, the rear video frame, and the three-dimensional model of the head; and the posture fluctuation rate of the three-dimensional model of the head is determined based on the front angle parameter, the current angle parameter, and the current angle parameter.

[0062] The relevant contents of generating a three-dimensional model of the target object based on the target video frame, the target object detection model and the key point detection model and determining the posture fluctuation rate of the target object will be described in Figure 3 The corresponding embodiments are described in detail.

[0063] In S210, when the posture fluctuation rate is greater than the posture threshold, it is determined that the target object has the target behavior. The target behavior may be inattention behavior. If the user's head posture fluctuates greatly over a period of time, it means that the user is not focused enough. Or other behaviors that affect the user's learning. The target behavior may include multiple types. It is easy for those skilled in the art to think that different target behaviors can be distinguished by different posture thresholds.

[0064] According to the target behavior detection method of the present invention, by extracting the target video frame from the video data of the target object acquired in real time; and generating a three-dimensional model of the target object based on the target video frame, the target object detection model and the key point detection model; and determining the posture fluctuation rate of the target object based on the three-dimensional model of the target object; and determining whether the target object has the target behavior based on the posture fluctuation rate, the behavior of the user who is conducting real-time online learning through the network can be detected, and it can be determined whether the user has the target behavior, so that the learning effect and learning situation of the user can be evaluated, and when the user has bad behavior, reminders can be given to assist the user in improving the learning effect.

[0065] It should be clearly understood that the present invention describes how to form and use specific examples, but the principles of the present invention are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present invention, these principles can be applied to many other embodiments.

[0066] In one embodiment, it also includes: training a face detection model based on a video frame with a target object to generate the target object detection model; training a face key point detection model based on a video frame with key point annotations to generate the key point detection model.

[0067] Facial Recognition is a process of obtaining a user's facial image through a video capture device, and then using a face detection model to perform calculations and analysis on the position, shape and angle of the facial features, and then comparing it with existing templates in its own database to determine the user's true identity.

[0068] In the present invention, in order to make the subsequent recognition of the user's posture more accurate, when training the face detection model, the input is a video frame with different faces. First, the local area in the image needs to be defined, and different video frames need to be labeled with the posture parameters corresponding to the face. Then, the local area features of the face are extracted, and the face image vector is mapped to the face feature vector according to the transformation matrix obtained after sample training; then, local feature selection (optional); finally, classification is performed. The classifier can be in the form of a combined classifier, each local feature corresponds to a classifier, and then the final recognition result can be obtained by voting or linear weighting. In the present invention, the final recognition result includes a face detection frame and a head posture angle. The face detection model can include 4 types: a recognition algorithm based on face feature points, a recognition algorithm based on the entire face image, a recognition algorithm based on a template, and an algorithm for recognition using a neural network. The target object detection model in the present invention can be the result of training with one or more of the above combined face detection models.

[0069] Face key point detection is also called face key point detection, positioning or face alignment, which means locating the key areas of a face, including eyebrows, eyes, nose, mouth, facial contour, etc., given a face image. Similar to the face recognition algorithm mentioned above, the face key point detection algorithm also needs to first perform model training, and finally obtain a fully trained model as the key point detection model in the present invention.

[0070] Figure 3 is a flow chart of a target behavior detection method according to another exemplary embodiment. Figure 3 The process 30 shown is for Figure 2 The process shown includes a detailed description of "generating a three-dimensional model of the target object based on the target video frame, the target object detection model and the key point detection model; and determining the posture fluctuation rate of the target object based on the three-dimensional model of the target object."

[0071] like Figure 3 As shown, in S302, the target video frame is input into the target object detection model to generate a face detection frame and a head posture angle. The previous video frame, current video frame, and subsequent video frame data of the current user can be input into the target object detection model, and the face detection frame and the head posture angle of the user are output after calculation.

[0072] In S304, the target video frame is input into the key point detection model to generate facial key points. The previous video frame, current video frame, and next video frame data of the current user can be input into the key point detection model, and the key points of the user's face can be input through calculation.

[0073] In S306, a three-dimensional model of the target object's head is constructed based on the face detection frame, head posture angle, and face key points. The three-dimensional model of the head posture can be generated by locating the two-dimensional key points of the face. More specifically, the head posture angle can be obtained through a facial image, that is, the three Euler angles of pitch, yaw, and roll are calculated. The scientific names are pitch angle, yaw angle, and roll angle, which can be understood as raising the head, shaking the head, and turning the head. Pitch represents the angle of looking up or down, roll is the rotation angle of the horizontal plane, and yaw is the angle of turning the head left and right. For details, please refer to Figure 4 .

[0074] In S308, the front angle parameters of the front video frame, the current angle parameters of the current video frame, and the rear angle parameters of the rear video frame are respectively obtained based on the front video frame, the current video frame, the rear video frame and the three-dimensional model of the head.

[0075] In S310, the posture fluctuation rate of the three-dimensional head model is determined based on the previous angle parameter, the current angle parameter, and the current angle parameter. According to the changes of various angles of the user's head at three time points, the user's head posture change rate can be calculated.

[0076] Figure 5 is a flow chart of a target behavior detection method according to another exemplary embodiment. Figure 5 The process 50 shown is a detailed description of “determining a posture threshold value”.

[0077] like Figure 5 As shown, in S502, test data is generated by multiple video frames with labels. The video frames with labels may be video frames obtained by multiple other users when performing video learning. As described above, the video frames may specifically include a group of video frames consisting of a previous video frame, a current video frame, and a next video frame. The labels of the video frames may include two types: the presence of the target behavior and the absence of the target behavior.

[0078] In S504, the test data is input into the target object detection model and the key point detection model to generate a plurality of three-dimensional models of target objects. After each set of video frames is input into the target object detection model and the key point detection model, the three-dimensional model of the head of a user is obtained corresponding to each set of video frames by calculating the method described above.

[0079] In S506, multiple posture fluctuation rates of the target objects are generated based on the three-dimensional models of the multiple target objects. Multiple posture fluctuation rates are calculated based on the three-dimensional head models of multiple users.

[0080] In S508, the multiple posture fluctuation rates are compared with each threshold value respectively, and it is determined whether the target object has the target behavior according to the comparison result.

[0081] More specifically, first, a first threshold value among multiple threshold values ​​is obtained, and multiple posture fluctuation rates are compared with the first threshold value respectively to determine whether the users corresponding to the multiple posture fluctuation rates have the target behavior. Then, a second threshold value among multiple threshold values ​​is obtained, and multiple posture fluctuation rates are compared with the second threshold value respectively to determine whether the users corresponding to the multiple posture fluctuation rates have the target behavior. And so on, until multiple fluctuation rates are compared with each threshold value respectively.

[0082] In S510, the comparison result is compared with the label of the target object to determine the accuracy corresponding to each threshold. The conclusion of whether the user has the target behavior obtained by comparison is compared with the label of the original video frame. When the comparison conclusion is consistent with the attribute in the label, it is considered that the comparison is correct, otherwise it is determined that the comparison fails.

[0083] In S512, the threshold with the highest accuracy is used as the posture threshold. The accuracy corresponding to each threshold is calculated, and after traversing the comparison of all thresholds, the threshold with the highest score on the test set is used as the final threshold. It should be noted that the threshold in the present invention can be a certain value or a data range.

[0084] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present invention are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0085] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0086] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0087] Figure 6 FIG. 1 is a block diagram of a target behavior detection device according to an exemplary embodiment. Figure 6 As shown, the target behavior detection device 60 includes: The target behavior detection device 60 may also include: a real-time video module 602, a video frame module 604, a three-dimensional model module 606, a volatility module 608, a judgment module 610, a model generation module 612, and a threshold generation module 614.

[0088] The real-time video module 602 is used to obtain the video data of the target object in real time; the real-time video module 602 can be set on the mobile terminal, and then the video data of the head of the target object is obtained in real time through the camera device of the mobile terminal.

[0089] The video frame module 604 is used to extract the target video frame from the video data; the video frame module 604 can be set in a mobile terminal or a remote server, and the video frame module 604 is also used to extract the target video frame from the video data at a preset time point.

[0090] The three-dimensional model module 606 is used to generate a three-dimensional model of the target object based on the target video frame, the target object detection model and the key point detection model; the three-dimensional model module 606 can also be used to input the target video frame into the target object detection model to generate a face detection frame and head posture angle; input the target video frame into the key point detection model to generate face key points; and construct a three-dimensional model of the target object's head based on the face detection frame, head posture angle and face key points.

[0091] The fluctuation rate module 608 is used to determine the posture fluctuation rate of the target object based on the three-dimensional model of the target object; more specifically, for example, based on the front video frame, the current video frame, the rear video frame and the three-dimensional model of the head, the front angle parameters of the front video frame, the current angle parameters of the current video frame, and the rear angle parameters of the rear video frame can be respectively obtained; based on the front angle parameters, the current angle parameters, and the current angle parameters, the posture fluctuation rate of the three-dimensional model of the head is determined.

[0092] The judgment module 610 is used to determine that the target object has the target behavior when the posture fluctuation rate is greater than the posture threshold.

[0093] The model generation module 612 is used to train the face detection model based on the video frames with the target object to generate the target object detection model; it is also used to train the face key point detection model based on the video frames with key point annotations to generate the key point detection model.

[0094] The threshold generation module 614 is used to generate a plurality of posture fluctuation rates through a plurality of video data with labels; and compare the plurality of posture fluctuation rates with a plurality of thresholds to determine the posture threshold value from the plurality of thresholds.

[0095] According to the target behavior detection device of the present invention, video data of the target object is acquired in real time; a target video frame is extracted from the video data; a three-dimensional model of the target object is generated based on the target video frame, a target object detection model and a key point detection model; a posture fluctuation rate of the target object is determined based on the three-dimensional model of the target object; when the posture fluctuation rate is greater than a posture threshold, the target object is determined to have the target behavior. The device can detect the behavior of a user who is conducting real-time online learning through the network, and determine whether the user has the target behavior, so as to evaluate the user's learning effect and learning status, and when the user has bad behavior, a reminder is given to assist the user in improving the learning effect.

[0096] Figure 7 It is a structural schematic diagram of an electronic device of an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes a vehicle intelligent power-assisted propulsion method based on rotation angle monitoring.

[0097] like Figure 7 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0098] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0099] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0100] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0101] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0102] Figure 8 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 8 As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the vehicle intelligent power-assisted pushing method based on rotation angle monitoring of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate or transmit a program for use by or in combination with an instruction execution system, device or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0103] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0104] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0105] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target behavior detection method, characterized in that: include: Training a face detection model based on a video frame with a target object to generate a target object detection model; The face key point detection model is trained based on the video frames with key point annotations to generate a key point detection model; Generate multiple posture fluctuations through multiple labeled video data; comparing the plurality of posture fluctuation rates with a plurality of thresholds to determine a posture threshold from a set of the plurality of thresholds; Acquire video data of target objects in real time; Extracting a current video frame corresponding to the current time point from the video data at a preset time point; Extracting a previous video frame corresponding to a time point before the current time point; Extracting a subsequent video frame corresponding to a time point subsequent to the current time point; Generate a target video frame through the previous video frame, the current video frame, and the next video frame; Inputting the target video frame into the target object detection model to generate a face detection frame and a head posture angle; Inputting the target video frame into the key point detection model to generate facial key points; Constructing a three-dimensional model of the target object's head based on the face detection frame, head posture angle, and face key points; Based on the front video frame, the current video frame, the rear video frame and the three-dimensional model of the head, respectively obtain a front angle parameter of the front video frame, a current angle parameter of the current video frame, and a rear angle parameter of the rear video frame; Determine the posture fluctuation rate of the three-dimensional head model based on the front angle parameter, the current angle parameter, and the current angle parameter; When the posture fluctuation rate is greater than the posture threshold, it is determined that the target object has a target behavior.

2. The target behavior detection method according to claim 1, characterized in that: Acquire video data of the target object in real time, including: The video data of the head of the target object is acquired in real time through the camera device of the mobile terminal.

3. The target behavior detection method according to claim 1, characterized in that: Generate multiple pose fluctuations through multiple labeled video data, including: Generate test data from multiple labeled video frames; Inputting the test data into the target object detection model and the key point detection model to generate three-dimensional models of multiple target objects; A plurality of posture fluctuation ratios of the target objects are generated based on the three-dimensional models of the plurality of target objects.

4. The target behavior detection method according to claim 1, characterized in that: Comparing the plurality of posture fluctuation rates with a plurality of thresholds to determine the posture threshold from the plurality of thresholds comprises: Comparing the multiple posture fluctuation rates with each threshold value respectively, and determining whether the target object has the target behavior according to the comparison result; Comparing the comparison result with the label of the target object to determine the accuracy rate corresponding to each threshold; The threshold with the highest accuracy is used as the posture threshold.

5. A target behavior detection device, characterized in that: include: A model generation module is used to train a face detection model based on a video frame with a target object to generate a target object detection model; and to train a face key point detection model based on a video frame with key point annotations to generate a key point detection model; A threshold generation module, configured to generate a plurality of posture fluctuation rates through a plurality of video data with labels; compare the plurality of posture fluctuation rates with a plurality of thresholds to determine a posture threshold value from a set of the plurality of thresholds; Real-time video module, used to obtain video data of target objects in real time; A video frame module is used to extract a current video frame corresponding to a current time point from the video data at a preset time point; extract a previous video frame corresponding to a time point before the current time point; extract a subsequent video frame corresponding to a time point after the current time point; and generate a target video frame through the previous video frame, the current video frame, and the subsequent video frame; A three-dimensional model module is used to input the target video frame into the target object detection model to generate a face detection frame and a head posture angle; input the target video frame into the key point detection model to generate face key points; and construct a three-dimensional model of the head of the target object based on the face detection frame, the head posture angle, and the face key points; A fluctuation rate module is used to obtain the front angle parameter of the front video frame, the current angle parameter of the current video frame, and the rear angle parameter of the rear video frame based on the front video frame, the current video frame, the rear video frame and the three-dimensional model of the head; and determine the posture fluctuation rate of the three-dimensional model of the head based on the front angle parameter, the current angle parameter and the current angle parameter; The judgment module is used to determine that the target object has the target behavior when the posture fluctuation rate is greater than a posture threshold.

6. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer program is executed by the processor, the processor performs the method according to any one of claims 1 to 4.

7. A computer-readable medium storing a computer-executable program, characterized in that: When the computer executable program is executed, the method according to any one of claims 1 to 4 is implemented.

Images