A method, device, equipment and readable medium for identifying live broadcast behavior

Abstract

The embodiments of the present disclosure disclose a method, device, equipment and readable medium for identifying live broadcast behaviors. The method includes: constructing facial action features in the current live broadcast frame based on the target key points detected in the current live broadcast frame; identifying the teaching behaviors in the current live broadcast frame from the constructed teaching behavior library based on the position deviations between the target key points and the facial action features. The technical solution provided by the embodiments of the present disclosure realizes real-time recognition of live broadcast behaviors and improves the timeliness of live broadcast behavior recognition; at the same time, various teaching behaviors that do not meet the requirements of teaching specifications can be flexibly configured in the teaching behavior library to realize customized recognition of live broadcast behaviors and improve the flexibility and comprehensiveness of live broadcast behavior recognition.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video processing technology, and in particular to a live broadcast behavior recognition method, device, equipment and readable medium. Background Art

[0002] With the rapid development of artificial intelligence technology in the field of educational informatization, more and more teachers will use remote live broadcasts to teach students. Therefore, it is necessary to conduct behavioral recognition of online teaching live broadcasts to prevent the spread of some bad behavior information to students.

[0003] At present, the recognition of live broadcast behavior is mainly divided into two types: after-class manual recognition and real-time recognition. For after-class manual recognition, the live broadcast video of the class needs to be sent to the manual platform for behavior recognition after the teacher finishes the class. At this time, the timeliness of the recognition of live broadcast behavior cannot be guaranteed, which affects the students' remote class experience. For real-time behavior recognition, the network recognition model trained by the existing deep learning method is usually used to recognize the behavior in the teaching live broadcast in real time. However, since the behavioral norms required for teaching live broadcasts are different from those required for other live broadcasts, in addition to the need to identify whether the live broadcast content contains bad information, it is also necessary to identify whether the teacher's teaching posture conforms to the teaching norms. Therefore, the existing network recognition model cannot fully recognize various behaviors in the teaching live broadcast, which reduces the accuracy of live broadcast behavior recognition. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a method, device, equipment and readable medium for identifying live broadcast behaviors. On the basis of ensuring real-time identification of live broadcast behaviors, a teaching behavior library is used to realize customized configuration of specific live broadcast behaviors, thereby improving the comprehensiveness and timeliness of live broadcast behavior identification.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for identifying live broadcasting behavior, the method comprising:

[0006] Constructing facial motion features in the current live broadcast frame according to the target key points detected in the current live broadcast frame;

[0007] Based on the position deviation between each of the target key points and the facial movement features, the teaching behavior in the current live broadcast frame is identified from the constructed teaching behavior library.

[0008] In a second aspect, an embodiment of the present disclosure provides a live broadcast behavior recognition device, the device comprising:

[0009] A face action construction module, used to construct face action features in the current live frame according to the target key points detected in the current live frame;

[0010] The teaching behavior recognition module is used to identify the teaching behavior in the current live broadcast frame from the constructed teaching behavior library based on the position deviation between each of the target key points and the facial movement features.

[0011] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the device comprising:

[0012] one or more processors;

[0013] A memory for storing one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the live broadcast behavior identification method as described in any embodiment of the present disclosure.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a readable medium on which a computer program is stored, and when the program is executed by a processor, the live broadcast behavior recognition method as described in any embodiment of the present disclosure is implemented.

[0016] The embodiments of the present disclosure provide a method, device, equipment and readable medium for identifying live broadcast behaviors. A plurality of specific teaching behaviors are pre-customized in a constructed teaching behavior library, so as to construct facial motion features in the current live broadcast frame according to the target key points detected in the current live broadcast frame, and then the position deviation between the target key points and the facial motion features are used to identify the teaching behaviors in the current live broadcast frame from the constructed teaching behavior library, thereby realizing real-time recognition of live broadcast behaviors and improving the timeliness of recognition of live broadcast behaviors. At the same time, various teaching behaviors that do not meet the requirements of teaching specifications can be flexibly configured in the teaching behavior library, so as to realize customized recognition of live broadcast behaviors and improve the flexibility and comprehensiveness of recognition of live broadcast behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 A flow chart of a live broadcast behavior recognition method provided by an embodiment of the present disclosure is shown;

[0019] Figure 2A A flowchart of another live broadcast behavior recognition method provided by an embodiment of the present disclosure is shown;

[0020] Figure 2B A schematic diagram showing the principle of a live broadcast behavior recognition process provided by an embodiment of the present disclosure is shown;

[0021] Figure 3 A schematic diagram of the structure of a live broadcast behavior recognition device provided by an embodiment of the present disclosure is shown;

[0022] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0025] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0026] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple parties in the embodiments of the present disclosure are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0029] Figure 1A flowchart of a live broadcast behavior recognition method provided by an embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to the case of recognizing user behavior in any live broadcast. The live broadcast behavior recognition method provided by an embodiment of the present disclosure can be executed by a live broadcast behavior recognition device provided by an embodiment of the present disclosure, which can be implemented by software and / or hardware and integrated in a device that executes the method.

[0030] Specifically, Figure 1 As shown, the live broadcast behavior recognition method provided in the embodiment of the present disclosure may include the following steps:

[0031] S110, constructing facial motion features in the current live broadcast frame according to the target key points detected in the current live broadcast frame.

[0032] Specifically, in a remote live broadcast scenario, in order to positively disseminate information to the live broadcast audience, it is necessary to identify the live broadcast behavior of the host during the live broadcast to determine whether the host (such as a teacher) has any irregular live broadcast behavior during the live broadcast, so as to prevent the teacher from disseminating some bad information to the student audience during the live broadcast; at this time, in order to ensure the timeliness of the live broadcast behavior identification, this embodiment will identify the picture content of each live broadcast frame in real time during the remote live broadcast, that is, it is necessary to continuously determine the current live broadcast frame from the teaching live broadcast, and then perform live broadcast behavior identification on the current live broadcast frame.

[0033] At the same time, in the remote live teaching scenario, since the live audience is facing the special group of students, the live broadcast behavior will have its own customized recognition needs, such as identifying whether there are teachers who are inappropriately dressed or have non-compliant postures (not showing the full face or obviously sideways) during the remote teaching live broadcast, etc., and there are certain differences in the unsupported live broadcast behaviors for different categories of student groups. However, it is certain that for any type of live audience, it is necessary to analyze the host's posture in the current live broadcast frame to determine whether there are non-compliant teaching behaviors during the live broadcast process. At this time, in order to fully cover various non-compliant live broadcast behaviors that may exist in the remote teaching live broadcast scenario, this embodiment will specifically set corresponding target key points for analyzing the behavioral characteristics in the current live broadcast frame. It should be noted that based on the need for accuracy in live broadcast behavior recognition, this embodiment will pre-screen and set the target key points using a large number of historical video frames, so that the selected target key points can not only ensure the accuracy of live broadcast behavior recognition, but also ensure the efficiency of key point extraction. Through a large number of experiments, it can be found that dividing the target key points into two categories, namely face key points and hand key points, can fully support the complete recognition of teaching behaviors in the current live frame and ensure the accuracy of live behavior recognition. In addition, face key points can include five key points of the teacher's ears, eyes and nose, and hand key points can include two key points of the wrists. A total of seven target key points are selected to support the complete recognition of teaching behaviors in the current live frame and meet the lightweight requirements for key point detection. There is no need to detect too many key points during live behavior recognition, which greatly reduces the workload of key point detection and improves the efficiency of live behavior recognition.

[0034] Specifically, after obtaining the current live broadcast frame from this teaching live broadcast, key point detection will be performed on the current live broadcast frame first to determine the coordinates of each pre-set target key point in the current live broadcast frame, and then the coordinate relationship between each target key point will be analyzed, such as eye distance, ear distance, ear-eye weighted midpoint, face width, face length, etc. Then, according to the biological characteristics of the face and the movement principle driven by face deformation, combined with the coordinate relationship between each target key point, the face action feature in the current live broadcast frame can be constructed. The face action feature can include face rectangular frame detection, face side face degree, face elliptical frame and face tilt degree, etc., so that the face action feature can be used to judge whether there is any irregular teaching behavior in the live broadcast process, such as the face not facing the camera or not fully displayed.

[0035] In addition, since there are situations in which the same picture is contained in continuous live frames during teaching live broadcast, if live broadcast behavior recognition is performed once for each live broadcast frame, the workload of live broadcast behavior recognition will be greatly increased. Therefore, in order to ensure the high efficiency of live broadcast behavior recognition, this embodiment will first extract frames for this teaching live broadcast according to the preset frame extraction frequency during the teaching live broadcast process, and thus continuously obtain the current live broadcast frame. In other words, this embodiment pre-sets a frame extraction service, in which the preset frame extraction frequency can be flexibly adjusted, and can usually be set to 1fps, 2fps or 5fps, etc. At this time, the preset frame extraction frequency is used to continuously extract frames for the teaching live broadcast during the teaching live broadcast process, so as to continuously obtain the current live broadcast frame that needs to be identified for live broadcast behavior, thereby avoiding the problem of excessive recognition workload caused by performing live broadcast behavior recognition on multiple live frames of the same picture, and improving the efficiency of live broadcast behavior recognition.

[0036] S120, based on the position deviation between each target key point and the facial movement feature, identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library.

[0037] In this embodiment, in order to ensure the comprehensiveness of the identification of irregular live broadcast behaviors, a teaching behavior library will be set up in advance. The teaching behavior library will store various irregular live broadcast behaviors that need to be identified during the teaching live broadcast process, such as incomplete facial display, not facing the camera, supporting the head with hands, eating, and making and receiving phone calls, etc. At this time, the various teaching behaviors that need to be identified in the teaching behavior library can be flexibly adjusted according to the specific business needs of live broadcast behavior identification, thereby realizing the iterative update of irregular teaching behaviors in the teaching behavior library. This embodiment can flexibly configure irregular teaching behaviors in various remote live broadcast scenarios through the teaching behavior library, and realize customized identification of live broadcast behaviors.

[0038] Specifically, after constructing the facial motion features in the current live frame, by analyzing the position deviations between the target key points in the current live frame, it is possible to estimate whether the positions of the different parts represented by the target key points overlap, such as the position overlap between fingers and human face parts such as mouth or ears, so as to determine whether the fingers are in contact with human face parts such as mouth or ears in the current live frame. At the same time, by analyzing the facial motion features in the current live frame, the current posture of the face in the current live frame can be determined. For example, the face tilt frame can be used to determine whether the face is tilted, and the face rectangular frame detection can be used to determine whether the face is fully displayed or facing the camera, or the face elliptical frame can be used to more accurately determine the face shape to determine whether the distance between the hand and the face is such that the hand is supporting the face. Then, based on the positional overlap between different parts represented by each target key point, determine whether it meets the posture requirements of each teaching behavior stored in the teaching behavior library. If it meets the posture requirements of a certain teaching behavior in the teaching behavior library, the teaching behavior is identified as the live broadcast behavior in the current live broadcast frame. At this time, by determining whether it meets the posture requirements of each teaching behavior stored in the teaching behavior library, various non-standard teaching behaviors in the current live broadcast frame can be identified from the teaching behavior library to ensure the comprehensiveness of live broadcast behavior recognition.

[0039] Exemplarily, in order to ensure the recognition accuracy of the teaching behavior in the current live frame, this embodiment will specifically analyze the positions of each target key point contained in the teaching behavior and / or the facial movement features involved in the teaching behavior for each teaching behavior stored in the teaching behavior library, and then set the corresponding behavior judgment condition for the teaching behavior according to the position deviation of each target key point contained in the teaching behavior and / or the current posture of the face represented by the facial movement features involved in the teaching behavior. For example, when any side of the face rectangular box is only a few pixels away from the edge of the picture, the distance between any side of the face rectangular box and the edge of the picture less than a preset pixel can be set as a behavior judgment condition for incomplete display of the face. At this time, based on the position deviation between each target key point and the facial movement features, the teaching behavior in the current live frame is identified from the constructed teaching behavior library, which can specifically include: determining the pre-set behavior judgment conditions for each teaching behavior in the teaching behavior library; judging whether the position deviation between each target key point and the facial movement features meet the behavior judgment conditions; searching the teaching behavior library for the target teaching behavior corresponding to the behavior judgment conditions that are met, and identifying the target teaching behavior as the teaching behavior in the current live frame.

[0040] That is to say, after determining the position deviation between each target key point and the facial movement feature in the current live broadcast frame, the behavior judgment conditions pre-set for each teaching behavior stored in the teaching behavior library will be searched out from the constructed teaching behavior library first, and then it will be judged whether the position deviation between each target key point and the facial movement feature meet the various behavior judgment conditions. If a certain behavior judgment condition is met, it means that the live broadcast behavior in the current live broadcast frame matches the teaching behavior corresponding to the behavior judgment condition. Therefore, the target teaching behavior corresponding to the behavior judgment condition in the teaching behavior library can be identified as the teaching behavior in the current live broadcast frame, thereby ensuring the accuracy of live broadcast behavior recognition.

[0041] In addition, in order to ensure effective control of live broadcast behavior identification, the present embodiment will also analyze the categories of the identified teaching behaviors and provide live broadcast prompts for the identified teaching behaviors to perform corresponding live broadcast behavior adjustment operations. For example, when it is identified that the teacher makes live broadcast behaviors such as supporting the head with hands or eating, a corresponding reminder message can be sent. When the teacher makes actions with a high violation level such as indecent behavior, the teaching live broadcast can be closed, etc. By way of example, the live broadcast behavior prompts in the present embodiment may include real-time reminders, real-time warnings, and after-class review, etc., which are not limited to this.

[0042] The technical solution provided by the disclosed embodiment pre-customizes a variety of specific teaching behaviors in a constructed teaching behavior library, so as to construct facial motion features in the current live frame according to the target key points detected in the current live frame, and then uses the position deviation between the target key points and the facial motion features to identify the teaching behavior in the current live frame from the constructed teaching behavior library, thereby realizing real-time recognition of live behaviors and improving the timeliness of live behavior recognition; at the same time, various teaching behaviors that do not meet the requirements of teaching specifications can be flexibly configured in the teaching behavior library to realize customized recognition of live behaviors and improve the flexibility and comprehensiveness of live behavior recognition.

[0043] Figure 2A The flowchart of another live broadcast behavior recognition method provided by the embodiment of the present disclosure is shown. This embodiment is optimized based on the various optional solutions provided by the above embodiments. Specifically, this implementation mainly introduces in detail the specific recognition process of live broadcast behavior in teaching live broadcast.

[0044] Optional, such as Figure 2A As shown, the method in this embodiment may include the following steps:

[0045] S210, input the current live broadcast frame into the pre-trained key point detection model, and output the corresponding facial key points and hand key points to obtain the target key points.

[0046] Among them, the key point detection model is trained using historical teaching live broadcasts that have been denoised.

[0047] Optionally, for accurate identification of target key points, the present embodiment will use a large number of historical teaching live broadcasts that have been denoised as training samples for the key point detection model. At this time, the training samples are all live videos in remote teaching scenarios, which can include various types of live teaching information. Therefore, when the key point detection model in the present embodiment is trained using the historical teaching live broadcasts that have been denoised, the trained key point detection model can be more in line with the needs of live teaching, thereby ensuring the accuracy of the key point detection model in detecting target key points.

[0048] In this embodiment, if Figure 2B As shown, the current live frame in the teaching live broadcast will be input into the pre-trained key point detection model, and the key point detection model will be used to identify the key points of the current live frame, so as to accurately extract the pre-set hand key points and face key points from the current live frame to obtain each target key point.

[0049] S220, fitting the standard face template features using the face motion features in the historical live broadcast frames to establish a corresponding face action model.

[0050] Optionally, in order to accurately construct the facial motion features in the current live broadcast frame, a standard facial template is first set. The standard facial template can use a priori determined algebraic formulas to set standard facial part information for the facial template, such as standard inter-ear distance, inter-ear horizontal distance, ear-nose vertical distance, and binocular weighted center of both ears. Then, facial motion features that match the image of the host (teacher) are extracted from the historical live broadcast frames of the teaching live broadcast, and then the facial motion features in the historical live broadcast frames are used to perform deformation fitting on the standard facial template features, so that the fitted facial template can approximate the image of the host of this teaching live broadcast, thereby establishing a facial motion model that approximates the image of the host of this teaching live broadcast, so that the facial motion model can be controlled to simulate the corresponding facial posture information based on the facial key points in the current live broadcast frame.

[0051] S230, inputting the facial key points in the target key points into the facial motion model to obtain the facial motion features in the current live broadcast frame.

[0052] After establishing the corresponding facial motion model, the facial key points in the target key points can be input into the facial motion model. Through the position deviation between the facial key points, the established facial motion model can be controlled to perform corresponding motion adjustments to achieve the position of each facial key point, thereby obtaining the facial motion features in the current live frame.

[0053] For example, for the face rectangle frame in the face motion feature, the left and right ears after adjustment of the face motion model are used as the basic anchor points, and are expanded horizontally outward by a number of pixels as the width of the face rectangle frame. Then, based on the distance between the two ears after adjustment of the face motion model and the vertical distance between the ear and the nose, the length of the face is predicted as the length of the face rectangle frame. At the same time, the proportion of the length of the face rectangle frame in the upper and lower parts of the ear coordinates can also be predicted.

[0054] S240, based on the position deviation between each target key point and the facial movement feature, identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library.

[0055] S250: If the teaching behavior in the current live broadcast frame includes hand behavior, the teaching behavior is input into a pre-trained behavior recognition model to verify the teaching behavior.

[0056] In this embodiment, since the teaching behavior library mainly relies on facial action features and the position deviation between the key points of the hand and the key points of the face to identify the teaching behavior in the current live frame, some normal hand actions may be misidentified. For example, when the teacher raises his hand and passes by the side of the face, it is very likely to be identified as a teaching behavior of the hand supporting the face. Therefore, in order to further improve the accuracy of live behavior recognition on the basis of ensuring the recall rate of live behavior recognition, this embodiment will use a large number of training samples containing various teaching behaviors to specifically train a behavior recognition model, which can accurately identify various teaching behaviors in the current live frame.

[0057] At this time, this embodiment can use the pre-trained behavior recognition model as a secondary filtering mechanism for live broadcast behavior recognition, such as Figure 2B As shown, after identifying the teaching behavior in the current live frame from the teaching behavior library, it is further determined whether the identified teaching behavior contains hand behavior, such as identifying teaching behaviors such as making a phone call, eating, and supporting the head with hands. If the teaching behavior in the current live frame contains hand behavior, the teaching behavior will be input into a pre-trained behavior recognition model, and the behavior recognition model will further identify the teaching behavior existing in the current live frame to determine whether the teaching behavior identified from the teaching behavior library actually exists in the current live frame, thereby performing a secondary verification of the accuracy of the teaching behavior containing hand behavior identified from the teaching behavior library, thereby improving the accuracy of live behavior recognition.

[0058] At this point, since the teaching behavior in the current live broadcast frame has been identified in the teaching behavior library as a pre-screening for re-identifying the teaching behavior containing hand behavior in the behavior recognition model, the behavior recognition model does not need to re-identify the teaching behavior that does not contain hand behavior. Therefore, the behavior recognition model will not bear a large workload, thereby further ensuring the efficiency of live broadcast behavior recognition on the basis of improving the accuracy of live broadcast behavior recognition.

[0059] S260, the current live broadcast frame and the teaching behavior in the current live broadcast frame are associated and recorded in a preset time sequence queue.

[0060] Optionally, in order to avoid the problem of misidentification of a single live frame in live teaching, this embodiment will comprehensively analyze the live behavior recognition results of the current live frame by analyzing the live behavior recognition situations of several consecutive frames. Therefore, this embodiment sets up a stateful online service strategy, which records each live frame that has completed live behavior recognition and the teaching behavior recognized in the live frame by maintaining a timing queue.

[0061] At this time, after identifying the teaching behavior in the current live frame from the teaching behavior library, the current live frame and the teaching behavior identified in the current live frame can be associated and recorded in a preset timing queue in sequence, so that the teaching behavior in the current live frame can be comprehensively analyzed through the teaching behaviors corresponding to multiple consecutive live frames associated and recorded in the timing queue.

[0062] It should be noted that the teaching behavior associated with each live frame in the timing queue may be a teaching behavior that does not include hand behavior and a teaching behavior that includes hand behavior verified by a behavior recognition model.

[0063] S270, verifying the teaching behavior in the current live frame by using the teaching behaviors associated with the preceding continuous live frames of the current live frame in the timing queue.

[0064] Optionally, the teaching behaviors associated with the multiple forward continuous live frames before the current live frame are found in the timing queue, and then the teaching behaviors corresponding to the current live frame are analyzed to see whether they are consistent with the teaching behaviors associated with each forward continuous live frame, so as to verify the teaching behaviors recognized in the current live frame. For example, if the teaching behaviors associated with the current live frame in the timing queue and the teaching behaviors associated with each forward continuous live frame are mostly the same behaviors, then it is determined that the teaching behaviors recognized in the current live frame have passed the verification. If the teaching behaviors associated with the current live frame in the timing queue are different from the teaching behaviors associated with each forward continuous live frame, it means that the teaching behaviors associated with the current live frame may be false actions in a very short period of time. At this time, it can be determined that the teaching behaviors recognized in the current live frame have not passed the verification, thereby improving the accuracy of live behavior recognition on the basis of reducing the false recall rate of live behavior recognition.

[0065] The technical solution provided by the disclosed embodiment verifies the teaching behavior including hand behavior recognized in the current live broadcast frame through a pre-trained behavior recognition model, thereby improving the accuracy of live broadcast behavior recognition, and the behavior recognition model does not need to re-recognize the teaching behavior that does not include hand behavior, thereby further ensuring the efficiency of live broadcast behavior recognition on the basis of improving the accuracy of live broadcast behavior recognition; at the same time, the teaching behavior recognized in the current live broadcast frame is verified for false actions through a timing queue, thereby improving the accuracy of live broadcast behavior recognition on the basis of reducing the false recall rate of live broadcast behavior recognition.

[0066] Figure 3 The structure diagram of a live broadcast behavior recognition device provided by the embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to the case of recognizing user behavior in any live broadcast. The device can be implemented by software and / or hardware and integrated into the device executing the method. Figure 3 As shown, the live broadcast behavior recognition device in the embodiment of the present disclosure may specifically include:

[0067] A face action construction module 310 is used to construct face action features in the current live frame according to the target key points detected in the current live frame;

[0068] The teaching behavior recognition module 320 is used to identify the teaching behavior in the current live broadcast frame from the constructed teaching behavior library based on the position deviation between each of the target key points and the facial movement features.

[0069] The technical solution provided by the disclosed embodiment pre-customizes a variety of specific teaching behaviors in a constructed teaching behavior library, so as to construct facial motion features in the current live frame according to the target key points detected in the current live frame, and then uses the position deviation between the target key points and the facial motion features to identify the teaching behavior in the current live frame from the constructed teaching behavior library, thereby realizing real-time recognition of live behaviors and improving the timeliness of live behavior recognition; at the same time, various teaching behaviors that do not meet the requirements of teaching specifications can be flexibly configured in the teaching behavior library to realize customized recognition of live behaviors and improve the flexibility and comprehensiveness of live behavior recognition.

[0070] Furthermore, the above target key points include facial key points and hand key points.

[0071] Furthermore, the above-mentioned face action construction module 310 can be specifically used for:

[0072] The facial motion features in the historical live frames are used to fit the standard facial template features to establish the corresponding facial motion model;

[0073] The facial key points in the target key points are input into the facial action model to obtain the facial action features in the current live broadcast frame.

[0074] Furthermore, the teaching behavior recognition module 320 can be specifically used for:

[0075] Determining a behavior judgment condition preset for each teaching behavior in the teaching behavior library, wherein the behavior judgment condition for each teaching behavior is set according to a target key point position included in the teaching behavior and / or a facial action feature involved in the teaching behavior;

[0076] Determine whether the position deviations between the target key points and the facial action features meet the behavior judgment conditions;

[0077] A target teaching behavior corresponding to the behavior judgment condition that is met is searched out from the teaching behavior library, and the target teaching behavior is identified as the teaching behavior in the current live broadcast frame.

[0078] Furthermore, the above-mentioned live broadcast behavior recognition device may also include:

[0079] An associated recording module, used for associating and recording the current live broadcast frame and the teaching behavior in the current live broadcast frame into a preset time sequence queue;

[0080] The first behavior verification module is used to verify the teaching behavior in the current live frame by using each teaching behavior associated with the forward continuous live frame of the current live frame in the timing queue.

[0081] Furthermore, the above-mentioned live broadcast behavior recognition device may also include:

[0082] The second behavior verification module is used to input the teaching behavior into the pre-trained behavior recognition model to verify the teaching behavior if the teaching behavior in the current live broadcast frame includes hand behavior.

[0083] Furthermore, the above-mentioned live broadcast behavior recognition device may also include:

[0084] A key point detection module, used for inputting the current live broadcast frame into a pre-trained key point detection model, and outputting corresponding facial key points and hand key points to obtain the target key points;

[0085] The key point detection model is trained using live historical teaching videos that have been denoised.

[0086] The live broadcast behavior identification device provided in the embodiment of the present disclosure belongs to the same inventive concept as the live broadcast behavior identification method provided in the above embodiment. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.

[0087] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the embodiment of the present disclosure. The device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0088] like Figure 4 As shown, the device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0089] Typically, the following devices may be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 may allow the device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The device 400 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0090] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0091] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0092] In some embodiments, the devices may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0093] The computer-readable medium may be included in the device, or may exist independently without being incorporated into the device.

[0094] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the device, the device: constructs facial action features in the current live broadcast frame based on the target key points detected in the current live broadcast frame; based on the position deviation between each of the target key points and the facial action features, identifies the teaching behavior in the current live broadcast frame from the constructed teaching behavior library.

[0095] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0097] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0098] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] According to one or more embodiments of the present disclosure, a live broadcast behavior recognition method is provided, the method comprising:

[0101] Constructing facial motion features in the current live broadcast frame according to the target key points detected in the current live broadcast frame;

[0102] Based on the position deviation between each of the target key points and the facial movement features, the teaching behavior in the current live broadcast frame is identified from the constructed teaching behavior library.

[0103] According to one or more embodiments of the present disclosure, in the above method, the target key points include facial key points and hand key points.

[0104] According to one or more embodiments of the present disclosure, in the above method, constructing facial motion features in the current live frame based on the target key points detected in the current live frame includes:

[0105] The facial motion features in the historical live frames are used to fit the standard facial template features to establish the corresponding facial motion model;

[0106] The facial key points in the target key points are input into the facial action model to obtain the facial action features in the current live broadcast frame.

[0107] According to one or more embodiments of the present disclosure, in the above method, identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library based on the position deviation between each of the target key points and the facial action features includes:

[0108] Determining a behavior judgment condition preset for each teaching behavior in the teaching behavior library, wherein the behavior judgment condition for each teaching behavior is set according to a target key point position included in the teaching behavior and / or a facial action feature involved in the teaching behavior;

[0109] Determine whether the position deviations between the target key points and the facial action features meet the behavior judgment conditions;

[0110] A target teaching behavior corresponding to the behavior judgment condition that is met is searched out from the teaching behavior library, and the target teaching behavior is identified as the teaching behavior in the current live broadcast frame.

[0111] According to one or more embodiments of the present disclosure, in the above method, after identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library, it also includes:

[0112] Associating and recording the current live broadcast frame and the teaching behavior in the current live broadcast frame into a preset time sequence queue;

[0113] The teaching behaviors in the current live frame are verified by using the teaching behaviors associated with the forward continuous live frames of the current live frame in the timing queue.

[0114] According to one or more embodiments of the present disclosure, in the above method, after identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library, it also includes:

[0115] If the teaching behavior in the current live broadcast frame includes hand behavior, the teaching behavior is input into the pre-trained behavior recognition model to verify the teaching behavior.

[0116] According to one or more embodiments of the present disclosure, in the above method, before constructing the facial action features in the current live frame according to the target key points detected in the current live frame, the method further includes:

[0117] Inputting the current live broadcast frame into a pre-trained key point detection model, and outputting corresponding facial key points and hand key points to obtain the target key points;

[0118] The key point detection model is trained using live historical teaching videos that have been denoised.

[0119] According to one or more embodiments of the present disclosure, a live broadcast behavior recognition device is provided, the device comprising:

[0120] A face action construction module, used to construct face action features in the current live frame according to the target key points detected in the current live frame;

[0121] The teaching behavior recognition module is used to identify the teaching behavior in the current live broadcast frame from the constructed teaching behavior library based on the position deviation between each of the target key points and the facial movement features.

[0122] According to one or more embodiments of the present disclosure, in the above-mentioned device, the target key points include facial key points and hand key points.

[0123] According to one or more embodiments of the present disclosure, in the above device, the face action construction module is specifically used to:

[0124] The facial motion features in the historical live frames are used to fit the standard facial template features to establish the corresponding facial motion model;

[0125] The facial key points in the target key points are input into the facial action model to obtain the facial action features in the current live broadcast frame.

[0126] According to one or more embodiments of the present disclosure, in the above device, the teaching behavior recognition module is specifically used to:

[0127] Determining a behavior judgment condition preset for each teaching behavior in the teaching behavior library, wherein the behavior judgment condition for each teaching behavior is set according to a target key point position included in the teaching behavior and / or a facial action feature involved in the teaching behavior;

[0128] Determine whether the position deviations between the target key points and the facial action features meet the behavior judgment conditions;

[0129] A target teaching behavior corresponding to the behavior judgment condition that is met is searched out from the teaching behavior library, and the target teaching behavior is identified as the teaching behavior in the current live broadcast frame.

[0130] According to one or more embodiments of the present disclosure, the above device further includes:

[0131] An associated recording module, used for associating and recording the current live broadcast frame and the teaching behavior in the current live broadcast frame into a preset time sequence queue;

[0132] The first behavior verification module is used to verify the teaching behavior in the current live frame by using each teaching behavior associated with the forward continuous live frame of the current live frame in the timing queue.

[0133] According to one or more embodiments of the present disclosure, the above device further includes:

[0134] The second behavior verification module is used to input the teaching behavior into the pre-trained behavior recognition model to verify the teaching behavior if the teaching behavior in the current live broadcast frame includes hand behavior.

[0135] According to one or more embodiments of the present disclosure, the above device further includes:

[0136] A key point detection module, used for inputting the current live broadcast frame into a pre-trained key point detection model, and outputting corresponding facial key points and hand key points to obtain the target key points;

[0137] The key point detection model is trained using live historical teaching videos that have been denoised.

[0138] According to one or more embodiments of the present disclosure, an electronic device is provided, the device comprising:

[0139] one or more processors;

[0140] A memory for storing one or more programs;

[0141] When the one or more programs are executed by the one or more processors, the one or more processors implement the live broadcast behavior identification method as described in any embodiment of the present disclosure.

[0142] According to one or more embodiments of the present disclosure, a readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the live broadcast behavior recognition method as described in any embodiment of the present disclosure is implemented.

[0143] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0144] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0145] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for identifying live broadcast behavior, characterized in that: include: Constructing facial motion features in the current live broadcast frame according to the target key points detected in the current live broadcast frame; The target key points include facial key points and hand key points, wherein the facial key points include five key points of ears, eyes and nose, the hand key points include two key points of wrists, and the facial action features include at least facial rectangular frame detection, facial profile degree, facial elliptical frame and facial tilt degree; Based on the position deviation between each of the target key points and the facial action features, identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library; Wherein, based on the position deviation between each of the target key points and the facial action features, identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library includes: Determining a behavior judgment condition preset for each teaching behavior in the teaching behavior library, wherein the behavior judgment condition for each teaching behavior is set according to a target key point position included in the teaching behavior and / or a facial action feature involved in the teaching behavior; wherein the teaching behavior library includes non-standard teaching behaviors; Determine whether the position deviation between each of the target key points and the facial action feature meet each of the behavior judgment conditions, the position deviation between each of the target key points reflects the positional overlap between different parts represented by each of the target key points, and the facial action feature reflects the current posture of the face in the current live frame; Searching out a target teaching behavior corresponding to the behavior judgment condition that meets the requirement from the teaching behavior library, and identifying the target teaching behavior as the teaching behavior in the current live broadcast frame; After identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library, the method further includes: Associating and recording the current live broadcast frame and the teaching behavior in the current live broadcast frame into a preset time sequence queue; The teaching behaviors in the current live frame are verified by using the teaching behaviors associated with the forward continuous live frames of the current live frame in the timing queue.

2. The method according to claim 1, characterized in that The step of constructing the facial motion features in the current live broadcast frame according to the target key points detected in the current live broadcast frame includes: The facial motion features in the historical live frames are used to fit the standard facial template features to establish the corresponding facial motion model; The facial key points in the target key points are input into the facial action model to obtain the facial action features in the current live broadcast frame.

3. The method according to any one of claims 1 to 2, characterized in that: After identifying the teaching behavior in the current live broadcast frame from the constructed teaching behavior library, the method further includes: If the teaching behavior in the current live broadcast frame includes hand behavior, the teaching behavior is input into the pre-trained behavior recognition model to verify the teaching behavior.

4. The method according to any one of claims 1 to 2, characterized in that: Before constructing the facial action features in the current live broadcast frame according to the target key points detected in the current live broadcast frame, the method further includes: Inputting the current live broadcast frame into a pre-trained key point detection model, and outputting corresponding facial key points and hand key points to obtain the target key points; The key point detection model is trained using live historical teaching videos that have been denoised.

5. A live broadcast behavior recognition device, characterized in that: include: A face action construction module is used to construct face action features in the current live frame according to the target key points detected in the current live frame, wherein the target key points include face key points and hand key points, wherein the face key points include five key points of ears, eyes and nose, and the hand key points include two key points of wrists, and the face action features include at least face rectangle frame detection, face profile degree, face elliptical frame and face tilt degree; A teaching behavior recognition module, used to recognize the teaching behavior in the current live broadcast frame from the constructed teaching behavior library based on the position deviation between each of the target key points and the facial action features; The teaching behavior recognition module is specifically used for: Determining a behavior judgment condition preset for each teaching behavior in the teaching behavior library, wherein the behavior judgment condition for each teaching behavior is set according to a target key point position included in the teaching behavior and / or a facial action feature involved in the teaching behavior; wherein the teaching behavior library includes non-standard teaching behaviors; Determine whether the position deviation between each of the target key points and the facial action feature meet each of the behavior judgment conditions, the position deviation between each of the target key points reflects the positional overlap between different parts represented by each of the target key points, and the facial action feature reflects the current posture of the face in the current live frame; Searching out a target teaching behavior corresponding to the behavior judgment condition that meets the requirement from the teaching behavior library, and identifying the target teaching behavior as the teaching behavior in the current live broadcast frame; An associated recording module, used for associating and recording the current live broadcast frame and the teaching behavior in the current live broadcast frame into a preset time sequence queue; The first behavior verification module is used to verify the teaching behavior in the current live frame by using each teaching behavior associated with the forward continuous live frame of the current live frame in the timing queue.

6. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the live broadcast behavior recognition method as described in any one of claims 1-4.

7. A readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the live broadcast behavior recognition method as described in any one of claims 1-4 is implemented.

Images