Facial action detection method, device and system
By using the target network model in video surveillance technology to analyze multiple color and deep face images in video clips, automatically detect and classify facial movements, the problem of inefficiency in the existing technology is solved and efficient abnormal behavior is achieved Positioning and detection.
Patent Information
- Application Number
- CN201910760634.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-08-16
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-08-16
AI Technical Summary
In existing video surveillance technology, multiple video clips need to be manually checked to locate abnormal behavior, resulting in low efficiency.
By obtaining multiple color face images and deep face images associated with the current video clip, the target network model (including feature fusion network and multi-frame analysis network) is used to automatically detect the target's facial movements, and determine whether they belong to the action classification results based on the action classification results. Abnormal action, thereby positioning video clips of abnormal behavior.
The efficiency and accuracy of determining abnormal behavior video clips are improved, and the need for manual investigation is avoided.
Smart Images

Figure CN112395922B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video surveillance technology, and in particular to a facial action detection method, device and system. Background Art
[0002] In daily life, targets such as users may engage in abnormal behaviors, such as illegal and criminal behaviors. When abnormal behaviors occur, they often have a negative impact on social stability. In order to understand the on-site situation when abnormal behaviors occur, it is usually necessary to locate the scene where the abnormal behaviors occur.
[0003] In related technologies, video surveillance can usually be used to locate the scene when a target exhibits abnormal behavior. For example, multiple video clips containing the target can be screened out from the surveillance video through human eye search or face recognition technology. Then, the multiple video clips are manually checked to determine the video clips when the target exhibits abnormal behavior, such as video clips of a crime scene, so as to locate the scene where the abnormal behavior occurs.
[0004] However, since multiple video clips need to be manually checked, the workload is large and the manual checking speed is slow, resulting in low efficiency in determining video clips containing abnormal behaviors. Summary of the invention
[0005] The present application provides a facial action detection method, device and system, which can solve the problem of low efficiency in determining video clips containing abnormal behaviors in the related art. The technical solution is as follows:
[0006] In one aspect, a facial action detection method is provided, the method comprising:
[0007] Obtain multiple color face images and corresponding depth face images associated with the current video clip;
[0008] Based on the multiple color face images and the corresponding depth face images, determining the facial action of the target through a target network model;
[0009] When it is determined that the facial action of the target is an abnormal action, target video data is acquired, where the target video data includes the current video clip and / or a video image corresponding to the facial action that is an abnormal action.
[0010] In a possible implementation of the present application, the step of obtaining multiple face images and corresponding depth face images associated with the current video clip includes:
[0011] Acquire a plurality of color video images and corresponding depth video images associated with the video clip and including the target;
[0012] Performing face detection on the acquired multiple color video images and corresponding depth video images respectively;
[0013] Determine, according to the face detection result, a plurality of face color region images and corresponding face depth region images from the plurality of color video images and corresponding depth video images;
[0014] The multiple face color area images and the corresponding face depth area images are determined as the multiple color face images and the corresponding depth face images.
[0015] In a possible implementation of the present application, determining the multiple face color area images and the corresponding face depth area images as the multiple color face images and the corresponding depth face images includes:
[0016] Performing face alignment processing on a plurality of face color region images and corresponding face depth region images respectively;
[0017] Adjusting the sizes of the multiple face color region images and the corresponding face depth region images after face alignment processing to the same size;
[0018] The multiple face color area images and the corresponding face depth area images after resizing are used as the multiple color face images and the corresponding depth face images.
[0019] In a possible implementation of the present application, the target network model includes a feature fusion network and a multi-frame analysis network, and the facial action of the target is determined by the target network model based on the multiple color face images and the corresponding depth face images, including:
[0020] Inputting the plurality of color face images and the corresponding depth face images into the target network model, extracting key features of each face image and the corresponding depth face image through the feature fusion network in the target network model and fusing them to obtain fusion features corresponding to each face image;
[0021] The obtained multiple fusion features are analyzed and processed by a multi-frame analysis network in the target network model to determine the facial action of the target.
[0022] In a possible implementation of the present application, after determining the facial action of the target, the method further includes:
[0023] classifying facial movements of the target;
[0024] Determine whether the facial action is an abnormal action according to the classification result.
[0025] In a possible implementation of the present application, after acquiring the target video data, the method further includes:
[0026] Extracting, from other video segments including the target, video subsegments corresponding to the facial action or extracting video subsegments corresponding to the action categories to which the facial action belongs;
[0027] synthesizing the target video data and the extracted video sub-segment into a video according to shooting time of the target video data and the extracted video sub-segment and / or image frame numbers of the target video data and the extracted video sub-segment;
[0028] Play the synthesized video.
[0029] In a possible implementation of the present application, the method further includes:
[0030] Obtaining image information of a video image corresponding to a facial action belonging to a specified category;
[0031] Determine the position of the camera that captures the facial action belonging to the specified category according to the image information;
[0032] The determined position of the camera is sent to a designated terminal, and / or the determined position of the camera is added to the image information for display.
[0033] In a possible implementation of the present application, the target network model is obtained by training the network model to be trained based on multiple face color image samples and corresponding face depth image samples, as well as actual facial action categories of the faces in the multiple face image samples.
[0034] In another aspect, a facial action detection device is provided, the device comprising:
[0035] A first acquisition module acquires a plurality of color face images and corresponding depth face images associated with a current video clip;
[0036] A determination module, based on the plurality of color face images and the corresponding depth face images, determines the facial action of the target through a target network model;
[0037] The second acquisition module acquires target video data when it is determined that the target facial action is an abnormal action, wherein the target video data includes the current video clip and / or a video image corresponding to the facial action that is an abnormal action.
[0038] In a possible implementation of the present application, the first acquisition module is used to:
[0039] Acquire a plurality of color video images and corresponding depth video images associated with the video clip and including the target;
[0040] Performing face detection on the acquired multiple color video images and corresponding depth video images respectively;
[0041] Determine, according to the face detection result, a plurality of face color region images and corresponding face depth region images from the plurality of color video images and corresponding depth video images;
[0042] The multiple face color area images and the corresponding face depth area images are determined as the multiple color face images and the corresponding depth face images.
[0043] In a possible implementation of the present application, the first acquisition module is used to:
[0044] Performing face alignment processing on a plurality of face color region images and corresponding face depth region images respectively;
[0045] Adjusting the sizes of the multiple face color region images and the corresponding face depth region images after face alignment processing to the same size;
[0046] The multiple face color area images and the corresponding face depth area images after resizing are used as the multiple color face images and the corresponding depth face images.
[0047] In a possible implementation of the present application, the target network model includes a feature fusion network and a multi-frame analysis network, and the determination module is used to:
[0048] Inputting the plurality of color face images and the corresponding depth face images into the target network model, extracting key features of each face image and the corresponding depth face image through the feature fusion network in the target network model and fusing them to obtain fusion features corresponding to each face image;
[0049] The obtained multiple fusion features are analyzed and processed by a multi-frame analysis network in the target network model to determine the facial action of the target.
[0050] In a possible implementation of the present application, the determining module is further configured to:
[0051] classifying facial movements of the target;
[0052] Determine whether the facial action is an abnormal action according to the classification result.
[0053] In a possible implementation of the present application, the second acquisition module is used to:
[0054] Extracting, from other video segments including the target, video subsegments corresponding to the facial action or extracting video subsegments corresponding to the action categories to which the facial action belongs;
[0055] synthesizing the target video data and the extracted video sub-segment into a video according to shooting time of the target video data and the extracted video sub-segment and / or image frame numbers of the target video data and the extracted video sub-segment;
[0056] Play the synthesized video.
[0057] In a possible implementation of the present application, the second acquisition module is further used to:
[0058] Obtaining image information of a video image corresponding to a facial action belonging to a specified category;
[0059] Determine the position of the camera that captures the facial action belonging to the specified category according to the image information;
[0060] The determined position of the camera is sent to a designated terminal, and / or the determined position of the camera is added to the image information for display.
[0061] In a possible implementation of the present application, the target network model is obtained by training the network model to be trained based on multiple face color image samples and corresponding face depth image samples, as well as actual facial action categories of the faces in the multiple face image samples.
[0062] In another aspect, a monitoring system is provided, the monitoring system comprising a processor and at least one camera, the processor being configured to:
[0063] Acquire a plurality of color face images and corresponding depth face images associated with the current video clip collected by the at least one camera;
[0064] Based on the multiple color face images and the corresponding depth face images, determining the facial action of the target through a target network model;
[0065] When it is determined that the facial action of the target is an abnormal action, target video data is acquired, where the target video data includes the current video clip and / or a video image corresponding to the facial action that is an abnormal action.
[0066] In a possible implementation of the present application, the processor is further configured to:
[0067] When the at least one camera includes a red, green, blue, depth (RGBD) camera, a color video image is acquired through the RGBD camera and a corresponding depth video image is acquired in the presence of infrared light; or
[0068] When the at least one camera includes two red, green and blue (RGB) cameras, the two RGB cameras respectively capture color video images, and determine the corresponding depth video image according to the color video images respectively captured by the two RGB cameras; or,
[0069] When the at least one camera includes an RGB camera and a depth camera, a color video image is acquired through the RGB camera, and a depth video image is acquired through the depth camera.
[0070] On the other hand, a control device is provided, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, the memory is used to store computer programs, and the processor is used to execute the programs stored in the memory to implement the steps of the facial action detection method described in the above aspect.
[0071] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the facial action detection method described in the above aspect are implemented.
[0072] On the other hand, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the facial action detection method described in the first aspect.
[0073] The technical solution provided by this application can at least bring the following beneficial effects:
[0074] Acquire multiple color face images and corresponding depth face images associated with the current video clip. Input the multiple color face images and corresponding depth face images into the target network model, and the target network model determines the facial action of the target based on the multiple color face images and corresponding depth face images. When the facial action is an abnormal action, acquire the target video data. In this way, for any video clip, the facial action of the target can be automatically determined by the target network model based on the color face image and the depth face image, thereby determining whether the facial action is an abnormal action, and then locating the target video data with abnormal action, avoiding the need for manual investigation of multiple video clips, thereby improving the efficiency of determining video clips with abnormal behavior and improving the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0076] Figure 1 is a flow chart of a facial action detection method provided by an embodiment of the present application;
[0077] Figure 2 is a schematic diagram of a video image and a face area image provided in an embodiment of the present application;
[0078] Figure 3 is a schematic diagram of a face region image provided in an embodiment of the present application;
[0079] Figure 4 is a schematic diagram of another face region image provided in an embodiment of the present application;
[0080] Figure 5 is a structural schematic diagram of a facial action detection device provided in an embodiment of the present application;
[0081] Figure 6 It is a structural schematic diagram of a control device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0082] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0083] Before introducing in detail the facial action detection method provided in the embodiment of the present application, a brief introduction to the terms and implementation environment involved in the embodiment of the present application is given.
[0084] First, the terms involved in the embodiments of the present application are briefly introduced.
[0085] Depth image: refers to an image that uses the distance from the camera to each point in the scene as pixel values.
[0086] Color image: refers to an image in which each pixel is composed of R, G, and B components.
[0087] Next, the implementation environment involved in the embodiments of the present application is briefly introduced.
[0088] The implementation environment involved in the embodiments of the present application may include a monitoring system, which may include a processor and at least one camera. The processor may be configured in a control device, and the at least one camera may be connected to the control device, or may also be configured on the control device.
[0089] As an example, the at least one camera may include an RGBD (Red Green Blue Depth) camera, and a color video image and a corresponding depth video image may be acquired by exposing the RGBD camera twice in different ways. For example, the RGBD camera may acquire a color video image under conventional exposure conditions, and may acquire a corresponding depth video image under infrared exposure conditions.
[0090] As another example, the at least one camera may include two RGB (Red Green Blue Depth) cameras, so as to respectively capture images through the two RGB cameras to obtain a color video image. Further, the obtained color video image may be processed to obtain a depth video image.
[0091] As another example, the at least one camera may also include an RGB camera and a depth camera, the RGB camera may be used to capture color video images, and the depth camera may be used to capture depth video images. As an example, the depth camera may be monocular structured light, TOF (Time Of Fly), binocular vision, etc.
[0092] As an example, the control device may be a computer, a Pocket PC (PPC), a tablet computer, etc.
[0093] After introducing the nouns and implementation environment involved in the embodiments of the present application, the facial action detection method provided in the embodiments of the present application will be described in detail with reference to the accompanying drawings.
[0094] Please refer to Figure 1 , Figure 1 FIG. 1 is a flowchart of a facial action detection method according to an exemplary embodiment. The facial action detection method can be applied to the above implementation environment. The facial action detection method can include the following steps:
[0095] Step 101: Acquire multiple color face images and corresponding depth face images associated with the current video clip.
[0096] The current video clip may be any video clip currently shot in real time, or may be any video clip of a plurality of pre-stored video clips. In addition, the monitoring scene in the video clip may be pre-set by the user according to actual conditions, for example, the monitoring scene may be an ATM (Automatic Teller Machine) withdrawal area, a subway exit area, a hospital entrance area, etc.
[0097] Among them, the facial images in the multiple color facial images and the corresponding depth facial images can be target facial images, and the target can be specified by the user according to actual needs. As an example, the control device can provide a user interaction interface, and when a query on a target is required, the user can specify the target through the user interaction interface.
[0098] In the embodiment of the present application, the control device determines the facial action of the target based on the facial image of the target. At present, the color facial image obtained from the surveillance video is generally processed, but the color facial image has poor resistance to interference from lighting, posture, etc., and the result obtained in this way is not accurate enough. Therefore, the embodiment of the present application obtains the color facial image and the corresponding depth facial image of the target, so as to improve the accuracy of the subsequent determination of facial actions.
[0099] Therein, the relationship between the multiple color face images and the depth face images may be one-to-one, or many-to-one, or one-to-many.
[0100] In addition, since the facial movements of the target are often dynamic, the control device usually needs to obtain multiple color face images and corresponding depth face images to determine the facial movements of the target.
[0101] As an example, the specific implementation of obtaining multiple face images and corresponding depth face images associated with the current video clip may include: obtaining multiple color video images and corresponding depth video images associated with the video clip and including the target, performing face detection on the obtained multiple color video images and corresponding depth video images respectively, and according to the face detection results, determining multiple face color area images and corresponding face depth area images from the multiple color video images and corresponding depth video images, and determining the multiple face color area images and corresponding face depth area images as the multiple color face images and corresponding depth face images.
[0102] As mentioned above, the control device can obtain multiple color video images and corresponding depth video images associated with the video clip through an RGB camera and a depth camera respectively. Then, the control device performs face detection on the multiple color video images and corresponding depth video images respectively, determines the multiple color video images and corresponding depth video images containing the target face, and then cuts out the area where the face is located according to the face detection result to obtain the face area image corresponding to each color video image and the corresponding depth video image. It is not difficult to understand that the face area image includes the face color area image and the corresponding face depth area image.
[0103] For example, Figure 2 As shown in FIG. 1 , face detection is performed on a color video image to determine the target in the color video image, such as Figure 2 As shown in (a), the face area is cut out according to the face detection result, as shown in Figure 2 As shown in (b), the color area image of the face is obtained.
[0104] As an example, face detection can be implemented through single CNN (Convolutional Neural Networks) face detection, cascade CNN face detection, OpenCV face detection and other methods to determine whether an image contains a face and whether the face is a target face.
[0105] Furthermore, after obtaining the determined multiple face color region images and the corresponding face depth region images, the sizes of the multiple face color region images and the corresponding face depth region images can be adjusted to the same size, and the adjusted multiple face color region images and the corresponding face depth region images are determined as the multiple face color images and the corresponding face depth images of the target. In addition, when adjusting the size, it can be adjusted according to the reference size, and the reference size can be set in advance according to actual needs.
[0106] Furthermore, before determining the determined multiple face color area images and the corresponding face depth area images as a multiple color face images and the corresponding depth face images, face alignment processing is performed on the multiple face color area images and the corresponding face depth area images respectively, the sizes of the multiple face color area images and the corresponding face depth area images after the face alignment processing are adjusted to the same size, and the multiple face color area images and the corresponding face depth area images after the size adjustment are used as the multiple color face images and the corresponding depth face images.
[0107] Since the face region image may have various sizes and the postures of the faces in the face region image may also be different, in order to more accurately determine the facial movements of the target, the face region image may be adjusted, such as performing face alignment processing and image size adjustment on the face region image.
[0108] Among them, face alignment processing refers to aligning the faces in each face area image through the coordinates of the key points of the face area image, that is, normalizing the face area images with different face postures so that the postures of the faces in all face area images are as similar as possible.
[0109] For example, for any face region image, the coordinates of the two eyes, nose, mouth and other key parts can be determined. Based on these coordinates, the face alignment algorithm can be used to align the face and obtain a frontal face region image. Figure 3 As shown, Figure 3 (a) is the face area image without face alignment processing. Figure 3 (b) is the face area image after face alignment processing.
[0110] Since the position of the target in the video captured by the camera may be different from the camera, the size of the obtained face area image may be different. In order to ensure the accuracy of the facial action category judgment result of the target, the face area image after the face alignment process can be adjusted so that the size of the face area image is exactly the same. Among them, standard data of a face area image can be set in advance, such as standard length, standard width and standard key point coordinates. These data can be adjusted according to actual conditions, and all face area images need to be adjusted according to the set standard data.
[0111] As an example, Figure 4 As shown, the width of each face region image is adjusted to a standard width, the height of each face region image is adjusted to a standard height, and the coordinates of the key points in each face region image are adjusted to standard key point coordinates, where Figure 4 (a) Figure 4 (b) is the image after face alignment. After image size adjustment, it becomes Figure 4 Images of the same size as in (c).
[0112] Step 102: Based on the multiple color face images and the corresponding depth face images, determine the facial movements of the target through the target network model.
[0113] Facial movements may include smiling, frowning, eyes widening, etc., and facial movements can usually indicate the target's emotions, so the target's facial movements can be determined first and then processed later.
[0114] As an example, when the target network model includes a feature fusion network and a multi-frame analysis network, the facial movements of the target are determined through the target network model based on the multiple color face images and the corresponding depth face images, including: inputting the multiple color face images and the corresponding depth face images into the target network model, extracting the key features of each face image and the corresponding depth face image through the feature fusion network in the target network model and fusing them to obtain fused features corresponding to each face image, and analyzing and processing the obtained multiple fused features through the multi-frame analysis network in the target network model to determine the facial movements of the target.
[0115] Since features are extracted separately for the target face color image and the corresponding target face depth image, the recognition results obtained based on the analysis of the depth features and the recognition results obtained based on the analysis of the color features are not accurate enough. In order to obtain more accurate recognition results, it is necessary to fuse the extracted color features and depth features, and then process them according to the obtained fused features. Therefore, the target network model includes a feature fusion network.
[0116] The control device can perform convolution operations on the target face color image and the corresponding target face depth image through the feature fusion network in the target network model, extract the color features in the target face color image and the depth features in the corresponding target face depth image, connect the color features and the depth features, and then use the fully connected layer to extract fusion features from the connected color features and depth features. Feature fusion can be achieved based on a neural network, which automatically selects key features from color features and depth features through learning to fuse them to achieve the best judgment result.
[0117] For example, when determining facial movements, the most critical parts are usually the eyes, mouth, eyebrows, etc. Different movements of these parts represent a variety of facial movements. In comparison, the nose moves less, so the nose can be analyzed as little as possible to speed up the judgment and improve the accuracy of the judgment. In other words, the color features and depth features of the eyes, mouth, and eyebrows can be selected as key features.
[0118] After the control device obtains multiple fused features through the feature fusion network in the target network model, the multiple fused features are input into the multi-frame analysis network to determine the facial movements of the target by analyzing the facial motion units through the multi-frame analysis network.
[0119] Among them, the facial motion unit can be used to indicate the movement of various areas of the face, for example, it can indicate the movement of the corners of the mouth, eyelids, philtrum and other areas. Please refer to Table 1, which is a definition of a facial motion unit (AU) according to an exemplary embodiment. Different facial motion units can be combined to represent different facial movements.
[0120] Table 1
[0121] AU definition AU definition AU1 Raised inner eyebrow corner AU14 Tightening of mouth corners AU2 Outer eyebrow corners raised AU15 Pull the corners of your mouth downward AU4 frown AU16 Pull your lips down AU5 Upper eyelid lift AU17 Push the lower lip upward AU6 Cheek lift and orbicularis oculi tightening AU20 Mouth corner stretch AU7 Retraction of eyelids AU23 Tighten your lips AU9 Wrinkled nose AU24 Shut up AU10 Lift your upper lip AU25 Lips parted AU11 The skin of the philtrum is upward AU26 Open your mouth AU12 Pull the corners of the mouth upward AU32 Biting your lips
[0122] Furthermore, the target network model is obtained by training the network model to be trained based on multiple face color image samples and corresponding face depth image samples, as well as actual facial action categories of the faces in the multiple face image samples.
[0123] If the target network model processes and analyzes multiple face color images and corresponding face depth images based on the initial model parameters, the recognition result of the target facial action may not be accurate enough. In order to accurately determine the target facial action, the target network model needs to be trained first. As an example, multiple face color image samples and corresponding face depth image samples can be pre-set, and face color image samples and corresponding face depth image samples that can represent different facial actions are selected as face image samples. Afterwards, the face color image samples and corresponding face depth image samples of continuous time are set as a group of face image samples, that is, each group of face image samples corresponds to a video clip.
[0124] Determine the actual facial movements of multiple face color image samples and corresponding face depth image samples, input the multiple face color image samples and the corresponding face depth image samples into a network model to be trained, the network model to be trained analyzes and processes the multiple face color image samples and the corresponding face depth image samples based on initial model parameters, outputs recognition results for facial movements, compares the output recognition results of facial movements with actual facial movements, and if the output recognition results of facial movements are wrong, adjust the initial model parameters until a large number of face image samples, such as 1,000 groups, are input, and when the accuracy of facial movement recognition results is high, for example, when the accuracy is greater than or equal to 95%, it can be considered that the network model to be trained has been trained, and the network model obtained at this time can be determined as the target network model. In this way, the target network model can be used to detect the facial movements of any target based on multiple face color image samples and corresponding face depth image samples of the target.
[0125] Furthermore, the number of facial image samples used for training may be limited, for example, N groups are selected. Correspondingly, when detecting the target facial image, N groups of facial images are also obtained.
[0126] Furthermore, the sizes of the face image samples used for training may be adjusted to the same size, for example, according to the above reference size.
[0127] Furthermore, after determining the facial action of the target, the method further includes: classifying the facial action of the target, and determining whether the facial action is an abnormal action according to the classification result.
[0128] Exemplarily, facial movements can be classified into different categories according to different classification rules. For example, when facial movements are classified according to the emotions of the targets they indicate, facial movements such as smiling, laughing, and stretching eyebrows indicating that the target's emotions are happy can be classified into the happy category, and facial movements such as frowning, drooping eyes, and downturned mouth corners indicating that the target's emotions are angry can be classified into the angry category. In this way, facial movements can be classified into the angry category, the happy category, the fear category, the angry category, the nervous category, etc. according to the different emotions of the targets they indicate.
[0129] This embodiment can determine whether a facial action is an abnormal action based on the classification result of the facial action. Abnormal actions can usually be set by the user according to actual conditions. Generally speaking, since various facial actions have certain correlations, such as anger, rage, and tension can all express a negative emotion, abnormal actions may include more than one type of facial action, that is, abnormal actions can include one type of facial action, or multiple types of facial actions. For example, the abnormal action can be set as angry or nervous facial actions, or as angry facial actions.
[0130] When the classification result of the facial action is that the facial action belongs to the facial action category included in the abnormal action, it is determined that the facial action belongs to the abnormal action. When the classification result of the facial action is that the facial action does not belong to the facial action category included in the abnormal action, it is determined that the facial action does not belong to the abnormal action.
[0131] For example, when the facial action of anger is set as an abnormal action, the target detection model determines that the target's facial action is wide-eyed, and classifies the facial action of wide-eyed as anger. Since the facial action of anger is an abnormal action, it can be determined that the facial action is an abnormal action.
[0132] Step 103: when it is determined that the facial action of the target is an abnormal action, target video data is acquired, where the target video data includes the current video clip and / or a video image corresponding to the facial action that is an abnormal action.
[0133] That is, the target video data may include the video image corresponding to the abnormal action, may include the video segment to which the video image corresponding to the abnormal action belongs, or may include the video image corresponding to the abnormal action and the video segment to which the video image belongs.
[0134] Furthermore, after acquiring the target video data, the video sub-segments corresponding to the facial action can be extracted from other video segments including the target, or the video sub-segments corresponding to the action categories to which the facial action belongs can be extracted. According to the shooting time of the target video data and the extracted video sub-segments, and / or the image frame numbers of the target video data and the extracted video sub-segments, the target video data and the extracted video sub-segments are synthesized into a video, and the synthesized video is played.
[0135] That is to say, if the target's facial action is an abnormal action, when viewing the video later, there is a high possibility that there will be a need to retrieve all video data with the same facial action. To this end, after obtaining the target video data, the video sub-segment including the facial action can be extracted from other video segments, or the video sub-segments corresponding to the action category to which the facial action belongs can be extracted, and then the extracted video sub-segments and the obtained target video data can be synthesized and played.
[0136] Furthermore, because a type of facial action indicates the same emotion, it can be considered that the facial actions in the action category to which the facial action belongs are all abnormal actions. In some embodiments, there may be a need to retrieve all video data of the action category to which the facial action belongs. To this end, after obtaining the target video data, the video sub-segments corresponding to the action category to which the facial action belongs can be extracted from other video segments, and then the extracted video sub-segments and the obtained target video data are synthesized and played.
[0137] In this way, the video data belonging to abnormal actions are synthesized into a video, which can facilitate users to quickly locate abnormal videos and avoid the need to play and view a large number of video clips one by one, thereby improving video search efficiency.
[0138] In the synthesis process, the target video data and the extracted video sub-segment may be synthesized according to the shooting time of the target video data and the extracted video sub-segment and the chronological order of the shooting time. Alternatively, the target video data and the extracted video sub-segment may be synthesized according to the image frame numbers of the target video data and the extracted video sub-segment and the chronological order of the shooting. Alternatively, the target video data and the extracted video sub-segment may also be synthesized according to the shooting time of the target video data and the extracted video sub-segment and the shooting time of the target video data and the extracted video sub-segment.
[0139] Furthermore, image information of a video image corresponding to a facial action belonging to a specified category is obtained, and a position of a camera that captured the facial action belonging to the specified category is determined based on the image information, the determined position of the camera is sent to a specified terminal, and / or the determined position of the camera is added to the image information and then displayed.
[0140] The image information may include but is not limited to the information of the camera, such as the camera number, the location of the camera, etc. As an example, the location of the camera may be represented by a coordinate or by specific address information, such as the south section of No. 1 Street, District 1, Gate 1 of Shopping Mall 1, etc.
[0141] The designated terminal may be a security device, a device held by an emergency contact, or an alarm, etc.
[0142] The image information corresponding to the video image of the facial action that is an abnormal action is obtained, that is, the number and / or location information of the camera that shot the video image is obtained. By determining the location of the camera, the location where the abnormal action occurred can be determined. After determining the location of the camera, the location of the camera can be sent to a designated terminal for display, or the location of the camera can be added to the image information and then displayed, or the location of the camera can be sent to a designated terminal and the location information can be added to the image information and then displayed. For example, only one location information can be displayed on the designated terminal, such as "the shooting location is Gate 1 of Mall 1".
[0143] Furthermore, the image information with the location information of the camera added can be displayed on the video image with the abnormal action, so that the location where the abnormal action occurs can be known based on the video image corresponding to the abnormal action.
[0144] In an embodiment of the present application, a plurality of color face images and corresponding depth face images associated with the current video clip are obtained. The plurality of color face images and the corresponding depth face images are input into a target network model, and the target network model determines the facial action of the target based on the plurality of color face images and the corresponding depth face images. When the facial action is an abnormal action, the target video data is obtained. In this way, for any video clip, the facial action of the target can be automatically determined by the target network model based on the color face image and the depth face image, thereby determining whether the facial action is an abnormal action, and then locating the target video data with abnormal action, avoiding the need for manual investigation of multiple video clips, thereby improving the efficiency of determining video clips with abnormal behavior and improving the accuracy of detection.
[0145] Figure 51 is a schematic diagram of a facial action detection device according to an exemplary embodiment. The image display device can be implemented by software, hardware or a combination of the two. The facial action detection device may include:
[0146] A first acquisition module 510 acquires a plurality of color face images and corresponding depth face images associated with a current video clip;
[0147] A determination module 520 determines the facial action of the target through a target network model based on the multiple color face images and the corresponding depth face images;
[0148] The second acquisition module 530 acquires target video data when it is determined that the target facial action is an abnormal action, wherein the target video data includes the current video clip and / or a video image corresponding to the facial action that is an abnormal action.
[0149] In a possible implementation of the present application, the first acquisition module 510 is used to:
[0150] Acquire a plurality of color video images and corresponding depth video images associated with the video clip and including the target;
[0151] Performing face detection on the acquired multiple color video images and corresponding depth video images respectively;
[0152] Determine, according to the face detection result, a plurality of face color region images and corresponding face depth region images from the plurality of color video images and corresponding depth video images;
[0153] The multiple face color area images and the corresponding face depth area images are determined as the multiple color face images and the corresponding depth face images.
[0154] In a possible implementation of the present application, the first acquisition module 510 is used to:
[0155] Performing face alignment processing on a plurality of face color region images and corresponding face depth region images respectively;
[0156] Adjusting the sizes of the multiple face color region images and the corresponding face depth region images after face alignment processing to the same size;
[0157] The multiple face color area images and the corresponding face depth area images after resizing are used as the multiple color face images and the corresponding depth face images.
[0158] In a possible implementation of the present application, the target network model includes a feature fusion network and a multi-frame analysis network, and the determination module 520 is used to:
[0159] Inputting the plurality of color face images and the corresponding depth face images into the target network model, extracting key features of each face image and the corresponding depth face image through the feature fusion network in the target network model and fusing them to obtain fusion features corresponding to each face image;
[0160] The obtained multiple fusion features are analyzed and processed by a multi-frame analysis network in the target network model to determine the facial action of the target.
[0161] In a possible implementation of the present application, the determining module 520 is further configured to:
[0162] classifying facial movements of the target;
[0163] Determine whether the facial action is an abnormal action according to the classification result.
[0164] In a possible implementation of the present application, the second acquisition module 530 is used to:
[0165] Extracting, from other video segments including the target, video subsegments corresponding to the facial action or extracting video subsegments corresponding to the action categories to which the facial action belongs;
[0166] synthesizing the target video data and the extracted video sub-segment into a video according to shooting time of the target video data and the extracted video sub-segment and / or image frame numbers of the target video data and the extracted video sub-segment;
[0167] Play the synthesized video.
[0168] In a possible implementation of the present application, the second acquisition module 530 is further configured to:
[0169] Obtaining image information of a video image corresponding to a facial action belonging to a specified category;
[0170] Determine the position of the camera that captures the facial action belonging to the specified category according to the image information;
[0171] The determined position of the camera is sent to a designated terminal, and / or the determined position of the camera is added to the image information for display.
[0172] In a possible implementation of the present application, the target network model is obtained by training the network model to be trained based on multiple face color image samples and corresponding face depth image samples, as well as actual facial action categories of the faces in the multiple face image samples.
[0173] In an embodiment of the present application, a plurality of color face images and corresponding depth face images associated with the current video clip are obtained. The plurality of color face images and the corresponding depth face images are input into a target network model, and the target network model determines the facial action of the target based on the plurality of color face images and the corresponding depth face images. When the facial action is an abnormal action, the target video data is obtained. In this way, for any video clip, the facial action of the target can be automatically determined by the target network model based on the color face image and the depth face image, thereby determining whether the facial action is an abnormal action, and then locating the target video data with abnormal action, avoiding the need for manual investigation of multiple video clips, thereby improving the efficiency of determining video clips with abnormal behavior and improving the accuracy of detection.
[0174] It should be noted that: the facial action detection device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when implementing the facial action detection method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the facial action detection device provided in the above embodiment and the facial action detection method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0175] Figure 6 It is a structural diagram of a control device 600 provided in an embodiment of the present application. The control device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 and one or more memories 602, wherein the memory 602 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 601 to implement the facial action detection method provided in the above-mentioned various method embodiments.
[0176] Of course, the control device 600 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The control device 600 may also include other components for realizing device functions, which will not be elaborated here.
[0177] The embodiment of the present application also provides a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal can execute the above Figure 1 The illustrated embodiment provides a facial action detection method.
[0178] The present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the above Figure 1 The illustrated embodiment provides a facial action detection method.
[0179] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0180] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A facial action detection method, characterized in that: The method comprises: Provide a user interaction interface; determining a target specified by a user through the user interaction interface; Acquire multiple color video images and corresponding depth video images associated with a current video clip and including the target, wherein the current video clip is any video clip currently captured in real time, or the current video clip is any video clip among a plurality of pre-stored video clips; Performing face detection on the acquired multiple color video images and corresponding depth video images respectively; Determine, according to the face detection result, a plurality of face color region images and corresponding face depth region images from the plurality of color video images and corresponding depth video images; Determining the plurality of face color area images and the corresponding face depth area images as the plurality of color face images and the corresponding depth face images; For each color face image and the corresponding depth face image, the color face image and the corresponding depth face image are input into a feature fusion network, a convolution operation is performed on the color face image and the corresponding depth face image through the feature fusion network, the color features of the color face image and the depth features of the depth face image are extracted, the color features and the depth features are connected, and the color features of the eyes, mouth, and eyebrows in the color face image are selected as the first key features for the connected color features by a fully connected layer, and the depth features of the eyes, mouth, and eyebrows in the depth face image are selected as the second key features for the connected depth features, and the first key features are fused with the second key features, and the fused features corresponding to the multiple face images are respectively input into a multi-frame analysis network, and the facial movements of the target are obtained by analyzing the facial motion units through the multi-frame analysis network, wherein the facial motion units are used to indicate the movements of the eyes, mouth, and eyebrows; When it is determined that the facial action of the target is an abnormal action, acquiring target video data, the target video data including the current video clip and / or a video image corresponding to the facial action belonging to the abnormal action; Extracting, from other video segments including the target, video subsegments corresponding to the facial action or extracting video subsegments corresponding to the action categories to which the facial action belongs, and adding image information corresponding to the video subsegment to each extracted video subsegment, wherein the image information corresponding to the video subsegment includes the position of a camera that shoots the video subsegment; synthesizing the target video data and the extracted video sub-segment into a video according to shooting time of the target video data and the extracted video sub-segment and / or image frame numbers of the target video data and the extracted video sub-segment; Play the synthesized video.
2. The method according to claim 1, characterized in that The step of determining the plurality of face color region images and the corresponding face depth region images as the plurality of color face images and the corresponding depth face images comprises: Performing face alignment processing on a plurality of face color region images and corresponding face depth region images respectively; Adjusting the sizes of the multiple face color region images and the corresponding face depth region images after face alignment processing to the same size; The multiple face color area images and the corresponding face depth area images after resizing are used as the multiple color face images and the corresponding depth face images.
3. The method according to claim 2, characterized in that After determining the facial action of the target, the method further includes: classifying facial movements of the target; Determine whether the facial action is an abnormal action according to the classification result.
4. The method according to claim 1, characterized in that The method further comprises: Obtaining image information of a video image corresponding to a facial action belonging to a specified category; Determine the position of the camera that captures the facial action belonging to the specified category according to the image information; The determined position of the camera is sent to a designated terminal, and / or the determined position of the camera is added to the image information for display.
5. The method according to claim 1, characterized in that The feature fusion network and the multi-frame analysis network are obtained by training the network model to be trained based on multiple face color image samples and corresponding face depth image samples, as well as actual facial action categories of the faces in the multiple face image samples.
6. A facial action detection device, characterized in that: The device comprises: The first acquisition module provides a user interaction interface; determines a target specified by the user through the user interaction interface; acquires multiple color video images and corresponding depth video images associated with the current video clip and including the target, wherein the current video clip is any video clip currently shot in real time, or the current video clip is any video clip among multiple pre-stored video clips; performs face detection on the acquired multiple color video images and corresponding depth video images respectively; determines multiple face color area images and corresponding face depth area images from the multiple color video images and corresponding depth video images according to the face detection result; determines the multiple face color area images and corresponding face depth area images as the multiple color face images and corresponding depth face images; A determination module, for each color face image and the corresponding depth face image, inputs the color face image and the corresponding depth face image into a feature fusion network, performs a convolution operation on the color face image and the corresponding depth face image through the feature fusion network, extracts the color features of the color face image and the depth features of the depth face image, performs a connection operation on the color features and the depth features, selects the color features of the eyes, mouth, and eyebrows in the color face image as the first key features for the connected color features by a fully connected layer, selects the depth features of the eyes, mouth, and eyebrows in the depth face image as the second key features for the connected depth features, and fuses the first key feature with the second key feature, inputs the fused features corresponding to the multiple face images respectively into a multi-frame analysis network, and obtains the facial movements of the target by analyzing the facial motion units through the multi-frame analysis network, wherein the facial motion units are used to indicate the movements of the eyes, mouth, and eyebrows; A second acquisition module, when determining that the facial action of the target is an abnormal action, acquires target video data, wherein the target video data includes the current video clip and / or a video image corresponding to the facial action belonging to the abnormal action; Extracting, from other video segments including the target, video subsegments corresponding to the facial action or extracting video subsegments corresponding to the action categories to which the facial action belongs, and adding image information corresponding to the video subsegment to each extracted video subsegment, wherein the image information corresponding to the video subsegment includes the position of a camera that shoots the video subsegment; synthesizing the target video data and the extracted video sub-segment into a video according to shooting time of the target video data and the extracted video sub-segment and / or image frame numbers of the target video data and the extracted video sub-segment; Play the synthesized video.
7. A monitoring system, characterized in that: The monitoring system includes a processor and at least one camera, wherein the processor is used to: Providing a user interaction interface; determining a target specified by a user through the user interaction interface; Acquire multiple color video images and corresponding depth video images associated with a current video clip and including the target, wherein the current video clip is any video clip currently captured in real time, or the current video clip is any video clip among a plurality of pre-stored video clips; Performing face detection on the acquired multiple color video images and corresponding depth video images respectively; Determine, according to the face detection result, a plurality of face color region images and corresponding face depth region images from the plurality of color video images and corresponding depth video images; Determining the plurality of face color area images and the corresponding face depth area images as the plurality of color face images and the corresponding depth face images; For each color face image and the corresponding depth face image, the color face image and the corresponding depth face image are input into a feature fusion network, a convolution operation is performed on the color face image and the corresponding depth face image through the feature fusion network, the color features of the color face image and the depth features of the depth face image are extracted, the color features and the depth features are connected, and the color features of the eyes, mouth, and eyebrows in the color face image are selected as the first key features for the connected color features by a fully connected layer, and the depth features of the eyes, mouth, and eyebrows in the depth face image are selected as the second key features for the connected depth features, and the first key features are fused with the second key features, and the fused features corresponding to the multiple face images are respectively input into a multi-frame analysis network, and the facial movements of the target are obtained by analyzing the facial motion units through the multi-frame analysis network, wherein the facial motion units are used to indicate the movements of the eyes, mouth, and eyebrows; When it is determined that the facial action of the target is an abnormal action, acquiring target video data, the target video data including the current video clip and / or a video image corresponding to the facial action belonging to the abnormal action; Extracting, from other video segments including the target, video subsegments corresponding to the facial action or extracting video subsegments corresponding to the action categories to which the facial action belongs, and adding image information corresponding to the video subsegment to each extracted video subsegment, wherein the image information corresponding to the video subsegment includes the position of a camera that shoots the video subsegment; synthesizing the target video data and the extracted video sub-segment into a video according to shooting time of the target video data and the extracted video sub-segment and / or image frame numbers of the target video data and the extracted video sub-segment; Play the synthesized video.
8. The monitoring system according to claim 7, characterized in that: The processor is further configured to: When the at least one camera includes a red, green, blue, depth (RGBD) camera, acquiring a color video image through the RGBD camera and acquiring a corresponding depth video image in the presence of infrared light; or, When the at least one camera includes two red, green, and blue (RGB) cameras, the two red, green, and blue (RGB) cameras respectively capture color video images, and determine the corresponding depth video image according to the color video images respectively captured by the two RGB cameras; or, When the at least one camera includes an RGB camera and a depth camera, a color video image is acquired through the RGB camera, and a depth video image is acquired through the depth camera.
Citation Information
Patent Citations
Interaction method and interaction device based on lip language
CN106774856A
Expression identification method fusing depth image and multi-channel features
CN106778506A
Virtual learning environment natural interaction method based on multimode emotion recognition
CN106919251A
Facial expression capturing method and device and storage device
CN107368778A
Human face detection method and device
CN107368810A
