Program, device and method for extracting frames for recognizing instantaneous state of object

Abstract

To provide a program that extracts a frame for recognizing an instantaneous state of an object which is reflected in a video, or the like.SOLUTION: As a training stage, normal distribution having a frame, in which intensity to be in a state is highest, as a center of a time sequence as likelihood 1 is created from a plurality of frames of a time sequence, in which an instantaneous state of an object is reflected, a frame is defined as an explanatory variable using teacher data to which the likelihood of normal distribution is given for each frame, and the likelihood is functioned as a machine learning engine trained as an objective variable. And, as an estimation stage, the machine learning engine is functioned as normal distribution recurrence means in which the plurality of frames of the time sequence having the instantaneous state of the object reflected are input, the likelihood is estimated for each frame, and the normal distribution is reproduced from each likelihood for the plurality of frames of the time sequence, and frame extraction means that extracts the plurality of frames included in ± predetermined sections in the normal distribution as recognition frames.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for extracting frames for recognizing the instantaneous state of an object captured in a captured video, and is particularly suitable for recognizing human facial expressions. [Background technology]

[0002] In recent years, HCI (Human-Computer Interaction) technology has been attracting attention as a way to improve communication between people and computers in accordance with the social environment. This technology analyzes physical characteristics based on a person's psychology. At this time, it is necessary to instantly detect changes in a person's psychological state as changes in facial expressions.

[0003] Conventionally, there is a technology for recognizing a person's facial expression in real time (see, for example, Patent Document 1). According to this technology, first, the positions of the eyes and mouth are estimated from a facial image captured so that the center of gravity of the skin-colored area coincides with the center of the input image. The inclination angle of the face is estimated based on the positions of both eyes, and the facial image in the detection area is rotated by that angle. For the rotated detection area, a two-dimensional discrete cosine transform is performed to obtain the change in spatial frequency components from when a neutral expression was detected. Then, using parameters learned in advance by a genetic algorithm, the change in spatial frequency components is converted into a deformation of a three-dimensional face model to reproduce the facial expression.

[0004] There is also a technique for detecting points of change in a user's facial expression from the amount of deformation of multiple parts of the user's face (see, for example, Patent Document 2). According to this technique, the amount of deformation of multiple parts is acquired as AU (Action Unit) feature values. These AU feature values ​​are subjected to principal component analysis, and average values ​​are calculated for each predetermined time from the coefficients of each principal component. The point in time when the user's facial expression changed is detected based on the average values.

[0005] There is also a technique for detecting the timing of the start (Onset) and end (Offset) of a facial expression from AU features detected from a face image using a deep learning model (see, for example, Non-Patent Document 1). There is also a technology that uses a deep learning model to detect the timing of the start (onset) and end (offset) of a facial expression from optical flow features detected from a face image (see, for example, Non-Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2000-268161 A [Patent Document 2] JP 2018-036734 A [Non-patent literature]

[0007] [Non-Patent Document 1] Bo Yang, Jianming Wu, Gen Hattori, etc. “Facial Action Unit-based Deep Learning Framework for Spotting Macro- and Micro-expressions in Long Video Sequences”, ACM Multimedia conference(2021). [Non-Patent Document 2] Gen-Bing Liong, John See, Lai-Kuan Wong, “Shallow Optical Flow Three-Stream CNN for Macro- and Micro-expression Spotting from Long Videos”, IEEE ICIP conference(2021). Summary of the Invention [Problem to be solved by the invention]

[0008] The technology described in Patent Document 1 does not detect the timing of the start (Onset) and end (Offset) of a facial expression when it occurs, nor does it detect changes in features other than the eyes and mouth. Furthermore, according to the techniques described in Patent Document 2 and Non-Patent Documents 1 and 2, the criteria for determining the timing of facial expression changes are set in advance as assumptions, and optimization is not achieved for a variety of facial expression changes. Furthermore, all of the above-mentioned prior art documents are limited to the use of a single feature amount, and do not extract a dominant feature amount in response to various changes in facial expression.

[0009] As a preliminary step in recognizing a person's facial expressions using a facial expression recognition engine, it is important to detect the timing of the start (Onset) and end (Offset) of a facial expression. Only the frames between the start and end can be targeted for facial expression recognition. In other words, there is no need to process other frames for facial expression recognition. The inventors of the present application considered that, in contrast to the prior art described above, the criteria for determining the timing of facial expression changes must be optimized for a variety of facial expression changes, and that it is not desirable to rely on the superiority of a single feature.

[0010] Therefore, an object of the present invention is to provide a program, device, and method for extracting frames for recognizing the instantaneous state of an object captured in an image. [Means for solving the problem]

[0011] According to the present invention, there is provided a program for causing a computer to function to extract a frame for recognizing an instantaneous state of an object, the program comprising: As a training stage, The above From multiple frames of a time series that capture the instantaneous state of the object, a normal distribution is created with the frame with the highest intensity of the state as the center of the time series, with a likelihood of 1. The above Using training data with normal distribution likelihood, The aboveThe frame is the explanatory variable, The normal distribution A machine learning engine trained on likelihood as the objective variable and make it work. As an estimation stage, The above The machine learning engine inputs multiple frames of a time series that capture the instantaneous state of the object, estimates the likelihood for each frame, and A normal distribution regression means for reconstructing a normal distribution from the likelihoods for multiple frames of the time series; The reconstructed data is then reconstructed by the normal distribution regression means. A frame extraction means for extracting a plurality of frames included in a ±predetermined range in the normal distribution as recognition frames; The present invention is characterized in that it causes a computer to function.

[0012] According to another embodiment of the program of the present invention, The above The instantaneous state of the object is the state of a human face, which changes from no expression to an expression to no expression, Training data for the machine learning engine About The above With status, The above With facial expression It is also preferable to function the computer in such a manner.

[0013] According to another embodiment of the program of the present invention, A facial expression recognition engine that inputs frames containing human faces and estimates facial expression labels. and make it work. The above The facial expression recognition engine The above Only the recognition frames extracted by the frame extraction means are input, and facial expression labels are estimated. It is also preferable to function the computer in such a manner.

[0014] According to another embodiment of the program of the present invention, The reconstructed data is then reconstructed by the normal distribution regression means. Among the frames included in the ± predetermined interval of the normal distribution, the frame on the most negative side is defined as the onset frame, and the frame on the most positive side is defined as the offset frame. a state duration detection means for detecting a difference between a shooting time of the onset frame and a shooting time of the offset frame as a state duration; It is also preferable to make the computer function as follows.

[0015] According to another embodiment of the program of the present invention, The frame extraction means The ±specified interval of the normal distribution is ±2σ It is also preferable to function the computer in such a manner.

[0016] According to another embodiment of the program of the present invention, The above The machine learning engine takes as input a time series of frames, From the specified time A macro feature extraction means for extracting macro features predicted over a long sequence; The predetermined time or less A micro feature extraction means for extracting predicted micro features from a short sequence; The above Macro features and The above a fusion layer that outputs a fusion feature obtained by fusing the micro feature; The above An error backpropagation means trained to output the likelihood of the normal distribution from the fusion feature; It is also preferable to configure the computer to have the following:

[0017] According to another embodiment of the program of the present invention, The above The macro feature extraction means is an AU (Action Units) feature extraction engine of a deep learning model, The above The micro feature extraction method is the OpticalFlow feature extraction engine of the deep learning model. It is also preferable to function the computer in such a manner.

[0018] According to the present invention, in an apparatus for extracting frames for recognizing a momentary state of an object, As a training stage, The above From multiple frames of a time series that capture the instantaneous state of the object, a normal distribution is created with the frame with the highest intensity of the state as the center of the time series, with a likelihood of 1. The above Using training data with normal distribution likelihood, The above The frame is the explanatory variable, The normal distribution A machine learning engine trained on likelihood as the objective variable having As an estimation stage, The above The machine learning engine inputs multiple frames of a time series that capture the instantaneous state of the object, estimates the likelihood for each frame, and A normal distribution regression means for reconstructing a normal distribution from the likelihoods for multiple frames of the time series; The reconstructed data is then reconstructed by the normal distribution regression means. A frame extraction means for extracting a plurality of frames included in a ±predetermined range in the normal distribution as recognition frames; The present invention is characterized by having the following.

[0019] According to the present invention, there is provided a method for extracting a recognition frame for an apparatus for extracting a frame for recognizing an instantaneous state of an object, comprising the steps of: The device is As a training stage, The above From multiple frames of a time series that capture the instantaneous state of the object, a normal distribution is created with the frame with the highest intensity of the state as the center of the time series, with a likelihood of 1. The above Machine learning engine using training data with normal distribution likelihood of , The above The frame is the explanatory variable, The normal distribution First step: training on likelihood as the objective variable Run As an estimation stage, The above A 21st step in which the machine learning engine inputs a plurality of frames of a time series showing a momentary state of the object and estimates a likelihood for each frame; A 22nd step of reconstructing a normal distribution from the likelihoods for multiple frames of the time series; The regenerated product obtained by the 22nd step A 23rd step of extracting a plurality of frames included in a ±predetermined interval in the normal distribution as recognition frames; Run It is characterized by: Effect of the Invention

[0020] According to the program, device, and method of the present invention, the present invention can extract a frame for recognizing the instantaneous state of an object captured in an image. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing a system configuration having an application server according to the present invention. [Diagram 2] FIG. 2 is a functional configuration diagram of a recognition frame extraction unit in the present invention. [Diagram 3] FIG. 2 is an explanatory diagram showing teacher data in a training stage in the present invention. [Figure 4] FIG. 2 is an explanatory diagram of a machine learning engine and a normal distribution regression unit in the estimation stage in the present invention. [Diagram 5] FIG. 2 is an explanatory diagram showing a recognition frame to be input to a facial expression recognition engine. [Figure 6] FIG. 2 is a functional configuration diagram of a machine learning engine according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0023] FIG. 1 is a diagram showing the configuration of a system having an application server according to the present invention.

[0024] According to the system of Fig. 1, the application server 1 functions as a server connected to the Internet. The application server 1 receives video containing a human face from a terminal 2 operated by a user. Then, the application server 1 recognizes the human facial expression from the video and processes it using various applications.

[0025] According to FIG. 1, the application server 1 includes a recognition frame extraction unit 11 and a facial expression recognition engine 12. The recognition frame extraction unit 11 is an essential functional component of the present invention, and extracts frames for recognizing the instantaneous state of an object captured in a video. The "instantaneous state of an object" may be, for example, a person's facial expression, but is of course not limited to this. The facial expression recognition engine 12 is a constructed learning model that is pre-trained by a teacher video. The facial expression recognition engine 12 inputs only a plurality of recognition frames that include a human face, which are extracted by the recognition frame extraction unit 11, and estimates a facial expression label. The facial expression recognition engine 12 may be an existing one.

[0026] The terminal 2 is equipped with a camera and transmits an image of a person's face to the application server 1. The terminal 2 is a smartphone or a mobile terminal carried by each user, and is connected to an access network such as a mobile phone network or a wireless LAN. Of course, the terminal 2 is not limited to a smartphone, but may be, for example, a web camera installed in a home. Also, video data captured by the web camera may be recorded on an SD card, and the recorded video data may be input to the application server 1.

[0027] FIG. 2 is a functional block diagram of the recognition frame extraction unit in the present invention.

[0028] The recognition frame extraction unit 11 extracts a frame in which the instantaneous state of an object (for example, a person's facial expression) should be recognized. 2, the recognition frame extraction unit 11 includes a machine learning engine 111, a normal distribution regression unit 112, a frame extraction unit 113, and a state duration detection unit 114. These functional components are realized by executing a program that causes a computer installed in the server to function. The process flow of these functional components can also be understood as a recognition frame extraction method.

[0029] [Machine Learning Engine 111] The machine learning engine 111 consists of a <training stage> using <teacher data> and an <estimation stage>.

[0030] <Teacher data> FIG. 3 is an explanatory diagram showing teacher data in the training stage in the present invention.

[0031] First, from multiple frames of a time series that capture the instantaneous state of the object, which serve as training data, a normal distribution (or Gaussian function) is created with the "apex frame" with the highest intensity of indicating a state as the center of the time series, with a likelihood of 1.

[0032] Here, in the case of facial expression recognition, "state is present" means "expression is present." In the case of a human face, the "instantaneous state of the object" refers to a state that changes from no expression to an expression to no expression again. Moreover, the "frame with the highest intensity of a state present" refers to a frame in which some facial expression (laughing, anger, crying, etc.) is most strongly detected within a given time period (for example, one second). For example, if a video is one second long, and there are 10 frames per second, each frame is 10 ms. Each frame changes as follows: No facial expression (high frame with low intensity of state) ↓ With expression (frame with highest intensity of with state) ↓ No facial expression (the frame with the lowest intensity of the expression)

[0033] This is centered on the "frame with the highest intensity of having a state," with the preceding part being in the process of changing from no expression to an expression, and the following part being in the process of changing from an expression to no expression. Thinking about it in this way, it can be considered as a normal distribution with the "frame with the highest intensity of having a state" as the average value within a given time period and the likelihood as the intensity of having a state. Likelihood 0: No facial expression (lowest intensity of facial expression) ↓ Likelihood 1: Facial expression (highest intensity of facial expression) ↓ Likelihood 0: No facial expression (lowest intensity of facial expression)

[0034] The normal distribution describes a probability distribution in which data clusters around a mean value. N(μ,σ 2 )=1 / √(2πσ 2 )·exp(-(x-μ) 2 / (2σ 2 )) μ: Average σ 2 :Dispersion N: Normal distribution (Gaussian function) Here, the frame with the mean μ represents the frame with the highest strength of the presence state. Then, for each frame, a real number with a normal distribution likelihood [0,1] is assigned and this is defined as "teaching data."

[0035] It should be noted here that, unlike Non-Patent Documents 1 and 2, a time window is not required, and the intensity (likelihood) of a facial expression is assigned as a label for each frame.

[0036] <Training stage> In the training stage, the machine learning engine 111 inputs teacher data and performs training. That is, the machine learning engine 111 performs training using a frame (an image in which a human face is reflected) as an explanatory variable and a likelihood (the stronger the intensity of the presence of a facial expression, the closer it is to 1) as a target variable. In this way, internal parameters are constructed.

[0037] <Estimation stage> FIG. 4 is an explanatory diagram of the machine learning engine and normal distribution regression unit in the estimation stage in the present invention.

[0038] The machine learning engine 111 inputs a plurality of time-series frames that capture the instantaneous state (facial expression) of a target object (human face) as an estimation target, and estimates the likelihood (intensity of the presence of a facial expression) for each frame. The estimated likelihoods are sequentially output to the normal distribution regression unit 112 .

[0039] [Normal Distribution Regression Part 112] The normal distribution regression unit 112 reconstructs a normal distribution from the likelihoods for multiple frames in a time series. According to Fig. 5, a normal distribution with the likelihood on the vertical axis is reconstructed.

[0040] [Frame extraction unit 113] The frame extraction unit 113 extracts a plurality of frames included in a predetermined ± interval in the normal distribution as recognition frames. These recognition frames are output to the facial expression recognition engine 12. The facial expression recognition engine 12 is required to analyze facial expressions only for these recognition frames.

[0041] FIG. 5 is an explanatory diagram showing a recognition frame to be input to the facial expression recognition engine.

[0042] Among the frames included in the ±predetermined interval of the normal distribution, the frame furthest on the negative side is the onset frame (start of the facial expression change), and the frame furthest on the positive side is the offset frame (end of the facial expression change). Here, the ±predetermined interval of the normal distribution may be ±2σ. In other words, this means that the frame with the strongest intensity of facial expression is included within this predetermined interval.

[0043] It should be noted here that there is no need to set any threshold when detecting frames with high facial expression intensity. In particular, instead of looking at the likelihood of each frame, after reproducing the normal distribution, only a number of frames included in a predetermined section (frames between the onset frame and the offset frame) are output to the facial expression recognition engine 12.

[0044] [State duration detection unit 114] The state duration detection unit 114 detects the difference between the shooting time of the onset frame and the shooting time of the offset frame as the state duration. The state duration is output to the facial expression recognition engine 12. The facial expression recognition engine 12 only needs to analyze facial expressions from the video for the duration of that state.

[0045] FIG. 6 is a functional configuration diagram of a machine learning engine in the present invention.

[0046] The machine learning engine 111 includes a macro feature extraction unit 1111 , a micro feature extraction unit 1112 , a fusion layer 1113 , and an error backpropagation unit 1114 . The time-series frames input to the machine learning engine 111 are input to a macro feature extraction unit 1111 and a micro feature extraction unit 1112 .

[0047] [Macro feature extraction unit 1111] The macro feature extraction unit 1111 extracts macro features predicted from a relatively long sequence. Specifically, the macro feature extraction unit 1111 is an AU (Action Units) feature extraction engine of a deep learning model that has an advantage in extracting macro features. The AU feature represents a change in facial expression as a feature by combining the movements of multiple parts (eyebrows, cheeks, chin, etc.) based on facial muscles. This represents a frame with a relatively long duration of about 0.5 to 4 seconds on a human face, and with a relatively high intensity of facial expression. In general, the AU feature is used as a Facial Action Coding System (FACS).

[0048] [Micro feature extraction unit 1112] The micro feature extraction unit 1112 extracts predicted micro features from a relatively short sequence. Specifically, the micro feature extraction unit 1112 is an OpticalFlow feature extraction engine of a deep learning model that has an advantage in extracting micro features. OpticalFlow features are vector representations of the displacement of the same feature points between adjacent frames caused by changes in the object. They represent the features of frames with relatively short facial expression durations of less than 0.5 seconds and relatively low intensity of facial expressions on a human face. OpticalFlow, for example, configures Two-stream ConvNets by combining a spatial stream ConvNet (CNN) and a temporal stream ConvNet (CNN). When capturing facial expressions in images, the brightness (appearance) does not change between consecutive frames and adjacent pixels move in a similar way, making them suitable for OpticalFlow.

[0049] [Fusion layer 1113] The fusion layer 1113 outputs a fusion feature obtained by fusing a macro feature and a micro feature. By fusing a macro feature and a micro feature, the advantages of both features can be self-adaptively learned. The fusion layer 1113 may combine the macro-features and the micro-features by a self-adaptive deep learning model such as TextCNN / GRU / LSTM / Transformer, etc. The fusion layer 1113 can self-adaptively adjust the attention weights given to both features, and can optimize the prediction results.

[0050] [Backpropagation unit 1114] The error backpropagation unit 1114 receives the fusion feature amount as input and is trained to output the likelihood of a normal distribution that is the correct answer.

[0051] By incorporating the machine learning engine 111 in FIG. 6 into the recognition frame extraction unit in FIG. 2, it is possible to optimize the detection accuracy of the timing of the start and end of facial expressions, taking into account both macro- and micro-facial expressions.

[0052] As described above in detail, the program, device, and method of the present invention make it possible to extract frames for recognizing the instantaneous state of an object captured in an image.

[0053] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0054] 1 Application Server 11 Recognition frame extraction section 111 Machine Learning Engine 1111 Macro feature extraction unit 1112 Micro feature extraction unit 1113 Fusion layer 1114 Error Backpropagation Unit 112 Normal Distribution Regression Section 113 Frame Extraction Unit 114 State duration detector 12 Facial Expression Recognition Engine 2. Terminal

Claims

1. A program for causing a computer to extract a frame for recognizing an instantaneous state of an object, As a training stage, a machine learning engine that creates a normal distribution from a plurality of frames in a time series that reflects the instantaneous state of the object, with the frame with the highest intensity of the state being set as the center of the time series with a likelihood of 1, and trains the machine learning engine by using training data in which the likelihood of the normal distribution is assigned to each frame as an explanatory variable and the likelihood of the normal distribution as a target variable; and make it work. As an estimation stage, The machine learning engine inputs a plurality of time-series frames that capture the instantaneous state of an object, and estimates a likelihood for each frame; A normal distribution regression means for reconstructing a normal distribution from the likelihoods for multiple frames of the time series; a frame extraction means for extracting, as recognition frames, a plurality of frames included in a ±predetermined interval in the normal distribution reproduced by the normal distribution regression means; A program for causing a computer to function.

2. The instantaneous state of the object is, for a human face, a state that changes from no expression to an expression to no expression, Regarding the training data of the machine learning engine, the presence of the state means the presence of the facial expression.

2. The program according to claim 1, which causes a computer to function as described above.

3. A facial expression recognition engine that inputs frames containing human faces and estimates facial expression labels. and make it work. The facial expression recognition engine inputs only the recognition frames extracted by the frame extraction means and estimates facial expression labels.

3. The program according to claim 2, which causes a computer to function as described above.

4. Among the frames included in the ±predetermined interval of the normal distribution reproduced by the normal distribution regression means, the frame on the most negative side is defined as an onset frame, and the frame on the most positive side is defined as an offset frame, a state duration detection means for detecting a difference between a shooting time of the onset frame and a shooting time of the offset frame as a state duration; 4. The program according to claim 1, further comprising: a program for causing a computer to function as described above.

5. The ±predetermined interval of the normal distribution in the frame extraction means is ±2σ.

5. The program according to claim 4, which causes a computer to function as described above.

6. The machine learning engine inputs a time series of frames, a macro feature extraction means for extracting a macro feature predicted from a sequence longer than a predetermined time; a micro feature extraction means for extracting a micro feature predicted from a short sequence equal to or shorter than the predetermined time; a fusion layer that outputs a fusion feature obtained by fusing the macro feature and the micro feature; an error backpropagation means trained to output the likelihood of the normal distribution from the fusion feature; 6. The program according to claim 1, which causes a computer to function so as to have the following:

7. The macro feature extraction means is an AU (Action Units) feature extraction engine of a deep learning model, The micro feature extraction means is an OpticalFlow feature extraction engine of a deep learning model.

7. The program according to claim 6, which causes a computer to function as described above.

8. In an apparatus for extracting a frame for recognizing a momentary state of an object, As a training stage, a machine learning engine that creates a normal distribution from a plurality of frames of a time series in which the instantaneous state of the object is captured, with the frame with the highest intensity of the state being set as the center of the time series with a likelihood of 1, and trains the machine learning engine by using training data in which the likelihood of the normal distribution is assigned to each frame as an explanatory variable and the likelihood of the normal distribution as a target variable; having As an estimation stage, The machine learning engine inputs a plurality of time-series frames that capture the instantaneous state of an object, and estimates a likelihood for each frame; A normal distribution regression means for reconstructing a normal distribution from the likelihoods for multiple frames of the time series; a frame extraction means for extracting, as recognition frames, a plurality of frames included in a ±predetermined interval in the normal distribution reproduced by the normal distribution regression means; An apparatus comprising:

9. A method for extracting a recognition frame for an apparatus for extracting a frame for recognizing an instantaneous state of an object, comprising: The device is As a training stage, A first step of creating a normal distribution from a plurality of frames of a time series in which the instantaneous state of the object is captured, with the frame with the highest intensity of the state being set as the center of the time series with a likelihood of 1, and training a machine learning engine using training data to which the likelihood of the normal distribution is assigned for each frame, with the frames as explanatory variables and the likelihood of the normal distribution as a target variable. Run As an estimation stage, A 21st step in which the machine learning engine inputs a plurality of time-series frames that capture a momentary state of an object and estimates a likelihood for each frame; A twenty-second step of reconstructing a normal distribution from the likelihoods for a plurality of frames of the time series; a 23rd step of extracting, as recognition frames, a plurality of frames included in a ±predetermined interval in the normal distribution reproduced by the 22nd step; Run A recognition frame extraction method comprising:

Images