Video processing system, video processing method, and program

Abstract

A moving image data acquisition unit (201) of a moving image processing system (1) acquires moving image data related to a moving image showing a face of a user who performs a predetermined action. A feature point detection unit (202) detects a plurality of feature points related to the face on the basis of the moving image data. An estimation unit (203) estimates whether or not a two-dimensional face is shown in the moving image data on the basis of position changes of each of the plurality of feature points in the moving image data and a learning model in which training data related to position changes of each of a plurality of feature points related to a face for training has been learned.

Description

[Technical Field]

[0001] This disclosure relates to a video processing system, a video processing method, and a program. [Background technology]

[0002] Conventionally, technologies have been considered to prevent impersonation by malicious actors. For example, Patent Document 1 describes a technology that performs facial recognition using a 3D camera capable of detecting depth and a thermal camera capable of detecting temperature. Even if a malicious actor attempts to impersonate someone by using a photograph with someone else's face printed on it, or a screen displaying someone else's face, it is believed that the 3D camera and thermal camera described in Patent Document 1 can prevent such impersonation. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-029968 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, conventional technologies required special hardware, such as the 3D camera and thermal camera described in Patent Document 1, which are not typically found in general computers, to estimate whether a 2D face is present in video data. Therefore, there is a need to estimate whether a 2D face is present in video data without requiring special hardware.

[0005] One of the purposes of this disclosure is to estimate whether or not a two-dimensional face is present in video data without requiring special hardware. [Means for solving the problem]

[0006] The video processing system according to the present disclosure includes a video data acquisition unit that acquires video data related to a video in which a user's face is shown, a feature point detection unit that detects a plurality of feature points related to the face based on the video data, and an estimation unit that estimates whether or not a two-dimensional face is shown in the video data based on the position changes of each of the plurality of feature points in the video data and a learning model in which training data related to the position changes of each of the plurality of feature points related to a training face is learned.

Advantages of the Invention

[0007] The present disclosure can estimate whether or not a two-dimensional face is shown in video data, for example, without requiring special hardware.

Brief Description of the Drawings

[0008] [Figure 1] It is a diagram showing an example of the hardware configuration of the video processing system. [Figure 2] It is a diagram showing an example of a screen displayed on the user terminal. [Figure 3] It is a diagram showing an example of a state where a malicious user impersonates someone. [Figure 4] It is a diagram showing an example of a function realized by the video processing system. [Figure 5] It is a diagram showing an example of a training database. [Figure 6] It is a diagram showing an example of a user database. [Figure 7] It is a diagram showing an example of processing for each individual frame of video data. [Figure 8] It is a diagram showing an example of processing for the entire video data. [Figure 9] It is a diagram showing an example of a process executed by the video processing system. [Figure 10] It is a diagram showing an example of a process executed by the video processing system. [Figure 11] It is a diagram showing an example of a function realized in a modified example. [Figure 12]This figure shows an example of a learning model for variation 1. [Modes for carrying out the invention]

[0009] [1. Hardware configuration of the video processing system] An example of an embodiment of the video processing system, video processing method, and program related to this disclosure will be described. Figure 1 is a diagram showing an example of the hardware configuration of the video processing system. For example, the video processing system 1 includes a learning terminal 10, a server 20, and a user terminal 30. Each of the learning terminal 10, server 20, and user terminal 30 is connected to a network N such as the Internet or a LAN.

[0010] The learning terminal 10 is a computer that performs training on the learning model described later. For example, the learning terminal 10 is a personal computer, a tablet, or a smartphone. For example, the learning terminal 10 includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, and a display unit 15. For example, the control unit 11 includes at least one processor. The storage unit 12 includes at least one of volatile memory such as RAM and non-volatile memory such as flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a touch panel or a mouse. The display unit 15 is a display such as a liquid crystal or organic EL.

[0011] Server 20 is a server computer that stores the trained model. For example, Server 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The hardware configurations of the control unit 21, the storage unit 22, and the communication unit 23 may be the same as those of the control unit 11, the storage unit 12, and the communication unit 13, respectively.

[0012] The user terminal 30 is the user's computer. For example, the user terminal 30 is a personal computer, smartphone, tablet, or wearable device. For example, the user terminal 30 includes a control unit 31, a storage unit 32, a communication unit 33, an operation unit 34, a display unit 35, and a camera unit 36. The hardware configuration of the control unit 31, storage unit 32, communication unit 33, operation unit 34, and display unit 35 may be the same as that of the control unit 11, storage unit 12, communication unit 13, operation unit 14, and display unit 15, respectively. The camera unit 36 ​​includes at least one camera.

[0013] Furthermore, programs stored in memory units 12, 22, and 32 may be supplied to the learning terminal 10, server 20, or user terminal 30 via the network N. Additionally, at least one of a reading unit (e.g., a memory card slot) for reading computer-readable information storage media and an input / output unit (e.g., a USB port) for inputting and outputting data to and from external devices may be included in the learning terminal 10, server 20, or user terminal 30. For example, a program stored on the information storage media may be supplied to the learning terminal 10, server 20, or user terminal 30 via at least one of the reading unit and the input / output unit.

[0014] Furthermore, the video processing system 1 may include at least one computer. The computers included in the video processing system 1 are not limited to the example in Figure 1. For example, the video processing system 1 may include only the learning terminal 10 and the server 20. In this case, the user terminal 30 is located outside the video processing system 1. The video processing system 1 may include only the server 20. In this case, the learning terminal 10 and the user terminal 30 are located outside the video processing system 1. The video processing system 1 may include only the server 20 and a computer not shown in Figure 1.

[0015] [2. Overview of the video processing system] In this embodiment, we take the example of a case where the user's face is captured by the camera unit 36 ​​for user identity verification. Identity verification is a process to confirm that the user is who they claim to be. For example, eKYC (electronic Know Your Customer) is a type of identity verification. Identity verification may be performed at any point in any service. For example, identity verification may be performed when a user uses a payment service, financial service, communication service, e-commerce service, administrative service, or other service, or when registering as a member.

[0016] For example, when a situation arises where user terminal 30 requires identity verification, it activates the camera unit 36. The camera unit 36 ​​continuously captures the user's face in video mode. Based on the results captured by the camera unit 36, the user terminal 30 generates video data showing the user's face. The user terminal 30 transmits the video data to the server 20. The server 20 performs identity verification processing based on the video data.

[0017] Figure 2 shows an example of a screen displayed on the user terminal 30. For example, the user terminal 30 displays a shooting screen SC, which shows the shooting results from the shooting unit 36, on the display unit 35. The shooting screen SC displays a guide G to guide the position of the face and a message MS indicating the action required of the user. In the example in Figure 2, the user is required to turn their face from the front to the left. The user adjusts the position and orientation of the user terminal 30 so that their face is within the guide G and performs the action indicated by the message MS.

[0018] Furthermore, the actions requested from the user may be arbitrary. The actions requested from the user are not limited to the example in Figure 2. For example, the user may be asked to turn their face upwards from the front, turn their face to the right from the front, turn their face downwards from the front, blink, wink, or perform other actions. As shown in the modified examples below, the action requested from the user may be randomly determined from among several actions. These actions are performed to prevent impersonation by malicious users.

[0019] For example, a malicious user could impersonate someone by taking a picture of a piece of paper with another person's face printed on it using their own user terminal 30. A malicious user could also impersonate someone by taking a picture of the first user terminal 30 displaying another person's face with a second user terminal 30. Even if the actions shown in Figure 2 are required, a malicious user may cleverly circumvent identity verification.

[0020] Figure 3 shows an example of how a malicious user might impersonate someone else. For example, a malicious user holds a piece of paper P with another person's face printed on it in front of their face, with the side showing the other person's face facing the camera unit 36. The malicious user has the camera unit 36 ​​of their user terminal 30 take a picture of the paper P. If the malicious user is asked to turn their face from the front to the left, they might try to circumvent identity verification by turning the paper P from the front to the left.

[0021] For example, if other actions are requested, a malicious user may attempt to circumvent identity verification by performing actions such as turning paper P straight up. It is believed that such impersonation can be prevented if the person in charge of identity verification checks the video data uploaded from the user terminal 30. However, when verifying the identity of a large number of users, it is not practical for the person in charge to check all the video data. Therefore, the video processing system 1 analyzes the video data to estimate whether or not a two-dimensional face is shown in the video data.

[0022] For example, if the imaging unit 36 ​​is a 3D camera, stereo camera, or thermal camera, the video processing system 1 can estimate whether a 2D face is shown in the video data by detecting the surface shape or surface temperature of the subject. However, in this case, special hardware such as a 3D camera, stereo camera, or thermal camera is required. A typical user terminal 30, such as a smartphone, does not include this special hardware.

[0023] Therefore, the video processing system 1 of this embodiment has a configuration that estimates whether or not a two-dimensional face is shown in the video data by analyzing the video data, without requiring any special hardware. For example, the positional changes of feature points detected from each frame of the video data differ depending on whether or not a three-dimensional face is shown in the video data or a two-dimensional face is shown in the video data. The video processing system 1 estimates whether or not a two-dimensional face is shown in the video data based on a learning model that has learned these features. The details of the video processing system 1 will be described below.

[0024] [3. Functions implemented by the video processing system] Figure 4 shows an example of the functions implemented by the video processing system 1.

[0025] [3-1. Functions implemented on learning devices] For example, the learning terminal 10 includes a data storage unit 100 and a learning unit 101. The data storage unit 100 is implemented by a storage unit 12. The learning unit 101 is implemented by a control unit 11.

[0026] [Data Storage Unit] The data storage unit 100 stores the data necessary for training the learning model M. For example, the data storage unit 100 stores the actual data of the learning model M and the training database DB1. The learning model M is a model created using machine learning techniques. Machine learning may be defined in various ways. In this embodiment, machine learning is a broad term that includes deep learning, reinforcement learning, and AI (Artificial Intelligence). For example, the learning model M may be a supervised learning, semi-supervised learning, or unsupervised learning model. In this embodiment, the learning model M is assumed to be a supervised learning model.

[0027] The learning model M can utilize various models used in the field of image analysis. For example, the learning model M may be a neural network, vision transformer, support vector machine, random forest, or other method. The actual data of the learning model M includes a program that performs various processes such as the calculation of embedding representations, and parameters referenced by the program. For example, the parameters are weight coefficients and biases. The program and parameters included in the actual data of the learning model M may be programs and parameters adopted in known methods.

[0028] Figure 5 shows an example of the training database DB1. The training database DB1 is a database that stores the training data used in training the learning model M. For example, the training data includes an input portion that is input to the learning model M during training, and an output portion that indicates the content that should be output from the learning model M during training. The output portion of the training data can also be called the correct answer during training. If the learning model M is an unsupervised learning model, the training data is not annotated, so the training data does not include the output portion.

[0029] For example, the input portion of the training data is position change data showing the positional changes of each of several feature points relating to the training face. The training face is a face shown in a video prepared for training the learning model M. Feature points are characteristic parts of the face. For example, a feature point is at least one pixel that indicates a part of the face. Feature points are sometimes called facial landmarks. In this embodiment, we take the example of a case where the vertices that make up a mesh showing the surface shape of the face correspond to feature points.

[0030] The method for detecting feature points may be a publicly known method. For example, feature points may be detected based on the Dlib library provided in Python, a type of programming language. Many libraries for feature point detection are also available in other programming languages, so feature points may be detected based on libraries of other programming languages. Alternatively, instead of using a programming language library, machine learning techniques such as deep learning (for example, a method called Deep3DFaceReconstruction or RingNet) may be used to detect feature points.

[0031] For example, the position change data corresponding to the input portion of the training data represents the position changes of multiple feature points detected from each of multiple frames of video data related to a video showing a face for training. Position change can also be described as velocity or displacement. In this embodiment, the position change data corresponding to the input portion of the training data is generated using the same calculation method as the estimation unit 203 described later. Details of this calculation method will be described later.

[0032] In this embodiment, we take the case where the position change data is in vector form as an example. For example, if the number of frames in the video data is m (where m is a natural number), and k (where k is a natural number) feature points are detected from one frame, then the position change data is a vector of at least m × k dimensions. Each element of the vector represented by the position change data indicates the position change of a particular feature point corresponding to a particular frame.

[0033] For example, the first to kth dimensions of the vector represented by the position change data show the position changes of each of the k feature points detected from the first frame. The first dimension represents the position change of a feature point indicating a specific location on the forehead, the second dimension represents the position change of the feature point to its right, the third dimension represents the position change of the feature point further to its right, and so on. It is predetermined which dimension represents the position change of which feature point. From the k+1th dimension onward, the position changes are shown from the first frame onward. The position changes are arranged in a predetermined order up to the last dimension of the vector represented by the position change data.

[0034] The position change data may be in any format. It may be data relating to the time-series positional changes of each of the multiple feature points in the video data. The position change data is not limited to vector format. For example, the position change data may be multiple numerical values, a single numerical value, an array, a matrix, or other format. These formats may represent the time-series positional changes of each of the multiple feature points in the video data.

[0035] For example, the output portion of the training data is a label indicating whether the input portion of the training data represents a two-dimensional face or not. A two-dimensional face is not a three-dimensional face. For example, if the training video data shows a piece of paper with a photograph of someone else's face printed on it, or a computer displaying someone else's face, the input portion of the training data will indicate a two-dimensional face. Since the surface of the paper or the surface of the computer screen has a two-dimensional plane, the position change data (the input portion of the training data) obtained from the training video data showing these will represent a two-dimensional face.

[0036] The labels are classifications of the input portion of the training data. A label with a first value (e.g., 1) indicates that the input portion of the training data is a two-dimensional face. A label with a second value (e.g., 0) indicates that the input portion of the training data is not a two-dimensional face. The values ​​of the output portion of the training data may be specified by the creator of the learning model M, or they may be determined by a known annotation tool.

[0037] For example, position change data obtained from training video data showing a 3D face and position change data obtained from training video data showing a 2D face have different characteristics. These differences in characteristics are thought to be correlated with whether or not a 2D face is shown in the training video data. Such correlations can be taught to the learning model M using the training data.

[0038] As shown in Figure 2, taking the action of turning the face from the front to the left as an example, the positional change of the feature point near the left eye or left cheek of a two-dimensional face is greater than the positional change of the feature point near the left eye or left cheek of a three-dimensional face. This is because a two-dimensional face printed on paper P, or a two-dimensional face displayed on a screen, has no contours in the depth direction (the perpendicular direction of paper P or screen). Such positional changes are correlated with whether or not the two-dimensional face is shown in the training video data (i.e., the label value). Such positional changes can be trained on the learning model M using the training data. Not only are positional changes of the left eye or left cheek, which is in the direction the face is facing, correlated with positional changes of other parts such as the right eye or right cheek, as well as whether or not the two-dimensional face is shown in the training video data. The learning model M can learn such correlations using the training data.

[0039] For example, for other actions such as turning the face upwards from the front, there is a correlation between the change in the position of feature points and whether or not a two-dimensional face is shown in the training video data. Therefore, for other actions as well, by training data showing the relationship between position change data obtained from the training video data, the change in the position of feature points, and whether or not a two-dimensional face is shown in the training video data, the learning model M can learn these correlations.

[0040] Note that the training data is not limited to the example in Figure 2. The training data can be any data that shows at least one of the features of three-dimensional facial feature points and the features of two-dimensional facial feature points. For example, the input portion of the training data may be the video data itself relating to a video showing a face for training. The input portion of the training data may show the position of the feature points, rather than the positional changes of each of the multiple feature points detected from each of the multiple frames of the video data. The output portion of the training data may be a score indicating suspicion of impersonation (for example, a numerical value with an intermediate value, rather than a binary value like a label).

[0041] Furthermore, the data stored by the data storage unit 100 is not limited to the examples described above. The data storage unit 100 may store any data. For example, the data storage unit 100 may store a learning program that shows a series of processes during learning. The learning program may be a program used in known machine learning methods. For example, the learning program may be a program created based on known methods such as backpropagation or gradient descent.

[0042] [Learning Department] The learning unit 101 performs training of the learning model M based on the training data stored in the training database DB1. For example, the learning unit 101 performs training of the learning model M by adjusting the parameters of the learning model M so that when the input portion of the training data is input to the learning model M, the output portion of the training data is output from the learning model M, based on a known learning program. When the training of the learning model M is complete, the learning unit 101 records the trained learning model M in the data storage unit 100. The learning model M before training may be stored in the data storage unit 100 as separate data from the trained learning model M, or it may be overwritten with the trained learning model M. The learning unit 101 sends the trained learning model M to the server 20. The trained learning model M sent to the server 20 is made available for use by the user.

[0043] [3-2. Functions implemented by the server] For example, the server 20 includes a data storage unit 200, a video data acquisition unit 201, a feature point detection unit 202, an estimation unit 203, and an identity verification unit 204. The data storage unit 200 is implemented by a storage unit 22. Each of the video data acquisition unit 201, the feature point detection unit 202, the estimation unit 203, and the identity verification unit 204 is implemented by a control unit 21.

[0044] [Data Storage Unit] The data storage unit 200 stores the data necessary for estimation using the learning model M. For example, the data storage unit 200 stores the trained learning model M and the user database DB2. In this embodiment, since the learning model M is trained by the learning terminal 10, the data storage unit 200 stores the learning model M transmitted from the learning terminal 10. When the server 20 performs the training of the learning model M, the server 20 has the same functions as the learning unit 101.

[0045] Figure 6 shows an example of a user database DB2. The user database DB2 is a database that stores various data about users. For example, the user database DB2 stores user IDs, video data, and identity verification data. The user database DB2 may store any data about the user. The data stored in the user database DB2 is not limited to the example in Figure 6. For example, if identity verification is performed when a user registers for a service, data indicating the status of the application for the service may be stored in the user database DB2.

[0046] The user ID is an example of user identification information that can identify a user. For example, an email address or telephone number may be used as user identification information. The video data is data relating to a video showing the user indicated by the user ID performing an action such as turning their face from the front to the left. The video data may be in any data format (e.g., MP4 or AVI). The user terminal 30 of the user subject to identity verification transmits the video data generated based on the shooting results of the shooting unit 36 ​​to the server 20. The server 20 stores the video data received from the user terminal 30 in association with the user ID of that user. The identity verification data is data indicating the result of identity verification. For example, the identity verification data may indicate the estimation result by the estimation unit 203.

[0047] The data stored in the data storage unit 200 is not limited to the examples above. The data storage unit 200 may store any data. For example, the data storage unit 200 may store an identity verification program that shows other processes in the series of identity verification processes other than the processes of the estimation unit 203. The processes of the identity verification unit 204, described later, are shown in the identity verification program. If identity verification documents are checked along with the user's face, the data storage unit 200 may store document data showing the identity verification documents uploaded by the user.

[0048] [Video Data Acquisition Unit] The video data acquisition unit 201 acquires video data relating to a video showing a user's face performing a predetermined action. The predetermined action is an action using the user's face. In other words, the predetermined action is an action in which the position of at least one feature point changes. In yet another word, the predetermined action is an action in which the tendency of the change in the position of the feature point differs depending on whether a three-dimensional face is shown in the action data or a two-dimensional face is shown in the video data. For example, actions such as changing the direction of the face, changing the position of the face, actions using parts of the face (e.g., blinking or winking), actions that change facial expressions, or other actions are considered predetermined actions.

[0049] In this embodiment, since video data is stored in the user database DB2, the video data acquisition unit 201 acquires video data from the user database DB2. If video data is stored in a database other than the user database DB2, the video data acquisition unit 201 can acquire the video data from the other database. If video data is stored in a computer other than the server 20 (for example, a user terminal 30) or an external information storage medium, the video data acquisition unit 201 can acquire the video data from the other computer or external information storage medium.

[0050] In this embodiment, video data is generated without the use of special hardware such as a 3D camera or thermal camera. Therefore, the video data acquisition unit 201 acquires video data generated based on the shooting results of a general shooting unit 36 ​​(e.g., an RGB camera) that does not detect depth information and temperature information. For example, a general shooting unit 36 ​​does not include a depth sensor or a temperature sensor. The video data includes only two-dimensional information (planar information) of the subject, without including depth information and temperature information of the subject. The video data shows the color of each pixel in each frame. The video data does not include depth information or temperature information as information for each pixel.

[0051] [Feature point detection unit] The feature point detection unit 202 detects multiple feature points related to faces based on the video data. Feature point detection means obtaining the position of feature points from the frames of the video data. In this embodiment, the position of feature points is given as an example where it is represented as two-dimensional information, but there are also techniques for estimating three-dimensional information from two-dimensional images, so such techniques may be used to represent the position of feature points as three-dimensional information.

[0052] In this embodiment, the detection program for detecting feature points is stored in the data storage unit 200. As mentioned above, the method for detecting feature points may be a known method. The detection program shows the processing of a known detection method. The detection program may be a library such as the Dlib library mentioned above, or it may be a program for a machine learning method such as deep learning (for example, a method called Deep3DFaceReconstruction or RingNet).

[0053] Figure 7 shows an example of processing individual frames of video data. The upper frame x in Figure 7 i (i is an integer) represents the user's face as they begin to turn their head from the front to the left. The feature point detection unit 202 detects individual frames x of the video data based on the detection program. i Multiple feature points are detected from the frame x. For example, the feature point detection unit 202 detects the frame x based on the detection program. i The mesh of the face shown is detected, and the vertices that make up the mesh are detected as feature points. The central frame x in Figure 7 i Here, the feature points are shown as dots. The feature point detection unit 202 detects each frame x i Feature point data, which represents multiple feature points detected from the data, is recorded in the data storage unit 200.

[0054] For example, feature point data may include a feature point ID and the location of the feature point. The feature point ID is an ID that identifies the feature point. The feature point ID may also identify the facial part that the feature point represents. For example, the feature point ID may be used to identify whether a feature point represents the tip of the nose or the edge of the right eye. The method for identifying the feature point may be any method used in publicly known feature point detection. For example, the feature point ID may not be used at all, and the feature point data may only show the location of the feature point.

[0055] The position of the feature point is frame x i These are the coordinates of the coordinate axes set in the frame x. For example, frame x iThe position of the feature point is indicated by the coordinates of the coordinate axes with the upper left as the origin. By tracking the position associated with a certain feature point ID, the change in the position of the feature point indicated by the feature point ID can be tracked. The lower frame x in FIG. 7 i shows the change in the position of the feature point. The calculation method of the position change will be described later.

[0056] FIG. 8 is a diagram showing an example of processing for the entire video data. In the example of FIG. 8, the feature point detection unit 202 executes the processing described with reference to FIG. 7 for each of frames x1 to x m (in FIG. 8, m is an integer of 21 or more) to detect a plurality of feature points. The feature point detection unit 202 acquires the feature point data of each of frames x1 to x m . This feature point data can be said to be a time-series history of the feature points. In FIG. 8, the position changes corresponding to each of frames x1 to x m are indicated by the signs of h1 to h m . The signs of y1 to y n are information calculated for complementing the position change. Details of the complementing method will be described later.

[0057] Note that the feature point detection unit 202 may detect feature points from all frames x of the video data i , or may detect feature points from only some frames x i . Also, the number of frames x i constituting the video data may be an arbitrary number. Further, the number of frames x i to be the detection target of the feature points may also be an arbitrary number. For example, before frame x1, there may be a temporally previous frame x0, etc. Similarly, after frame x m , there may be a frame x m+1 of a temporally later frame, etc. Note that in the following description, when it is not necessary to refer to FIGS. 7 and 8, the signs of the frames are omitted.

[0058] [Estimation unit] The estimation unit 203 estimates whether a two-dimensional face is shown in the video data based on the positional changes of each of the multiple feature points in the video data and the training model M, which has been trained on training data regarding the positional changes of each of the multiple feature points related to the training face. The estimation unit 203 calculates the positional change of each of the multiple feature points detected by the feature point detection unit 202 and obtains positional change data. The estimation unit 203 inputs the obtained positional change data into the training model M. The training model M calculates an embedding representation of the positional change data based on parameters adjusted through learning. The training model M outputs a label corresponding to the calculated embedding representation. The estimation unit 203 estimates whether a two-dimensional face is shown in the video data by obtaining the label output from the training model M.

[0059] For example, the estimation unit 203 calculates the position change of the target frame based on at least two of the following: a plurality of feature points detected from the target frame, a plurality of feature points detected from the previous frame (a frame preceding the target frame), and a plurality of feature points detected from the subsequent frame (a frame following the target frame), for each target frame whose position change is to be calculated. The estimation unit 203 may use at least one previous frame and at least one subsequent frame in calculating the position change of the target frame. The estimation unit 203 may use one previous frame and one subsequent frame in calculating the position change of the target frame, or it may use multiple previous frames and multiple subsequent frames in calculating the position change of the target frame.

[0060] In this embodiment, an example is given where the estimation unit 203 calculates the position change by suppressing noise related to the video data by performing a predetermined filtering process. For example, the estimation unit 203 performs filtering on the history of feature points detected from the video data. The estimation unit 203 may also perform filtering on the video data itself, rather than on the history of feature points detected from the video data. The filtering process may be a known filtering process available for noise suppression. The filtering process may also be a known filtering process for smoothing features in an image. Note that the estimation unit 203 may calculate the position change without performing filtering. In this case, the estimation unit 203 does not have the function of filtering. An estimation unit 203 that does not perform filtering is also included in the scope of this disclosure.

[0061] For example, the estimation unit 203 performs filtering using a DoG (Derivative of Gaussian) filter. The Gaussian function used in the DoG filter may be a known function. The parameters specified by the coefficients of the DoG filter may be arbitrary values. The Gaussian distribution is adjusted by the coefficients. In addition to noise suppression, the DoG filter may also be used for edge enhancement. The estimation unit 203 reduces the noise generated in at least one of the multiple feature points by the DoG filter and smooths the positional changes of each of the multiple feature points to calculate the positional changes of each of the multiple feature points.

[0062] The estimation unit 203 may also perform filtering using filters other than the DoG filter. For example, the estimation unit 203 may perform filtering using a Gaussian filter, Sobel filter, median filter, or linear filter that is not classified as a DoG filter. The estimation unit 203 may perform these filtering processes to calculate the position changes of feature points. If a filter is used that does not allow filtering of the history of feature points, the estimation unit 203 may perform filtering using a Gaussian filter or Sobel filter, etc., on each frame of the video data, and then have the feature point detection unit 202 detect each of the multiple feature points and calculate the position change of each of the multiple feature points.

[0063] For example, the estimation unit 203 sets up a local window W that includes a calculation target frame, which is the frame for which the position change is to be calculated; a previous frame, which is a frame prior to the calculation target frame; and a subsequent frame, which is a frame after the calculation target frame. In the example in Figure 8, the calculation target frame is frame x4. The previous frames are frames x1 to x3. The subsequent frames are frames x5 to x7. The number of previous and subsequent frames can be any number. The number of previous and subsequent frames is not limited to three as in Figure 8. For example, the number of previous and subsequent frames may be one, two, or four or more.

[0064] For example, the estimation unit 203 calculates the position change of the frame to be calculated based on the position of each of the multiple feature points detected from the previous frame and the position of each of the multiple feature points detected from the subsequent frame. In the example in Figure 8, the estimation unit 203 calculates the position change of frame x4, which is the frame to be calculated, based on the position of each of the multiple feature points detected from each of the previous frames x1 to x3 and the position of each of the multiple feature points detected from each of the subsequent frames x5 to x7.

[0065] For example, the estimation unit 203 calculates the difference between the average value of the positions of multiple feature points detected from each of the previous frames x1 to x3 and the average value of the positions of multiple feature points detected from each of the subsequent frames x5 to x7 as the position change of the frame x4 being calculated. Even if a particular frame contains noise, the noise is suppressed by calculating the average value. The average value may be a simple average or a weighted average. If the average value is a weighted average, the weight coefficient may be larger the closer it is to the frame being calculated.

[0066] In this embodiment, the position change of frame x4, which is the frame to be calculated, is represented by a vector with the same number of dimensions as the number of feature points. The estimation unit 203 may calculate the position change of frame x4, which is the frame to be calculated, by performing the aforementioned filtering process on the history of feature points in the previous and subsequent frames, rather than the average value of the positions, in order to suppress noise and smooth the position changes of the feature points. For example, the estimation unit 203 may calculate the position change of frame x4, which is the frame to be calculated, by tracking the time-series position changes of feature points in frames x1 to x7 in the local window W using DoG.

[0067] For example, the estimation unit 203 moves the local window along the time axis of the video data (the t-axis in the example in Figure 8) to successively calculate the position changes of each of the multiple frames to be calculated. In the example in Figure 8, when the estimation unit 203 calculates the position change of frame x4, it moves the local window W by one and sets frame x5 as the frame to be calculated. In this case, the previous frames are frames x2 to x4. The subsequent frames are frames x6 to x8. The estimation unit 203 calculates the position change of frame x5, the frame to be calculated, based on the positions of multiple feature points detected from each of the previous frames x2 to x4 and the positions of multiple feature points detected from each of the subsequent frames x6 to x8.

[0068] Subsequently, the estimation unit 203 moves the local window W one by one, and calculates the last frame x that will be the subject of the position change calculation. m The estimation unit 203 calculates the position changes as well. Once the calculation of position changes is complete, the estimation unit 203 obtains the position change data of the video data to be estimated. This position change data shows the position changes of each feature point calculated from each frame of the video data to be estimated in chronological order. As mentioned above, in this embodiment, the case in which the position change data is in vector format is given as an example, but the position change data may be in a format other than vector format.

[0069] In the example shown in Figure 8, if there are frames x0, etc. prior to frame x1, these prior frames x0, etc. may be used to calculate the position changes of frames x1 to x3. If no prior frames x0, etc. exist, the estimation unit 203 may calculate the position changes of frames x1 to x3 without using the local window W. Similarly, frame x m Frame x later than m+1 If such as exist, frame x m To calculate the position changes of the following frames x m+1 The following may be used. m+1 If none of the above exist, the estimation unit 203 will use frame x without using the local window W. m The positional changes of the following can also be calculated.

[0070] For example, the estimation unit 203 may calculate the position change of frame x1 based on the position of each of the multiple feature points detected from frame x1 and the position of each of the multiple feature points detected from frame x2. For example, the estimation unit 203 may calculate the position change of frame x2 based on the position of each of the multiple feature points detected from frame x2 and the position of each of the multiple feature points detected from frame x3. For example, the estimation unit 203 may calculate the position change of frame x3 based on the position of each of the multiple feature points detected from frame x3 and the position of each of the multiple feature points detected from frame x4. mSimilarly, the positional changes of the last frames of the video data may be calculated in the same manner.

[0071] In this embodiment, the estimation unit 203 calculates the position change between frames based on the position change of at least one frame of the video data and the position change of the frame following that frame. The estimation unit 203 further estimates whether a two-dimensional face is shown in the video data based on the position change between frames. For example, the estimation unit 203 calculates the position change between frames based on the position change of at least one frame of the video data, a first weight coefficient associated with that frame, the position change of the frame following that frame, and a second weight coefficient associated with that frame.

[0072] In this embodiment, the estimation unit 203 determines a frame x i The position change yi between frames is calculated based on the following formula 1. In formula 1 below, h i ,h i+1 Each of these is frame x i ,x i+1 This is the change in the position of each of the multiple feature points detected from. (Equation 1, w) i This is the first weighting coefficient. First weighting coefficient w i This is as shown in equation 2. w in equation 1 i+1 This is the second weighting coefficient. Second weighting coefficient w i+1 This is as shown in Equation 3. In Equations 2 and 3, n is the number of frames between which position changes are calculated. If position changes are interpolated between all frames, n may be m-1. n can be any value.

[0073]

number

[0074]

number

[0075]

number

[0076] Note that the first weight coefficient w i and the second weight coefficient w i+1 This is not limited to the examples in equations 2 and 3. For example, the first weight coefficient w can be calculated using formulas that do not utilize floor and ceiling functions like those in equations 2 and 3. i and the second weight coefficient w i+1 The first weight coefficient w may be calculated. i and the second weight coefficient w i+1 This may be a fixed value (for example, 0.3, 0.7). First weight coefficient w i and the second weight coefficient w i+1 The sum of these is assumed to be 1. For example, the first weight coefficient w i and the second weight coefficient w i+1 If this is not used, the estimation unit 203 may calculate a simple average of the position change of at least one frame of the video data and the position change of the frame following that frame as the position change between these frames.

[0077] For example, the estimation unit 203 obtains the final position change data by inserting the position change between frames into the position change based on each of the multiple feature points detected from individual frames. By inserting the position change between frames, the number of dimensions of the vector represented by the position change data also increases by that amount. The position change data that is the input part of the training data may also have the position change between frames inserted. The number of dimensions of the position change data that is the input part of the training data and the number of dimensions of the position change data obtained from the video data to be estimated are assumed to be the same. That is, the number of dimensions of the position change data input to the learning model M during training and the number of dimensions of the position change data input to the learning model M during estimation are assumed to be the same.

[0078] Furthermore, the number of dimensions of the position change data, which is the input portion of the training data, and the number of dimensions of the position change data obtained from the video data to be estimated may differ from each other. The number of dimensions of the position change data input to the learning model M during training and the number of dimensions of the position change data input to the learning model M during estimation may differ from each other. In this case, the portion with insufficient dimensions may be treated as missing values. Alternatively, for example, the learning model M may calculate the embedding representation in a way that absorbs the difference in dimensionality. For example, if the embedding representation is also represented as a vector, the number of dimensions of the embedding representation may be predetermined. The learning model M may calculate an embedding representation with a predetermined number of dimensions regardless of the number of dimensions of the position change data, which is the input portion of the training data, and the number of dimensions of the position change data obtained from the video data to be estimated. The number of dimensions of the embedding representation may not be predetermined and may be undefined.

[0079] Furthermore, the estimation unit 203 may determine whether the positional change of each of the multiple frames of the video data is within a reference range. The reference range is the range that serves as the criterion for whether or not to interpolate the positional changes between frames. For example, if the user moves their face faster than expected, the positional change may be too large, and the estimation unit 203 may not be able to make an accurate estimation. In such cases, a positional change exceeding a predetermined threshold may correspond to being outside the reference range. The threshold may be arbitrarily determined by the administrator of the video processing system 1.

[0080] For example, the estimation unit 203 determines whether a position change in each of multiple frames of the video data is above a threshold, thereby determining whether the position change is within a reference range. The position change subject to the reference range determination may be the position change of all feature points, or it may be the position change of some feature points. Furthermore, the position change subject to the reference range determination may be the position change of all frames, or it may be the position change of some frames.

[0081] For example, the estimation unit 203 calculates the position change between frames if it determines that the position change is outside the reference range. Since the position change of only some frames in the video data may be outside the reference range, in this case, the estimation unit 203 may calculate the position change between frames in the frames where the position change is outside the reference range, or it may calculate the position change between frames in all frames of the video data.

[0082] For example, if the estimation unit 203 determines that the position change is outside the reference range, it does not calculate the position change between frames. In this case, y1~y in Figure 8 n h1~h m Only the above is used for estimation. Since the position changes of some frames in the video data may not be outside the reference range, in this case the estimation unit 203 may calculate the position changes between frames in other frames in which the position changes were determined to be outside the reference range, without calculating the position changes between frames in some frames in which the position changes were not determined to be outside the reference range. Furthermore, in this case the estimation unit 203 may not calculate the position changes between frames in all frames of the video data. In addition, regardless of whether the position changes are outside the reference range or not, the estimation unit 203 may perform estimation based on the position changes calculated from the frames without calculating the position changes between frames. In this case the estimation unit 203 may not have the function to calculate the position changes between frames.

[0083] [Identity Verification Department] The identity verification unit 204 performs user identity verification based on the estimation result of whether or not a two-dimensional face is shown in the video data. For example, if the identity verification unit 204 estimates that a two-dimensional face is shown in the video data, it determines that identity verification has failed. If the identity verification unit 204 estimates that a two-dimensional face is shown in the video data, it may request the user to perform a predetermined action again. The action requested again may be different from the action requested previously.

[0084] For example, the identity verification unit 204 determines that identity verification is successful if it is estimated that a two-dimensional face is not shown in the video data. If a next step for identity verification (for example, taking a picture of an identity verification document such as a driver's license) is also available, the identity verification unit 204 may proceed to the next step of identity verification. The steps required for identity verification may be the same as those of publicly known identity verification procedures. The information processing required for these steps may also be the same as that used in publicly known identity verification procedures. The identity verification unit 204 may notify the person in charge of identity verification if it is estimated that a two-dimensional face is shown in the video data. In this case, the person in charge may visually check the video data.

[0085] [3-3. Functions implemented on the user terminal] For example, the user terminal 30 includes a data storage unit 300, a display control unit 301, and an operation reception unit 302. The data storage unit 300 is implemented by a storage unit 32. The display control unit 301 and the operation reception unit 302 are each implemented by a control unit 31.

[0086] [Data Storage Unit] The data storage unit 300 stores the data necessary for uploading video data. For example, the data storage unit 300 stores video data generated based on the shooting results of the shooting unit 36.

[0087] [Display Control Unit] The display control unit 301 displays various screens from the video processing system 1 on the display unit 35. For example, the display control unit 301 displays the shooting screen SC, which shows the shooting results from the shooting unit 36, on the display unit 35.

[0088] [Operation Reception Section] The operation reception unit 302 receives various operations in the video processing system 1. Data indicating the operation received by the operation reception unit 302 is transmitted to the server 20 as appropriate.

[0089] [4. Processes executed by the video processing system] Figures 9 and 10 show examples of processes performed by the video processing system 1. The processes in Figures 9 and 10 are executed by the control units 11, 21, and 31 executing programs stored in the memory units 12, 22, and 32, respectively. In the examples in Figures 9 and 10, identity verification is performed when a user applies for a service.

[0090] As shown in Figure 9, the learning terminal 10 performs training of the learning model M based on the training data stored in the training database DB1 (S1). The learning terminal 10 sends the trained learning model M to the server 20 (S2). The server 20 receives the trained learning model M from the learning terminal 10 (S3). In S3, the server 20 records the trained learning model M in the storage unit 22.

[0091] The user terminal 30 performs the process for the user to apply for the service with the server 20 (S4). When the process reaches the step requiring identity verification, the server 20 sends request data to the user terminal 30 to request the user to perform a predetermined action (S5). The request data shows the contents of message MS. The user terminal 30 receives the request data (S6). The user terminal 30 activates the shooting unit 36 ​​and displays the shooting screen SC on the display unit 35 (S7). The user performs the action instructed in message MS.

[0092] The user terminal 30 transmits video data to the server 20 (S8). The server 20 receives the video data from the user terminal 30 (S9). Based on the video data, the server 20 detects multiple feature points related to the user's face (S10). Moving to Figure 10, the server 20 performs filtering using a DoG filter (S11). The server 20 calculates the position change for each frame (S12). The server 20 may perform processes S11 and S12 while moving the local window W. The server 20 calculates the position change between frames based on equations 1 to 3 (S13).

[0093] Server 20 estimates whether a two-dimensional face is present in the video data based on the position changes from frame to frame, the position changes between frames, and the trained model M (S14). If it is estimated in S14 that a two-dimensional face is present in the video data (S14:Y), Server 20 performs a process to display an error message with User Terminal 30 (S15), and this process ends. If it is not estimated in S14 that a two-dimensional face is present in the video data (S14:N), Server 20 performs a process to proceed to the next step of identity verification with User Terminal 30 (S16), and this process ends.

[0094] [5. Summary of Embodiments] The video processing system 1 of this embodiment estimates whether a two-dimensional face is shown in the video data based on the positional changes of each of several feature points in the video data and a learning model M that has learned training data regarding the positional changes of each of several feature points related to a training face. As a result, the video processing system 1 can estimate whether a two-dimensional face is shown in the video data without preparing any special hardware. For example, it is conceivable to analyze the motion shown in the video data by optical flow without preparing any special hardware, but optical flow can be weak against background movement, so the accuracy is not sufficient. In this respect, the estimation accuracy can be improved by focusing on the positional changes of feature points of the face, which is in the foreground. Furthermore, if estimation is to be performed by analyzing mouth movements, etc., a high frame rate shooting unit 36 ​​is required, and slow movements must be requested from the user. However, the video processing system 1 can estimate whether a two-dimensional face is shown in the video data even if the shooting unit 36 ​​can only acquire two-dimensional information. The video processing system 1 can determine whether a person is present at a location (liveness) by estimating whether a two-dimensional face is shown in the video data.

[0095] Furthermore, the video processing system 1 performs a predetermined filtering process to suppress noise related to the video data and calculate positional changes. By suppressing noise, the video processing system 1 can improve estimation accuracy. For example, the accuracy of feature point detection is not perfect, and the position of feature points may shift even if the user does not move their face. Even if such positional shifts occur, they can be suppressed as noise through filtering, thus improving estimation accuracy.

[0096] Furthermore, the video processing system 1 performs filtering using a DoG filter. The DoG filter allows the video processing system 1 to effectively suppress noise and improve estimation accuracy. The video processing system 1 can also smooth out positional changes indicated by feature points.

[0097] Furthermore, the video processing system 1 sets a local window W and calculates the position change of the target frame based on the positions of multiple feature points detected from the previous frame and the positions of multiple feature points detected from the subsequent frame. The video processing system 1 moves the local window along the time axis of the video data and successively calculates the position change of each of the multiple target frames. The video processing system 1 improves the accuracy of the position change calculation by calculating the position change of the target frame based on multiple feature points detected from both the preceding and succeeding frames, rather than just one of the preceding or succeeding frames.

[0098] Furthermore, the video processing system 1 calculates the positional change between frames based on the positional change of at least one frame of the video data and the positional change of the frame following that frame. Based on the positional change between frames, the video processing system 1 estimates whether or not a two-dimensional face is shown in the video data. The video processing system 1 can compensate for positional changes between frames even if the speed at which a predetermined action is performed varies among users, thus improving the estimation accuracy. For example, even if a user takes only one second to turn from the front to the left, instead of three seconds, the video processing system 1 can improve the estimation accuracy by compensating with the positional change between frames.

[0099] Furthermore, the video processing system 1 processes the position change of at least one frame of the video data and a first weight coefficient w associated with that frame. i The position change of the frame in question in the next frame, and the second weight coefficient w associated with that next frame. i+1 Based on this, the position change between frames is calculated. This allows the video processing system 1 to adjust which of the two frames to give more weight to when calculating the position change between frames.

[0100] Furthermore, the video processing system 1 determines whether the positional changes of each of the multiple frames of the video data are within a reference range. If the video processing system 1 determines that the positional changes are outside the reference range, it calculates the positional changes between frames. The video processing system 1 can calculate the positional changes between frames if interpolation of positional changes between frames is necessary. The video processing system 1 can choose not to calculate the positional changes between frames if interpolation of positional changes between frames is not necessary.

[0101] Furthermore, the video processing system 1 performs user identity verification based on the estimation result of whether or not a two-dimensional face is shown in the video data. This allows the video processing system 1 to prevent impersonation during identity verification by malicious users.

[0102] [6. Variant] This disclosure is not limited to the embodiments described above. This disclosure may be modified as appropriate without departing from the spirit of this disclosure.

[0103] Figure 11 shows an example of the functions realized in the modified version. As shown in Figure 11, the modified server 20 includes an operation request unit 205 and a registration unit 206. Each of the operation request unit 205 and the registration unit 206 is realized by the control unit 21.

[0104] [6-1. Variation 1] For example, in this embodiment, a learning model M is given that, upon input of position change data, outputs a label indicating whether or not a two-dimensional face is shown in the video data. The learning model M can be any model in which the position changes of each of multiple feature points related to the training face have been learned. The learning model M is not limited to this embodiment.

[0105] Figure 12 shows an example of the learning model M of Modification 1. The learning unit 101 of Modification 1 learns a first learning model M1, which has been learned first training data relating to the positional changes of each of multiple feature points of a three-dimensional training face, and a second learning model M2, which has been learned second training data relating to the positional changes of each of multiple feature points of a two-dimensional training face.

[0106] For example, the first training data is position change data calculated from training video data generated by capturing the face of a training human with a camera. The method for acquiring the position change data may be the same as in the embodiment. The second training data is position change data calculated from training video data generated by capturing a training medium with a camera. The training medium is paper with the face of a training human printed on it, or a computer displaying the face of a training human. The first and second training data are stored in the training database DB1. Labels do not need to be assigned to the first and second training data.

[0107] For example, the learning unit 101 trains the first learning model M1 on the features shown in the first training data. In the embodiment where the input part of the first training data is position change data and the output part of the first training data is a label, the learning unit 101 adjusts the parameters of the first learning model M1 so that when the position change data, which is the input part of the first training data, is input to the first learning model M1, the first learning model M1 outputs a label indicating that it is not a two-dimensional face. For example, if the first training data is not labeled, the learning unit 101 performs clustering of multiple first training data and adjusts the parameters of the first learning model M1 so that first training data with similar features belong to the same cluster.

[0108] For example, the learning unit 101 trains the second learning model M2 on the features shown in the second training data. In the embodiment, if the input part of the second training data is position change data and the output part of the second training data is a label, the learning unit 101 adjusts the parameters of the second learning model M2 so that when the position change data, which is the input part of the second training data, is input to the second learning model M2, the second learning model M2 outputs a label indicating that it is not a two-dimensional face. For example, if the second training data is not labeled, the learning unit 101 performs clustering of multiple second training data and adjusts the parameters of the second learning model M2 so that second training data with similar features belong to the same cluster.

[0109] The estimation unit 203 of Modification 1 acquires position change data relating to the characteristics of position changes in multidimensional space. The method for acquiring the position change data may be the same as in the embodiment. Based on the first learning model M1, the estimation unit 203 performs encoding to reduce the dimension of the position change data and decoding to restore the dimension of the position change data, and calculates a first displacement amount, which is the amount of movement of the position change data before and after the encoding and decoding.

[0110] Encoding is also called dimensionality reduction. The encoding method may be a known method. For example, the estimation unit 203 may reduce the dimensionality of the position change data based on a method such as principal component analysis or autocoding. Decoding may also be a known method. For example, the estimation unit 203 may restore the dimensionality-reduced position change data to its original dimension based on a method such as principal component analysis or autocoding. Encoding and decoding are performed based on the parameters of the first learning model M1.

[0111] For example, the estimation unit 203 performs encoding and decoding based on the second learning model M2 and calculates a second movement amount, which is the amount of movement of the position change data before and after the encoding and decoding. The meaning of encoding and decoding is the same as that of the first learning model M1. Encoding and decoding are performed based on the parameters of the second learning model M2.

[0112] For example, the estimation unit 203 estimates whether a two-dimensional face is present in the video data based on the first and second movement amounts. If a three-dimensional face is present in the video data to be estimated, the first movement amount before and after encoding and decoding will be smaller than the second movement amount. This is because the features of the video data are similar to the features of the first training data learned by the first learning model M1. If the first movement amount is less than the second movement amount, the estimation unit 203 does not estimate that a two-dimensional face is present in the video data.

[0113] On the other hand, if the video data to be estimated contains a two-dimensional face, the first movement amount before and after encoding and decoding will be greater than or equal to the second movement amount. This is because the features of the video data are similar to the features of the second training data learned by the second learning model M2. The estimation unit 203 estimates that a two-dimensional face is contained in the video data if the first movement amount is greater than or equal to the second movement amount.

[0114] The video processing system 1 of the modified example 1 calculates a first displacement based on a first learning model M1. The video processing system 1 calculates a second displacement based on a second learning model M2. Based on the first and second displacements, the video processing system 1 estimates whether or not a two-dimensional face is shown in the video data. This allows the video processing system 1 to improve the accuracy of its estimation of whether or not a two-dimensional face is shown in the video data.

[0115] [6-2. Variation 2] For example, in the embodiment, the user is asked to turn their face from the front to the left, but the actions required of the user are not limited to the examples in the embodiment. The video processing system 1 of Modification 2 includes an action request unit 205. The action request unit 205 requests the user to perform an action selected from a plurality of facial actions by a predetermined selection method.

[0116] The candidate actions requested by the user may be any actions related to the face. For example, candidate actions may be actions that change the direction of the face, actions that move parts of the face, actions that deform parts of the face, or other actions. The candidate actions may also be actions other than those exemplified in the embodiment. The data indicating the candidate actions is pre-stored in the data storage unit 200.

[0117] The predetermined selection method may be any method. Modification 2 describes the case where the selection method is random selection. The method of randomly selecting from candidates may be a known method. For example, the operation request unit 205 randomly selects one operation from among multiple operations based on a random number. The operation request unit 205 requests the user to perform the selected operation. In the example in Figures 2 and 3, the user is requested to perform an operation by displaying the message MS, but the user may be requested to perform an operation by other methods such as outputting audio.

[0118] The selection method is not limited to random selection. For example, the selection method may be to select each of a group of actions in a predetermined order. The selection method may also be to select an action according to data obtained from the user terminal 30 or the user's characteristics. The video data acquisition unit 201 acquires video data when the action selected by the action selection unit is performed by the user. Although the video is selected by the action selection unit, which differs from the embodiment, the method of acquiring video data is the same as in the embodiment.

[0119] In the modified example 2, the data storage unit 200 is assumed to have a learning model M prepared for each operation. The training database DB1 stored in the learning terminal 10 stores training data representing each of multiple operations. The learning unit 101 performs training of the learning model M corresponding to a certain operation based on the training data representing that operation. Although it differs from the embodiment in that training data and a learning model M are prepared for each operation, the method for creating the learning model M itself may be the same as in the embodiment.

[0120] For example, let's say there are four candidate actions: a first action of turning left from the front, a second action of turning right from the front, a third action of turning upward from the front, and a fourth action of blinking. In this case, the learning unit 101 performs training on four learning models M: a learning model M for the first action, a learning model M for the second action, a learning model M for the third action, and a learning model M for the fourth action. The learning terminal 10 transmits the four trained learning models M to the server 20. The server 20 records the four trained learning models M in the data storage unit 200.

[0121] The estimation unit 203 of Modified Example 2 estimates whether a two-dimensional face is shown in the video data based on the learning model M corresponding to the action selected by the action selection unit, from among the learning models M corresponding to each of the multiple actions. This differs from the embodiment in that the learning model M corresponding to the action selected by the action selection unit is used for estimation from among the multiple learning models M stored in the data storage unit 200, but the estimation based on the learning model M itself may be the same as in the embodiment.

[0122] For example, if the four actions mentioned above, from the first action to the fourth action, are candidates, the estimation unit 203 will perform estimation based on the learning model M corresponding to the first action among the four learning models M when the user performs the first action. The estimation unit 203 will perform estimation based on the learning model M corresponding to the second action among the four learning models M when the user performs the second action. The estimation unit 203 will perform estimation based on the learning model M corresponding to the third action among the four learning models M when the user performs the third action. The estimation unit 203 will perform estimation based on the learning model M corresponding to the fourth action among the four learning models M when the user performs the fourth action.

[0123] In the modified version 2, the video processing system 1 requests the user to perform an action selected from a predetermined selection method from among multiple actions. The video processing system 1 acquires video data when the selected action is performed by the user. The video processing system 1 estimates whether a two-dimensional face is shown in the video data based on the learning model M corresponding to the selected action from among the learning models M corresponding to each of the multiple actions. This allows the video processing system 1 to perform estimation with a learning model M appropriate to the action performed by the user. A malicious user would need to anticipate various actions in order to bypass identity verification, making it difficult for them to successfully impersonate someone else.

[0124] [6-3. Modified Example 3] For example, in this embodiment, the estimation result by the estimation unit 203 is used for identity verification. The estimation result by the estimation unit 203 can be used for any purpose other than identity verification. In Modification 3, the estimation result by the estimation unit 203 is used when a user registers image data for facial recognition. A malicious user may register another person's face as image data for facial recognition and then continue to use that face photo to impersonate that person. The video processing system 1 can also be used to prevent such impersonation.

[0125] In Modification 3, as in Modification 2, videos are requested randomly. The video processing system 1 in Modification 3 includes a registration unit 206. Based on the estimation result of whether a two-dimensional face is shown in the video data, the registration unit 206 registers a frame from among multiple frames of the video data that shows the user's face facing forward as image data for user face recognition. The registration unit 206 may determine whether the user's face is facing forward by image analysis (for example, template matching, a method that determines based on the shape of the contour, or a machine learning method), or it may determine a specific frame (for example, the first frame) as a frame that shows the user's face facing forward.

[0126] For example, if the registration unit 206 estimates that a two-dimensional face is shown in the video data, it will not register the user's facial recognition image data. In this case, since a malicious user is attempting to register image data for impersonation using someone else's face photograph, the registration unit 206 will not register the image data. The registration unit 206 will register the user's facial recognition image data only if it does not estimate that a two-dimensional face is shown in the video data. The facial recognition image data may be registered in the user database DB2 or in another database.

[0127] For example, the server 20 or another computer performs facial recognition based on image data registered by the registration unit 206. The authentication terminal for facial recognition includes a camera. The authentication terminal may be the user terminal 30 or another computer other than the user terminal 30. When the authentication terminal captures the user's face with its camera, it transmits the image data to the server 20 or the other computer. The server 20 or the other computer performs facial recognition based on the image data received from the authentication terminal and the image data registered by the registration unit 206. Facial recognition may be performed by a known method.

[0128] In the modified version 3, the video processing system 1, based on the estimation result of whether or not a two-dimensional face is shown in the video data, registers the frame showing the user's face facing forward from among multiple frames of the video data as image data for user face recognition. The video processing system 1 can prevent malicious users from impersonating others, registering image data, and performing face recognition.

[0129] [6-4. Other variations] For example, variations 1 to 3 described above may be combined.

[0130] For example, functions described as being implemented on server 20 may be implemented on other computers such as learning terminal 10. Functions described as being implemented on server 20 may be shared among multiple computers. Functions described as being implemented on learning terminal 10 may be implemented on other computers such as server 20.

[0131] [7. Addendum] For example, a video processing system can also be configured as follows: (1) A video data acquisition unit acquires video data related to a video showing the face of a user performing a predetermined action, A feature point detection unit detects multiple feature points relating to the face based on the aforementioned video data, An estimation unit estimates whether a two-dimensional face is shown in the video data based on the positional changes of each of the multiple feature points in the video data and a learning model that has learned training data relating to the positional changes of each of the multiple feature points relating to a training face. A video processing system that includes this. (2) The estimation unit calculates the position change by performing a predetermined filtering process to suppress noise related to the video data. (1) The video processing system described above. (3) The estimation unit performs the filtering process using a DoG (Derivative of Gaussian) filter. (2) The video processing system described above. (4) The estimation unit, A local window is set up that includes a calculation target frame, which is the frame for which the position change is to be calculated; a previous frame, which is a frame prior to the calculation target frame; and a subsequent frame, which is a frame after the calculation target frame. Based on the position of each of the multiple feature points detected from the previous frame and the position of each of the multiple feature points detected from the subsequent frame, the position change of the frame to be calculated is calculated. The local window is moved along the time axis of the video data, and the positional changes of each of the multiple frames to be calculated are calculated one after another. A video processing system as described in any of (1) to (3). (5) The estimation unit, Based on the position change of at least one frame of the video data and the position change of the frame following that frame, the position change between these frames is calculated. Based on the positional changes between frames, it is estimated whether the two-dimensional face is shown in the video data. A video processing system as described in any of (1) to (4). (6) The estimation unit calculates the position change between frames based on the position change of at least one frame of the video data, a first weighting coefficient associated with that frame, the position change of the frame following that frame, and a second weighting coefficient associated with that frame. (5) The video processing system described above. (7) The estimation unit determines whether the position change of each of the multiple frames of the video data is within the reference range, and if it is determined that the position change is outside the reference range, it calculates the position change between the frames. (5) or (6) The video processing system described above. (8) The estimation unit, We obtain position change data relating to the characteristics of the position change in multidimensional space, Based on a first learning model that has learned first training data relating to the positional changes of each of multiple feature points of a three-dimensional training face, encoding is performed to reduce the dimensionality of the positional change data, and decoding is performed to restore the dimensionality of the positional change data, and a first displacement amount, which is the amount of movement of the positional change data before and after the encoding and decoding, is calculated. Based on a second learning model that has learned second training data relating to the positional changes of each of multiple feature points of a two-dimensional training face, the encoding and decoding are performed, and a second movement amount, which is the amount of movement of the positional change data before and after the encoding and decoding, is calculated. Based on the first and second displacement amounts, it is estimated whether or not the two-dimensional face is shown in the video data. A video processing system as described in any of (1) to (7). (9) The video processing system further includes an action request unit that requests the user to perform an action selected from among a plurality of actions relating to the face by a predetermined selection method, The video data acquisition unit acquires the video data when the selected operation is performed by the user. The estimation unit estimates whether the two-dimensional face is shown in the video data based on the learning model corresponding to the selected action among the learning models corresponding to each of the plurality of actions. A video processing system as described in any of (1) to (8). (10) The video processing system further includes an identity verification unit that performs identity verification of the user based on the estimation result of whether or not the two-dimensional face is shown in the video data. A video processing system as described in any of (1) to (9). (11) The video processing system further includes a registration unit that, based on the estimation result of whether or not the two-dimensional face is shown in the video data, registers the frame from among a plurality of frames of the video data in which the user's face is facing forward as image data for the user's facial recognition. A video processing system as described in any of (1) to (10).

Claims

1. A video data acquisition unit acquires video data related to a video showing the face of a user performing a predetermined action, A feature point detection unit detects multiple feature points relating to the face based on the aforementioned video data, An estimation unit acquires position change data relating to the position change characteristics of each of the plurality of feature points in a multidimensional space, and performs encoding to reduce the dimension of the position change data and decoding to restore the dimension of the position change data based on a first learning model in which first training data relating to the position change of each of the plurality of feature points relating to a three-dimensional training face has been learned, calculates a first displacement amount which is the amount of movement of the position change data before and after the encoding and decoding, performs the encoding and decoding based on a second learning model in which second training data relating to the position change of each of the plurality of feature points relating to a two-dimensional training face has been learned, calculates a second displacement amount which is the amount of movement of the position change data before and after the encoding and decoding, and estimates whether or not a two-dimensional face is shown in the video data based on the first displacement amount and the second displacement amount. A video processing system that includes this.

2. The estimation unit calculates the position change by performing a predetermined filtering process to suppress noise related to the video data. The video processing system according to claim 1.

3. The estimation unit performs the filtering process using a DoG (Derivative of Gaussian) filter. The video processing system according to claim 2.

4. The estimation unit, A local window is set up that includes a calculation target frame, which is the frame for which the position change is to be calculated; a previous frame, which is a frame prior to the calculation target frame; and a subsequent frame, which is a frame after the calculation target frame. Based on the position of each of the multiple feature points detected from the previous frame and the position of each of the multiple feature points detected from the subsequent frame, the position change of the frame to be calculated is calculated. The local window is moved along the time axis of the video data, and the positional changes of each of the multiple frames to be calculated are calculated one after another. A video processing system according to any one of claims 1 to 3.

5. The estimation unit, Based on the position change of at least one frame of the video data and the position change of the frame following that frame, the position change between these frames is calculated. Based on the positional changes between frames, it is estimated whether the two-dimensional face is shown in the video data. A video processing system according to any one of claims 1 to 3.

6. The estimation unit calculates the position change between frames based on the position change of at least one frame of the video data, a first weight coefficient associated with that frame, the position change of the frame following that frame, and a second weight coefficient associated with that frame. The video processing system according to claim 5.

7. The estimation unit determines whether the position change of each of the multiple frames of the video data is within the reference range, and if it is determined that the position change is outside the reference range, it calculates the position change between the frames. The video processing system according to claim 5.

8. The video processing system further includes an action request unit that requests the user to perform an action selected from among a plurality of actions relating to the face by a predetermined selection method, The video data acquisition unit acquires the video data when the selected operation is performed by the user. The estimation unit estimates whether the two-dimensional face is shown in the video data based on the first and second learning models corresponding to the selected action, among the first and second learning models corresponding to each of the plurality of actions. A video processing system according to any one of claims 1 to 3.

9. The video processing system further includes an identity verification unit that performs identity verification of the user based on the estimation result of whether or not the two-dimensional face is shown in the video data. A video processing system according to any one of claims 1 to 3.

10. The video processing system further includes a registration unit that, based on the estimation result of whether or not the two-dimensional face is shown in the video data, registers the frame from among a plurality of frames of the video data in which the user's face is facing forward as image data for the user's facial recognition. A video processing system according to any one of claims 1 to 3.

11. A computer, A video data acquisition step involves acquiring video data related to a video showing the face of a user performing a predetermined action, A feature point detection step that detects multiple feature points relating to the face based on the aforementioned video data, An estimation step is performed to obtain position change data relating to the position change characteristics of each of the plurality of feature points in a multidimensional space, and to perform encoding to reduce the dimension of the position change data and decoding to restore the dimension of the position change data based on a first learning model in which first training data relating to the position change of each of the plurality of feature points relating to a three-dimensional training face have been learned, and to calculate a first displacement amount which is the amount of movement of the position change data before and after said encoding and said decoding, and to perform said encoding and said decoding based on a second learning model in which second training data relating to the position change of each of the plurality of feature points relating to a two-dimensional training face have been learned, and to calculate a second displacement amount which is the amount of movement of the position change data before and after said encoding and said decoding, and to estimate whether or not a two-dimensional face is shown in the video data based on said first displacement amount and said second displacement amount. A video processing method that performs this task.

12. A video data acquisition unit acquires video data related to a video showing the face of a user performing a predetermined action. A feature point detection unit detects multiple feature points related to the face based on the aforementioned video data. An estimation unit acquires position change data relating to the position change characteristics of each of the multiple feature points in a multidimensional space, performs encoding to reduce the dimension of the position change data and decoding to restore the dimension of the position change data based on a first learning model in which first training data relating to the position change of each of the multiple feature points relating to a three-dimensional training face has been learned, calculates a first displacement amount which is the amount of movement of the position change data before and after the encoding and decoding, performs the encoding and decoding based on a second learning model in which second training data relating to the position change of each of the multiple feature points relating to a two-dimensional training face has been learned, calculates a second displacement amount which is the amount of movement of the position change data before and after the encoding and decoding, and estimates whether or not a two-dimensional face is shown in the video data based on the first displacement amount and the second displacement amount. A program that makes a computer function.

Images