Normalization using character characteristics from an autoencoder

Abstract

This article generally discusses devices, systems, and methods. One method may include obtaining a normalized autoencoder trained on a first data sample of a template person and second data samples of various persons, normalizing the input data samples by combining the dynamic characteristics of persons in the input data samples with the static characteristics in the first data samples to generate normalized data, and feeding the normalized data as input to a classifier model to classify the input data based on the dynamic characteristics of the input data samples and the static characteristics of the first data samples.

Description

Background Technology

[0001] Facial expression recognition technology has been successfully used in a wide variety of contexts across various applications of human-centered artificial intelligence, such as improving human-computer interaction, monitoring depression, estimating patient pain, measuring television (TV) viewer engagement, and promoting driver safety. All of this is possible even considering that the meaning of different facial expressions can vary depending on the context. To help quantify facial expressions, researchers often rely on facial action unit coding systems (FACs), which break down facial movements into different muscle activations (e.g., AU12 for lip corner tension, often seen when smiling). Like other computer vision domains, this field has made significant progress over the past decade, at least in part due to advancements in deep neural network (DNN) and graphics processing unit (GPU) hardware that have enabled the training of complex models and the analysis of large datasets. Summary of the Invention

[0002] The summary portion of this invention provides a simplified description of aspects of the embodiments, which are further explained in the detailed description. This summary portion is not intended to identify essential or desired features of the claimed subject matter, and the combinations and order of elements listed in this summary portion are not intended to limit the elements of the claimed subject matter.

[0003] Systems, methods, devices, and computer or other machine-readable media can provide improvements over existing face normalization techniques. These improvements can include improved performance on data spanning a wide range of human characteristics. Typically, face normalization performs well on input data from individuals with the same or similar features as those represented in the training data. These improvements can be achieved using autoencoders trained and manipulated in a specific manner, which will be discussed in more detail elsewhere in this document.

[0004] This invention provides a method, apparatus, computer-readable medium, components, and system. An apparatus may include processing circuitry and a memory including instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations for data normalization. The operations may include a normalization autoencoder trained from a first data sample based on a template person and second data samples of various persons, normalizing the input data samples by combining dynamic characteristics of persons in the input data samples with static characteristics of the first data samples to generate normalized data. The static characteristics include the same characteristics found in the first data samples. The normalized data may be provided as input to a classifier model to classify the input data samples based on the dynamic characteristics of the input data samples and the static characteristics of the first data samples.

[0005] A normalized autoencoder can be trained using a single encoder and multiple decoders, where a first decoder is dedicated to reconstructing a first data sample and a second decoder is dedicated to reconstructing a second data sample. The encoder can be trained based on the reconstruction loss of both the first and second decoders. The first decoder can be trained based solely on its reconstruction loss. The second decoder can be trained based solely on its reconstruction loss. During runtime, the normalized autoencoder can operate by using the encoder to compress the representation of the input data samples and using the first decoder to construct normalized data based on the compressed representation. Training can include training the encoder and second decoder on a batch of second data samples, followed by training the encoder and first decoder on a batch of first data samples, or vice versa.

[0006] The first data sample can be an image of a template face. The second data sample can be images of various faces. The input data can be images of faces to be normalized. A classifier model can provide classifications of facial action units (FAUs) present in the input images. The operation may further include adjusting the angles and poses of the faces in the first and second data samples to be consistent before normalizing the input data, and wherein normalization is performed based on the input data adjusted for angles and poses. Static characteristics may include facial structure, and dynamic characteristics may include mouth formation and eyelid formation. Attached Figure Description

[0007] Figure 1 A diagram illustrating an embodiment of an autoencoder system for facial normalization is shown by way of example.

[0008] Figure 2 A diagram illustrating an embodiment of a system for face normalization is shown by way of example.

[0009] Figure 3 A diagram illustrating an example of the FAU classification system is shown.

[0010] Figure 4 A diagram illustrating an example of a method for facial image normalization using an autoencoder is shown by way of example.

[0011] Figure 5 A block diagram illustrating an example environment for a system used for neural network training is shown below.

[0012] Figure 6 A block diagram of an embodiment of a machine (e.g., a computer system) for implementing one or more embodiments is shown by way of example. Detailed Implementation

[0013] In the following description, reference is made to the accompanying drawings, which form a part thereof, and specific embodiments that can be practiced are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice them. It should be understood that other embodiments may be used, and structural, logical, and / or electrical changes may be made without departing from the scope of the embodiments. Therefore, the following description of the embodiments should not be construed as limiting, and the scope of the embodiments is defined by the appended claims.

[0014] In some embodiments, the operations, functions, or techniques described herein may be implemented in software. This software may include computer-executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transitory memories (e.g., non-transitory machine-readable media) or other types of hardware-based storage devices (local or network). Furthermore, such functionality may correspond to a subsystem, which may be software, hardware, firmware, or a combination thereof. Multiple functions may be performed in one or more subsystems as needed; the described embodiments are merely examples. The software may execute on a digital signal processor, application-specific integrated circuit (ASIC), microprocessor, central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or other types of processors operating on a computer system (such as a personal computer, server, or other computer systems that turn such computer systems into specially programmed machines). Functions, operations, or methods can be implemented using processing circuitry, which may include electronic and / or electrical components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates (e.g., AND, OR, XOR, NOT, etc.), buffers, caches, memories, GPUs, CPUs, FPGAs, ASICs, etc.). Examples consider the generalization of model problems (e.g., machine learning (ML) models, such as neural networks (NNs), which may include deep NNs (DNNs), convolutional NNs (CNNs), autoencoders, etc.) across different groups (a group is a collection of people with different, common characteristics). For example, a group may include images, audio, or other data from one or more people, while another group may include data from one or more other people. In another example, one group may include data from one or more people in a specific recording setting (e.g., location, such as office, home, etc.), and another group may include data from the same one or more people in different recording settings (e.g., office, home, etc.).

[0015] The implementation examples can improve model performance and also help prevent potential model bias. While the examples show performance improvements when considering different groups, some gaps remain across different data splits (e.g., better performance was achieved using tests with female subjects). Further improvements can help reduce these biases. Model analysis was performed, and improvements to current techniques are shown.

[0016] A challenge in deploying ML model systems is the ability to interpret and understand what the model does. Compared to existing work that often considers normalizing features using intuitive methods (e.g., range correction, relative variation), this implementation divides the learning process into two phases: one dedicated to reducing cross-individual variability, and the other dedicated to learning FAU recognition. This separation provides the opportunity to examine the output of each ML model after the normalization process. This examination can be used to isolate potential flaws in the generalization process. The use of shared facial appearances provides a familiar model introspection (facial) channel, facilitating intuitive detection of limitations in the ML model. Generally, separating the learning process helps isolate and debug potential flaws.

[0017] Human-centered artificial intelligence deploys ML models for various applications, such as facial expression recognition, pedestrian counting, speech recognition, vital sign monitoring, and emotion recognition, and is desirable in various application domains such as market research, psychotherapy, image captioning, entertainment, traffic control, urban planning, virtual assistants, and improving driver safety. However, large sources of variation, such as those associated with individual facial appearance, voice, and biological processes, can limit the potential generalization of trained models. To help address this issue, examples use DNN (e.g., CNN autoencoders) normalization methods, which reduce the variation in common features (e.g., facial appearance in facial expression recognition examples) while preserving one or more variable features (e.g., facial expressions in facial expression recognition examples).

[0018] The implementation example can use a self-supervised denoising autoencoder to transfer facial expressions from different individuals onto a common, learned facial template. The output of the autoencoder can be used to train and evaluate other human feature recognition models. Using a first benchmark dataset as a reference, the performance of the implementation example during training and testing was determined both intra-individual and cross-individual, gender (male and female), and skin color type (light and dark). The implementation example can provide consistent performance gains on normalized data. Furthermore, the implementation example can be used to improve cross-dataset generalization by leveraging a second benchmark dataset that differs from the first benchmark dataset in both demographics and data collection settings.

[0019] The challenge of applying human-centered ML models involves developing tools that perform well across diverse datasets despite variations in the data (e.g., different people represented in the data, different demographics of those people, different data collection settings, etc.). In the context of facial expression recognition, some key differences are associated with facial appearance, such as head shape and facial features (e.g., skin type, facial hair, nose, mouth, or ear shape, etc.). Therefore, machine learning (ML) models trained and tested using data from the same person (sometimes called "person-dependent" (PD) models) generally perform better than models trained and tested using data from different people (sometimes called "person-independent" (PI) models). Similar differences can be observed when considering intra- and cross-group comparisons of demographic aspects that influence specific facial features. From an ML perspective, these performance differences are partly explained by the assumption of independent and identically distributed (IID) features of the person used for training. This assumption requires as much consistency as possible across training and test sets to ensure appropriate generalization performance. These performance differences can be viewed as similar to outgroup homogeneity bias and cross-racial effects, suggesting that people are better able to identify differences between in-group and out-group members, and that people are better at recognizing the faces of people with similar demographic characteristics.

[0020] To help bridge the gap between in-group and out-of-group settings, embodiments explore reducing cross-personal variability by transferring facial expressions across training and testing groups to a common single facial representation (sometimes called a template facial image). Embodiments may utilize autoencoder denoising methods that allow for the transfer of appearances in a self-supervised manner without using any explicit FAU or facial markers. Embodiments may leverage facial expression transfer to minimize individual variability in the context of facial action unit recognition.

[0021] Figure 1 An embodiment of an autoencoder system 100 for data normalization is illustrated by way of example. The autoencoder system 100, as shown, includes template character data 102 and variable character data 104 as inputs to an autoencoder 101. The autoencoder 101 is trained to reconstruct the variable character data 104 and the template character data 102 into reconstructed variable data 114 and reconstructed template data 112, respectively. This is merely to help illustrate how normalization works. Figure 1 An example of data in image form is shown. Other example data forms include audio files, video data, sensor data associated with a person, or combinations thereof.

[0022] Template person data 102 is one of multiple data samples of the same person used to train autoencoder 101. Multiple data samples can include the same person with different characteristics (sometimes called dynamic characteristics) and static characteristics. Example data samples include physiological signals, audio files, images, etc. In an example where the template data sample is an image, different characteristics can include facial expressions, and static characteristics can include the facial structure of the template person.

[0023] Variable character data 104 is one of multiple data samples of one or more characters used to train the autoencoder 101. Variable character data 104 does not include data used as template character data 102. The multiple data samples may include one or more characters with different characteristics and static properties.

[0024] An autoencoder (such as autoencoder 101) learns to denoise the input and copies the denoised input to its output. The autoencoder has inner layers (hidden layers) that describe the “code” (sometimes referred to herein as “hidden feature vectors” or “hidden feature representations”) used to represent the input. Autoencoder 101 includes an encoder 106 that maps template character data 102 and variable character data 104 to corresponding hidden feature vectors. Autoencoder 101 includes decoders 108 and 110 that map the corresponding hidden feature vectors to reconstructions of: (i) template character data 102 as reconstructed template character data 112 and (ii) variable character data 104 as reconstructed variable character data 114.

[0025] Encoder 106 can be trained using a reconstruction loss term that describes (i) the difference between template character data 102 and reconstructed template data 112, and (ii) the difference between variable character data 104 and reconstructed variable data 114. Decoder 108 can be trained using a reconstruction loss term that takes into account the difference between template character data 102 and reconstructed template data 112. Decoder 110 can be trained using a reconstruction loss term that describes the difference between variable character data 104 and reconstructed variable data 114.

[0026] Facial expression transfer and expression synthesis have been recently studied, and this is an example application of the embodiment. Some of the most popular methods start by detecting facial landmarks or FAUs to help guide the transfer process. Since the embodiment can be used to improve the task of FAU recognition, it does not require any explicit indication of facial landmarks or FAUs. The autoencoder 101 can be self-supervised. As discussed, the autoencoder 101 includes an encoder 106(E) that reduces the dimensionality of the template person data 102 and the variable person data 104 to a lower-dimensional latent space. The decoder 108Dy attempts to recover the template person data 102, which includes a reference person selected for a normalization target. The decoder 110Dx attempts to recover the variable person data 104, which includes data of the individual to be normalized.

[0027] During the training phase, data 102 and data 104 are iteratively compressed using encoder 106. Furthermore, during training, decoders 108 and 110 are trained to reduce the errors between data 102, 104 and reconstructed data 112, 114, respectively. These errors can be root mean square error (RMSE), L2 reconstruction loss, mean squared error (MSE), mean absolute error (MAE), R-squared (e.g., 1 - MSE(model) / MSE(baseline)) or adjusted R-squared, mean squared percentage error (MSPE), mean squared absolute percentage error (MAPE), or mean mean squared logarithmic error (RMSLE).

[0028] The training process can be based on the following loss function to change the weights of the neurons in the autoencoder 101:

[0029]

[0030]

[0031] Where x i The data i represents the entity to be normalized, y = 104. j This indicates that the reference template has data j = 102, and x... i ′ and y j '' represents the corresponding preprocessed versions of data 104 and 102, respectively. Training can be repeated based on data 104 for each entity to be normalized.

[0032] Since the same encoder 106 is used to generate hidden feature vectors for recovering both template data 112 and variable data 114, the learned hidden feature vectors are configured to capture sources of variance shared by data 102, 104 (e.g., head pose, facial expression, etc.). Decoders 108, 110 learn to add information with smaller variables (facial appearance). As a preprocessing step in some embodiments where data 102, 104 includes images, the input images can be converted to grayscale. In some embodiments, histogram equalization techniques can be used to correct grayscale pixel values ​​to promote a more consistent distribution of pixel values ​​across individuals. Furthermore, image enhancement techniques can be used to increase the amount of variation in data 102, 104. Image enhancement techniques may include random affine transformations, Gaussian warps, etc.

[0033] During runtime, the individual data 104 to be normalized can be compressed (e.g., without increasing) by encoder 106 and recovered by template decoder 108 (Dy), which is trained to decode template person data 102 according to Equation 3:

[0034] X y =D y Equation 3 (E(X))

[0035] Where X y A normalized image representing a person.

[0036] For training, the autoencoder 101 can learn based on multiple data samples of data 102 and 104 (e.g., hundreds or thousands of samples of each of data 102 and 106). During training, a batch of template character data 102 can be input, followed by a batch of data 104 from the character to be normalized, and vice versa. The number of data samples in each batch of template character data 102 and variable character data 104 can be the same or different. The loss determined when inputting template character data 102 can be used to adjust the weights of encoder 106 and decoder 108, such as by using backpropagation. The loss determined based on the variable character data 104 as input can be used to adjust the weights of encoder 106 and decoder 110.

[0037] Tables 1 and 2 show the specific architectural implementations of encoder 106 and decoders 108 and 110, respectively. Note that these architectural implementations are merely examples, and many variations are possible, such as kernel size, number of filters, number of strides, layer type, etc.

[0038] layer Filter, kernel size, stride Output enter - 128,128,1 CONV2D / LEAKYRELU 128,5,2 64,64,128 CONV2D / LEAKYRELU 256,5,2 32,32,256 CONV2D / LEAKYRELU 512,5,2 16,16,512 CONV2D / LEAKYRELU 1024,5,2 8,8,1024 FLATTEN - 65536 DENSE - 1024 DENSE - 16384 RESHAPE - 4,4,1024 CONV2D / LEAKYRELU 2048,3,- 4,4,2048 PIXELSHUFFLER - 8,8,512

[0039] Table 1: Example encoder 106 network architecture used for compressing facial information

[0040]

[0041]

[0042] Table 2: Example decoder 108 and 110 network architectures used for facial information recovery

[0043] Furthermore, regarding specific applications of facial expression recognition, to facilitate the conversion of as many facial expressions as possible into a single facial appearance, the most expressive object in the image dataset can be used as the template face for template person data 102. This most expressive object is more likely to capture a wider range of facial variations compared to another, less expressive object. The most expressive object can include entities whose median Facial Expression Activation Unit (FAU) across all action units is the highest in the dataset. Besides FAU classification, another application of image normalization can include normalizing views of people in online meeting programs such as Zoom, Teams, FaceTime, GoToMeeting, BlueJeans, etc. Facial normalization can provide anonymity while providing facial expressions. Another application includes creating synthetic reference models, such as avatars whose facial expressions are controlled by another entity (e.g., the entity depicted in the variable facial image 104).

[0044] Figure 2 A diagram illustrating an embodiment of a system 200 for human-centered data normalization is shown as an example. Similar to... Figure 1 , Figure 2 The input and output data are visualized as images, but other types of data are also possible. The illustrated system 200 includes character data 220, which may include, for example, data used for training encoder 106 and decoder 110 (see [link to documentation]). Figure 1 The data is associated with the person. Person data 220 is input into encoder 106. Encoder 106 is trained based on template person data 102 and variable person data 104. Decoder 108 can reconstruct person data 220 based on compression performed by encoder 106 to generate normalized person data 224. Normalized person data 224 includes invariant components (sometimes called "static features") from template person data 102 and variable components (sometimes called "dynamic features") from person data 220 (e.g., mouth formation, eye position, head tilt, pitch variation, accent, etc.). Normalized person data 224 can be collected for various samples of person data 220. Normalized person data 224 can be used as a human-centered AI model (see [link to relevant documentation]). Figure 3 The input is used to determine the classification of person data 220, for example, by determining the classification based on the corresponding normalized facial image 224.

[0045] Depend on Figure 2Applications of personnel data normalization include user anonymity, such as for online conferencing applications, video calling applications, personnel counting, and driver safety. Figure 2 Another application of facial normalization is improved classification.

[0046] Figure 3 A diagram illustrating an embodiment of the classification system 300 is shown by way of example. The system 300, as shown, includes normalized person data 224 as input to a person-centered ML model 330. The person-centered ML model 330 determines a classification 332 based on the normalized person data 224.

[0047] After the variable properties of variable character data 104 are transferred to the static properties of public template character data 102 (according to...) Figures 1-2 These data can be fed into an ML classifier (human-centered ML model 330). The human-centered ML model 330 can operate on a single data sample and determine a classification 332. An exemplary human-centered ML model 330 is a LeNet-5 convolutional neural network (CNN) architecture. Other classifiers can be implemented using the human-centered ML model 330.

[0048] In the examples of facial normalization, example category 332 is FAU. FAU defines action units (AUs) on the face. An AU corresponds to the relaxation or contraction of a muscle. The muscles in an FAU are part of the face. Example FAUs include inner eyebrow lift, outer eyebrow lift, eyebrow lowering, cheek lift, eyelid tightening, upper lip lift, corner of the lip pull, dimple, corner of the lip pressure, chin lift, lip tightening, and lip pressing, etc.

[0049] The examples provide improvements to the generalization of the FAU model across different populations. To aid evaluation, multiple within-group and cross-group evaluations were performed across different subgroups. The examples were then operated on to evaluate whether they improved the performance of the ML model.

[0050] The following groupings were considered:

[0051] Person. The first group segmentation is at the individual level, which is most often considered the source of human variation. Intra-group evaluation involves models trained and tested using data from the same person (sometimes called PD models). Cross-group evaluation involves models trained and tested using data from different persons (sometimes called PI models). In this case, the person-dependent model captures the best performance scenario where person labels and data are available, and therefore it is easier to achieve better model generalization.

[0052] Gender. The second group, at the gender level (male and female), has been shown to influence facial appearance, voice, and other human differences due to physiological and hormonal differences. In terms of facial images, gender-related differences in facial features include the amount of hair or the shape of the chin. Intra-group evaluations included models trained and tested only by male participants, and other models trained and tested only by female participants. Cross-group evaluations included models trained only by male participants and tested only by female participants, and vice versa. For convenience, these models are referred to as “gender-dependent” (GD) and “gender-independent” (GI) models, respectively. However, both types of models fall under the category of human-independent models because the subjects used for training and validation differ from those used for testing.

[0053] Skin type. The third group segmentation is at the skin type level (lighter and darker), which affects facial analysis due to differences in type distribution. Intra-group evaluation includes models trained and tested only with participants of the light skin type, and models trained and tested only with participants of the dark skin type (sometimes called "skin-dependent" (SD) models). Cross-group evaluation includes models trained with participants of the light skin type and tested with participants of the dark skin type, and vice versa (sometimes called "skin-independent" (SI) models). One technique for annotating skin types is the Fitzpatrik Prototype Ratio, which categorizes skin types into six main categories.

[0054] Datasets. The fourth and final groups are at the dataset level, including differences due to many factors such as participant demographics and data collection settings. Intra-group evaluation involves models trained and tested on participants from the same dataset (sometimes called "database-dependent" (DD) models). Cross-group evaluation involves models trained on data samples from one dataset and tested on data samples from another dataset (sometimes called "database-independent" (DI) models).

[0055] The performance of the examples was evaluated under different group splits and using the first benchmark dataset (1BD). A second benchmark dataset (2BD) was also used to investigate generalization across datasets.

[0056] Table 3 shows a summary of some of the results.

[0057] When evaluating a model with normalized images ( Figure 2When the output of the normalized image was used, the PI model accuracy increased to 59.6%, which is higher than its unnormalized PI counterpart (p < 0.001) and very similar to the unnormalized PD results (p = 0.375). This finding indicates that the implementation can effectively reduce individual differences associated with appearance. The PD model with normalized images maintained 61.4% performance, which is similar to its unnormalized counterpart (p = 0.388), indicating that the face transfer process of the implementation does not lose relevant facial expression information.

[0058] To capture the average (with a threshold of 0.5) between overall performance for each model, F1 score for each action unit, and accuracy, these were then aggregated for each participant. For each condition, the mean and standard deviation were calculated across all participants. To compare performance across different conditions, a two-sample t-test with significance scores was used when p < 0.05.

[0059]

[0060]

[0061] Table 3: Summary of Analysis Results

[0062] When using the raw (unnormalized) data, the GI model achieved an average score of 52.6%, while the GD model achieved an average score of 55%, a significant difference (p = 0.009). This difference suggests that gender differences across the training and test sets account for approximately 2.4% of the data in this dataset. When normalized images were used, the GI model improved to 57.7%, higher than the unnormalized counterparts (p < 0.001), and the GS model improved to 60.2%, also higher than the unnormalized counterparts (p < 0.001). The GI model, running on normalized images, produced better results than the GD model without normalized images, indicating that normalization helps address individual differences beyond gender and that the implementation normalizes differences at the individual level. The use of image normalization produced a consistent average improvement of 5.2% across different conditions.

[0063] When using unnormalized images, the SI model achieved a mean score of 49.9%, while the SD model achieved a mean score of 55.2%, a significant difference (p = 0.025). This difference suggests that the influence of different skin types across the training and test sets is approximately 5.3%, which is slightly larger than the generalization gap associated with gender (2.4%). This finding appears to suggest that skin type may have a greater influence than gender in the context of model generalization. However, the number of objects in the skin type condition was less than the number of objects in the gender condition.

[0064] When normalized images are used, the SI model increases to 57.4%, higher than its unnormalized counterpart (p<0.001), and the SD model increases to 58.7%, similar to its unnormalized counterpart (p=0.110). In this case, the SI model with normalized images produces a higher, rather than comparable, result compared to the SD model with unnormalized images (p=0.290), indicating that the normalization method addresses the main source of data variance under these conditions.

[0065] Image normalization yielded a consistent average improvement of 5.7% across different conditions. This difference was minimal when training and testing on people with darker skin tones, where performance was comparable to that of PD models (approximately 60%).

[0066] When using unnormalized images, the average score of the DI model was 44.6%, while the average score of the DD model was 54%, which is different (p<0.001). This difference indicates that using different datasets across training and test sets has an impact of approximately 9.4%. This difference is the largest observed gap across all group conditions, indicating that cross-dataset generalization is one of the most difficult challenges to address.

[0067] When normalized images are used, the DI model increases to 51.1%, higher than its unnormalized counterpart (p<0.001), and the DD model increases to 61.5%, also higher than its unnormalized counterpart (p<0.001).

[0068] Regarding FAU classification, the examples provide an average improvement of approximately 4.6% (STD: 2.9) for the AUs under consideration, with the largest gain (11.4%) for AU04 and the smallest gain (0.8%) for AU14.

[0069] Figure 4 An embodiment of a method 400 for image normalization using an autoencoder is illustrated by way of example. The method 400, as shown, includes (optionally) altering the angles and poses of faces in a first data sample and a second data sample at operation 402 to make them consistent with each other; at operation 404, obtaining a normalized autoencoder trained based on a first data sample of a template person and second data samples of various people; at operation 406, normalizing the input data samples by the normalized autoencoder by combining the dynamic characteristics of the people in the input data samples with the static characteristics of the first data samples to generate normalized data; and at operation 408, providing the normalized data as input to a classifier model to classify the input data based on the dynamic characteristics of the input data and the static characteristics of the first data samples.

[0070] Method 400 may further include a normalized autoencoder using a single encoder and multiple decoders, wherein a first decoder is dedicated to reconstructing a first data sample and a second decoder is dedicated to reconstructing a second data sample. Method 400 may further include an encoder trained based on the reconstruction loss of both the first and second decoders. Method 400 may further include a first decoder trained only based on the reconstruction loss of the first decoder, and a second decoder trained only based on the reconstruction loss of the second decoder. Method 400 may further include, during runtime, the normalized autoencoder operating on the input data using the encoder and the second decoder. Method 400 may further include training the encoder and the first decoder on a second batch of data of a variable person, and subsequently training the decoder and the second decoder on a first batch of data of a template person, or vice versa.

[0071] Method 400 may further include a first data sample being an image of a template face, a second data sample being images of various faces, input data being images of faces to be normalized, and a classifier model providing classifications of facial action units (FAUs) present in the input images. Operation 402 may be performed if the first and second data samples are facial images. Method 400 may further include static features including facial structure, and dynamic features including mouth formation and eyelid formation.

[0072] Operation 402 can be performed in various ways. Many techniques exist for determining and altering facial poses and angles in images. Any of these techniques can be used at operation 402. Operation 402 can adjust the angle and pose of the face to a frontal view.

[0073] Artificial intelligence (AI) is a field that involves developing decision-making systems to perform cognitive tasks, which traditionally require a living actor, such as a person. A neural network (NN) is a computational structure based on loosely modeled biological neurons. Typically, NNs encode information (e.g., data or decisions) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the foundation for many AI applications, such as speech recognition.

[0074] Many neural networks (NNs) are represented as weight matrices corresponding to the modeled connections. NNs operate by taking data from a set of input neurons, which typically have many output connections to other neurons. In each traversal between neurons, the input is modified according to the corresponding weights and tested against a threshold at the destination neuron. If the weighted value exceeds the threshold, it is either reweighted or transformed through a non-linear function and passed to another neuron in the NN graph; if it does not exceed the threshold, it is typically not passed to the next neuron in the graph, and the synaptic connection remains inactive. The weighting and testing process continues until the output neurons are reached; the patterns and values ​​of the output neurons constitute the result of the ANN processing.

[0075] The correct operation of most neural networks (NNs) relies on accurate weights. However, NN designers often don't know which weights are appropriate for a given application. NN designers typically choose multiple layers of neurons or specific connections between layers, including circular connections. The training process can be used to determine appropriate weights by selecting initial weights. In some examples, initial weights can be chosen randomly. Training data is fed into the NN, and the results are compared to an objective function that provides an error indication. The error indication is a measure of how erroneous the NN's results are compared to the expected results. This error is then used to correct the weights. After several iterations, the weights will collectively converge to encode the operational data into the NN. This process can be called optimization of the objective function (e.g., cost or loss function) to minimize the cost or loss.

[0076] Gradient descent is a common technique used to perform objective function optimization. It computes gradients (e.g., partial derivatives) relative to layer parameters (e.g., aspects of the weights) to provide the direction and extent of the correction, but does not result in a single correction that sets the weights to the "correct" value. That is, through multiple iterations, the weights will move towards the "correct" or operationally useful value. In some implementations, the amount of movement or step size is fixed (e.g., the same from iteration to iteration). Small step sizes tend to take a long time to converge, while large step sizes may oscillate around the correct value or exhibit other undesirable behavior. Variable step sizes can be tried to provide faster convergence without the drawbacks of large step sizes.

[0077] Backpropagation is a technique that feeds training data forward through a neural network (NN). "Forward" means that the data starts from the input neurons, follows a directed graph of neuron connections, and continues until it reaches the output neurons. The objective function is then applied backward through the NN to correct the synaptic weights. At each step of the backpropagation process, the results of the previous step are used to correct the weights. Therefore, the corrected weights for the output neurons are applied to the neurons connected to them, and so on, until they reach the input neurons. Backpropagation has become a popular technique for training various neural networks. Any known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0078] Figure 5 This is a block diagram illustrating an example environment including a system for training a neural network according to an embodiment. The system includes an artificial neural network (ANN) 505 trained using processing nodes 510. Processing nodes 510 may be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), digital signal processor (DSP), application-specific integrated circuit (ASIC), or other processing circuitry. In this example, multiple processing nodes may be employed to train different layers of the ANN 505, or even different nodes 507 within a single layer. Therefore, a set of processing nodes 510 is arranged to perform training of the ANN 505.

[0079] The processing node set 510 is arranged to receive the training set 515 of ANN 505. ANN 505 includes a hierarchically arranged node set 507 (shown as rows of nodes 507) and a set of inter-node weights 508 (e.g., parameters) between nodes in the node set. In this example, the training set 515 is a subset of the full training set. This subset allows processing nodes with limited storage resources to participate in training ANN 505.

[0080] Training data can include multiple numerical values ​​representing the domain, such as red, green, and blue pixel values ​​and intensity values ​​for images, discrete-time pitch and volume values ​​for speech recognition, and / or amplitude, frequency, and magnitude for vital signs (e.g., heart activity, respiratory waves, etc.). Once the ANN 505 is trained, each value of the training or input 517 to be classified is fed to the corresponding node 507 in the first or input layer of the ANN 505. These values ​​are propagated through layers and modified by the objective function.

[0081] As described above, the set of processing nodes is arranged to train a neural network to create a trained neural network. Once trained, the data input to the ANN will produce valid classifications 520 (e.g., input data 517 will be assigned to a category). The training performed by the set of processing nodes 507 is iterative. In the example, each iteration of training the neural network is performed independently between the layers of ANN 505. Therefore, two different layers can be processed in parallel by different members of the set of processing nodes. In the example, the different layers of ANN 505 are trained on different hardware. The members of the different members of the set of processing nodes can reside in different packages, shells, computers, cloud-based resources, etc. In the example, each iteration of training is performed independently between the nodes in the node set. This example is an additional parallelization where a single node 507 (e.g., a neuron) is trained independently. In the example, the nodes are trained on different hardware.

[0082] Figure 6 Block diagrams of an embodiment of machine 600 (e.g., a computer system) for implementing one or more embodiments are shown by way of example. Machine 600 may use an autoencoder to implement techniques for image normalization. Autoencoders 101, 222, FAU model 330, or components thereof may include one or more components of machine 600. One or more of autoencoders 101, 222, FAU model 330, or components thereof may be implemented at least in part using components of machine 600. An exemplary machine 600 (in the form of a computer) may include a processing unit 602, a memory 603, a removable storage device 610, and a non-removable storage device 612. Although an example computing device is shown and described as machine 600, the computing device may take different forms in different embodiments. For example, the computing device may alternatively be a smartphone, tablet, smartwatch, or include... Figure 6 Other computing devices with the same or similar elements shown and described. Devices such as smartphones, tablets, and smartwatches are generally referred to as mobile devices. In addition, although various data storage elements are shown as part of machine 600, the storage device may also or alternatively include cloud-based storage accessible via a network (such as the Internet).

[0083] Memory 603 may include volatile memory 614 and non-volatile memory 608. Machine 600 may include or have access to a computing environment that includes various computer-readable media, such as volatile memory 614 and non-volatile storage device 608, removable storage device 610, and non-removable storage device 612. Computer storage devices include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technologies, optical disc read-only memory (CD ROM), digital versatile disc (DVD) or other optical disc storage, cassette tape, magnetic tape, disk storage devices, or other magnetic storage devices capable of storing computer-readable instructions to perform the functions described herein.

[0084] Machine 600 may include or have access to a computing environment including input 606, output 604, and communication connection 616. Output 604 may include a display device, such as a touchscreen, which can also be used as an input device. Input 606 may include one or more of the following: touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated into machine 600 or coupled to machine 600 via a wired or wireless data connection, and other input devices. The computer may use the communication connection to operate in a networked environment to connect to one or more remote computers, such as database servers, including cloud-based servers and storage. Remote computers may include personal computers (PCs), servers, routers, network PCs, peer-to-peer devices, or other public network nodes. Communication connections may include local area networks (LANs), wide area networks (WANs), cellular networks, IEEE 802.11 (Wi-Fi), Bluetooth, or other networks.

[0085] Computer-readable instructions stored on a computer-readable storage device are executable by a processing unit 602 (sometimes referred to as processing circuitry) of machine 600. Hard disk drives, CD-ROMs, and RAM are some examples of entries that include non-transitory computer-readable media such as storage devices. For example, a computer program 818 can be used to cause the processing unit 602 to perform one or more methods or algorithms described herein.

[0086] Other notes and examples:

[0087] Example 1 includes an apparatus comprising processing circuitry; and a memory including instructions that, when executed by the processing circuitry, cause the processing system to perform operations including: normalizing input data samples by combining dynamic characteristics of the characters in the input data samples with static characteristics of the characters in the first data samples, the static characteristics including the same characteristics in the first data samples; and providing the normalized data as input to a classifier model to classify the input data samples based on the dynamic characteristics of the input data samples and the static characteristics of the first data samples.

[0088] In Example 2, Example 1 may also include a normalized autoencoder that uses a single encoder and multiple decoders, wherein a first decoder is dedicated to reconstructing a first data sample and a second decoder is dedicated to reconstructing a second data sample.

[0089] In Example 3, Example 2 may also include an encoder trained based on the reconstruction loss of both the first decoder and the second decoder.

[0090] In Example 4, Example 3 may also include a first decoder trained only based on the reconstruction loss of the first decoder, and a second decoder trained only based on the reconstruction loss of the second decoder.

[0091] In Example 5, Example 4 may also include a normalized autoencoder operation that, during runtime, uses an encoder to compress the representation of the input data samples and uses a first decoder to construct the normalized data.

[0092] In Example 5, Example 6 may further include training an encoder and a second decoder on a batch of second data samples, followed by training the encoder and the first decoder on a batch of first data samples, or vice versa.

[0093] In Example 7, at least one of Examples 1-6 may further include, wherein the first data sample is an image of a template face, the second data sample is an image of various faces, the input data is an image of a face to be normalized, and the classifier model provides a classification of facial action units (FAUs) present in the input image.

[0094] In Example 8, Example 7 may further include, wherein the operation further includes, adjusting the angles and poses of the faces in the first data sample and the second data sample to be consistent before normalizing the input data, and wherein normalization is performed based on the input data for which the angles and poses are adjusted.

[0095] In Example 9, at least one of Examples 7-8 may further include, wherein the static characteristics include facial structure and the dynamic characteristics include mouth formation and eyelid formation.

[0096] Example 10 may include a computer-implemented method comprising obtaining a normalized autoencoder trained on a first data sample of a template person and second data samples of various persons; normalizing the input data sample by the normalized autoencoder by combining dynamic characteristics of the person in the input data sample with static characteristics in the first data sample to generate normalized data; and providing the normalized data as input to a classifier model to classify the input data based on the dynamic characteristics of the input data and the static characteristics of the first data sample.

[0097] In Example 11, Example 10 may also include a normalized autoencoder that uses a single encoder and multiple decoders, wherein a first decoder is dedicated to reconstructing a first data sample and a second decoder is dedicated to reconstructing a second data sample.

[0098] In Example 12, Example 11 may also include an encoder trained based on the reconstruction loss of both the first decoder and the second decoder.

[0099] In Example 13, Example 12 may also include a first decoder being trained based solely on the reconstruction loss of the first decoder, and a second decoder being trained solely on the reconstruction loss of the second decoder.

[0100] In Example 14, Example 13 may also include a normalized autoencoder operating on the input data using an encoder and a second decoder during runtime.

[0101] In Example 15, Example 14 may also include training the encoder and the first decoder on a second batch of data for a variable character, followed by training the encoder and the second decoder on a first batch of data for a template character, or vice versa.

[0102] In Example 16, at least one of Examples 10-15 may further include, wherein the first data sample is an image of a template face, the second data sample is an image of various faces, the input data is an image of a face to be normalized, and the classifier model provides a classification of facial action units (FAUs) present in the input image.

[0103] In Example 17, Example 16 may further include adjusting the angles and poses of the faces in the first and second data samples to be consistent before normalizing the input data, and wherein normalization is performed based on the input data for which the angles and poses are adjusted.

[0104] In Example 18, at least one of Examples 16-17 may further include, wherein the static characteristics include facial structure and the dynamic characteristics include mouth formation and eyelid formation.

[0105] Example 19 may include a non-transitory machine-readable medium, including instructions that, when executed by a machine, cause the machine to perform an operation comprising: normalizing an input data sample by combining dynamic characteristics of the characters in the input data sample with static characteristics of the first data sample, the static characteristics including the same characteristics in the first data sample; and providing the normalized data as input to a classifier model to classify the input data sample based on the dynamic characteristics of the input data sample and the static characteristics of the first data sample.

[0106] In Example 20, Example 19 may also include a normalized autoencoder that uses a single encoder and multiple decoders, wherein a first decoder is dedicated to reconstructing a first data sample and a second decoder is dedicated to reconstructing a second data sample.

[0107] In Example 21, Example 20 may also include an encoder trained based on the reconstruction loss of both the first decoder and the second decoder.

[0108] In Example 22, Example 21 may also include a first decoder being trained only based on the reconstruction loss of the first decoder, and a second decoder being trained only based on the reconstruction loss of the second decoder.

[0109] In Example 23, Example 22 may also include, during runtime, a normalization autoencoder operation using an encoder to compress a representation of the input data samples and using a first decoder to construct normalized data.

[0110] In Example 24, Example 23 may also include operations that further include training an encoder and a second decoder on a second batch of data samples, followed by training the encoder and the first decoder on a first batch of data samples, or vice versa.

[0111] In Example 25, at least one of Examples 19-24 may further include, wherein the first data sample is an image of a template face, the second data sample is an image of various faces, the input data is an image of a face to be normalized, and the classifier model provides a classification of facial action units (FAUs) present in the input image.

[0112] In Example 26, Example 25 may further include, wherein the operation further includes, adjusting the angles and poses of the faces in the first data sample and the second data sample to be consistent before normalizing the input data, and wherein normalization is performed based on the input data for which the angles and poses are adjusted.

[0113] In Example 27, Example 26 may also include, wherein static features include facial structure and dynamic features include mouth formation and eyelid formation.

[0114] While some embodiments have been described in detail above, other modifications are possible. For example, the logical flow shown in the figures does not require the indicated order or sequence to achieve the desired result. Other steps may be provided from the described flow, or steps may be eliminated, and other components may be added to or removed from the described system. Other embodiments are within the scope of the following claims.

Claims

1. A device for person characteristic normalization, comprising: processing circuitry; memory comprising instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations comprising: normalizing, by a normalization autoencoder, an input data sample of a person to generate normalized facial image data by combining dynamic characteristics of the person in the input data sample with static characteristics of a template person in a first data sample, the normalization autoencoder (i) being trained based on the first data sample of the template person and second data samples of various persons different from the template person, and (ii) comprising (a) a single encoder trained based on the first data sample and the second data samples, (b) a first decoder trained based on the first data sample only, and (c) a second decoder trained based on the second data samples only, the static characteristics comprising characteristics that are the same in the first data sample; and providing the normalized facial image data as input to a classifier model to classify facial action units (FAUs) of the normalized facial image data based on the dynamic characteristics of the person in the input data sample and the static characteristics of the template person in the first data sample, and the classifier model providing a classification of facial action units (FAUs) that are present in the input data sample.

2. The device of claim 1, wherein the first decoder is specialized to reconstruct the first data sample and the second decoder is specialized to reconstruct the second data sample.

3. The device of claim 2, wherein the encoder is trained based on reconstruction loss of both the first decoder and the second decoder.

4. The device of claim 3, wherein the first decoder is trained based on reconstruction loss of the first decoder only, and the second decoder is trained based on reconstruction loss of the second decoder only.

5. The device of claim 4, wherein during runtime, the normalization autoencoder operates to compress a representation of the input data sample using the encoder and to construct the normalized facial image data using the first decoder.

6. The device of claim 5, wherein the operations further comprise: the encoder and the second decoder are trained on a batch of the second data samples, followed by the encoder and the first decoder being trained on a batch of the first data samples, or vice versa.

7. The device of claim 1, wherein the input data is an image of a face to be normalized.

8. The device of claim 7, wherein the operations further comprise: prior to normalizing the input data, angles and poses of the face in the first data sample and the second data sample are adjusted to be consistent, and wherein normalizing is performed based on input data that is adjusted for angles and poses.

9. The device of claim 7, wherein the static characteristics comprise facial structure and the dynamic characteristics comprise mouth formation and eyelid formation.

10. A computer-implemented method for person characteristic normalization, comprising: obtaining a normalization autoencoder comprising (a) a single encoder trained based on first data samples and second data samples, (b) a first decoder trained based only on the first data samples of a template person, and (c) a second decoder trained based only on the second data samples of various persons different from the template person; normalizing, by the normalization autoencoder, an input data sample by combining dynamic characteristics of a person in the input data sample with static characteristics of the template person in the first data samples to generate normalized facial image data; and providing the normalized facial image data as input to a classifier model to classify facial action units (FAUs) of the normalized facial image data based on the dynamic characteristics of the person in the input data sample and the static characteristics of the template person in the first data samples, and the classifier model provides a classification of facial action units (FAUs) present in the input data sample.

11. The method of claim 10, wherein the first decoder is specialized to reconstruct the first data samples and the second decoder is specialized to reconstruct the second data samples, wherein the encoder is trained based on reconstruction losses of both the first decoder and the second decoder, and wherein the first decoder is trained based only on reconstruction losses of the first decoder and the second decoder is trained based only on reconstruction losses of the second decoder.

12. The method of claim 11, wherein during runtime, the normalization autoencoder operates on the input data using the encoder and the first decoder.

13. The method of claim 10, wherein the input data is an image of a face to be normalized.

14. The method of claim 13, further comprising adjusting angles and poses of the face in the first data samples and the second data samples to be consistent prior to normalizing the input data, and wherein normalizing is performed based on the input data adjusted for angles and poses, wherein the static characteristics comprise facial structure and the dynamic characteristics comprise mouth formation and eyelid formation.

15. A non-transitory machine readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations comprising: a normalized autoencoder trained from a first data sample based on a template person and a second data sample of various persons normalizes an input data sample by combining dynamic characteristics of a person in the input data sample with static characteristics of the first data sample to generate normalized facial image data, the static characteristics comprising characteristics that are the same in the first data sample, the normalized autoencoder (a) a single encoder trained based on the first data sample and the second data sample, (b) a first decoder trained based on only the first data sample of the template person, and (c) a second decoder trained based on only the second data sample of the various persons; and providing the normalized facial image data as input to a classifier model to classify a facial action unit (FAU) in the input data sample based on the dynamic characteristics of the person in the input data sample and the static characteristics of the template person in the first data sample, and the classifier model provides a classification of a facial action unit (FAU) that is present in the input data sample.

16. The non-transitory machine readable medium of claim 15, wherein the first decoder is specialized to reconstruct the first data sample and the second decoder is specialized to reconstruct the second data sample.

17. The non-transitory machine readable medium of claim 16, wherein the encoder is trained based on reconstruction loss of both the first decoder and the second decoder, and wherein the first decoder is trained based on only reconstruction loss of the first decoder and the second decoder is trained based on only reconstruction loss of the second decoder.

18. The non-transitory machine readable medium of claim 17, wherein during runtime, the normalized autoencoder operates on the input data using the encoder and the first decoder.

19. The non-transitory machine readable medium of claim 15, wherein the input data is an image of a face to be normalized.

20. The non-transitory machine readable medium of claim 19, further comprising: prior to normalizing the input data, angles and poses of the face in the first data sample and the second data sample are adjusted to be consistent, and wherein normalizing is performed based on input data that is adjusted for angles and poses, wherein the static characteristics comprise facial structure and the dynamic characteristics comprise mouth formation and eyelid formation.

Images