Multi-modal three-dimensional face modeling and tracking for generating expressive icons

Abstract

Examples are disclosed that relate to generating expressive icons using multi-modal three-dimensional face modeling and tracking. One example includes a computer system including a processor coupled to a storage system storing instructions. When executed by the processor, the instructions cause the processor to receive initialization data describing an initial state of the facial model. The instructions also cause the processor to receive a plurality of multi-modal data signals. The instructions also cause the processor to perform a fitting process using the initialization data and the plurality of multi-modal data signals. The instructions also cause the processor to determine a set of parameters based on the fitting process, wherein the determined set of parameters describes an updated state of the face model.

Description

Background Art

[0001] An avatar is a graphical representation of a user. An avatar can take the form of a virtual character that reflects the user's real-life self or has completely fictional characteristics. One area of ​​research includes three-dimensional computer models of animated facial expressions that can be used in various virtual reality / augmented reality / mixed reality (VR / AR / MR) applications. Of great interest is the ability to adjust the user's facial expressions to animate the computer model in a similar capacity. Different motion capture and computer vision techniques have been implemented to fit the user's facial expressions to the manipulated computer model to perform the desired animation. For example, landmark fitting techniques have been proposed to reconstruct the user's face into a three-dimensional morphable facial model. Summary of the invention

[0002] This summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure.

[0003] The present application discloses examples involving the use of multimodal three-dimensional facial modeling and tracking to generate expressive icons. One example includes a computer system including a processor coupled to a storage system storing instructions. When executed by the processor, the instructions cause the processor to receive initialization data describing an initial state of a facial model and receive a plurality of multimodal data signals. A fitting process is performed using the initialization data and the plurality of multimodal data signals. The fitting process is performed by simulating measurements using the initialization data and comparing the simulated measurements with actual measurements obtained from the plurality of multimodal data signals. The initialization data is updated based on the comparison of the simulated measurements with the actual measurements. The instructions cause the processor to determine a set of parameters based on the fitting process, the determined set of parameters describing an updated state of the facial model. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A schematic diagram of an example computing system including a computing device configured to perform a multimodal three-dimensional (3D) face modeling and tracking (MMFMT) process for determining facial expressions is shown.

[0005] Figure 2 A diagram schematically illustrates an example process for generating an expressive 3D facial model using multimodal data signals from a head mounted display.

[0006] Figure 3A diagram schematically illustrates an example process of determining expression parameters of a 3D face at a given moment using data signals from an eye camera, an antenna, and a microphone.

[0007] Figure 4 An example wearable device including multiple antennas is shown.

[0008] Figure 5 An example MMFMT fitter module implementing a parallel plate capacitor forward model is shown.

[0009] Figure 6 An example MMFMT fitter module implementing an eye camera forward model is shown.

[0010] Figure 7 An example MMFMT fitter module that implements the fitting process for an audio data signal is shown.

[0011] Figure 8 A flow chart of an example method for generating an expressive facial model using the MMFMT process is shown.

[0012] Fig. 9 An example computing system is schematically illustrated that may implement one or more of the methods and processes described above. DETAILED DESCRIPTION

[0013] Many techniques for 3D facial modeling and reconstruction based on the user's face have been proposed to create expressive icons for various VR / AR / MR applications. Some such techniques include recording and tracking the user's face to determine various facial landmarks. Facial landmarks can be mapped to a 3D facial model to enable animation of the model by tracking the movement of facial landmarks over a period of time. The use of additional inputs (such as depth imaging or differentiable rendering techniques) can also be implemented to more accurately reconstruct the user's face. However, these techniques are limited in their scope of application. For example, VR / AR / MR applications generally tend to be simple hardware implementations, and therefore may lack the ability to distinguish enough facial landmarks to reconstruct 3D facial models within an acceptable level of accuracy. Specifically, wearable devices (such as smart glasses or goggles) for VR / AR / MR applications may not include a camera for recording the user's entire face, and therefore may lack the ability to distinguish enough facial landmarks to create an expressive facial model.

[0014] In view of the above observations, examples related to multimodal 3D facial modeling and tracking for generating expressive icons are disclosed. The three-dimensional facial modeling and tracking technology according to the present disclosure utilizes input data from multiple different sensors to implement a multimodal framework for creating expressive 3D facial models. For example, the multimodal 3D facial modeling and tracking technology can utilize multiple different sensor devices, each of which provides one or more input signals and / or measurements for a user's face to detect, model, track, and / or vividly animate a three-dimensional facial model into an icon. In some examples, the three-dimensional facial modeling and tracking technology creates 3D vertices based on the user's face, and applies transformations to vertices from a neutral face to depict expressions on a digital facial model (e.g., an icon representation of the user's face). In some implementations, 3D vertices for the face are generated on a per-instance basis using 3D modeling from multimodal signals, and the vertices are tracked over time to create expressive animations.

[0015] In general, data signals from individual sensors typically found on wearable devices for VR / AR / MR applications are inherently noisy and fail to provide a holistic view of a user's facial expressions. Combining data signals from multimodal sources provides an improved framework for predicting a user's facial expressions. The framework can utilize sensors that have complementary characteristics to each other based on their associated correlations with the user's facial expressions. Different types of sensors and their associated data signals can be implemented in a multimodal framework. In some implementations, capacitance values ​​measured from an inductive / capacitive antenna on a wearable device are used in conjunction with image data and audio data of the user's eyes to determine the user's facial expressions. Various configurations of antenna circuits can be utilized, including LC oscillators and RC circuits.

[0016] In some implementations, the framework utilizes deep learning techniques and forward modeling to perform a parameter fitting process that converts a multimodal data signal into a set of parameters or expression codes that can be used to generate an expressive 3D facial model. The forward model receives an initialization parameter set that defines a face and simulates measurements related to the utilized data signal. The actual measurements from the data signal are compared with the simulated measurements to calculate the loss. The parameters are adjusted based on the calculated loss, and the process continues iteratively for a predetermined number of iterations or until the loss criterion is reached. The process outputs a set of parameters that can be used to generate an expressive 3D facial model. By using multimodal data signals, measurements with different units and different scales from different sensors are provided. In this way, the loss function utilized in the deep learning technique can be designed to take into account the corresponding data measurement type of the data signal. These and other MMFMT techniques are discussed in more detail below.

[0017] Figure 1 A schematic diagram of an example computing system 100 is shown having a computing device 102 configured to perform an MMFMT process for determining facial expressions in accordance with an implementation of the present disclosure. As shown, the computing device 102 includes a processor 104 (e.g., one or more central processing units, or "CPUs") and a memory 106 (e.g., volatile and non-volatile memory) operably coupled to each other. The memory 106 stores an MMFMT program 108 containing instructions for executing various software modules described herein by the processor 104. The memory 106 also stores data 110 for use by the MMFMT program 108 and its software modules.

[0018] When executed by the processor 104, the instructions stored in the MMFMT program 108 cause the processor 104 to retrieve the initialization data 112 from the data 110 stored in the memory 106 for use by the MMFMT program 108. The initialization data 112 provides information defining the initial state of the parametric model of the user's head. For example, the initialization data 112 may include data describing the expression of the initial 3D facial model. The initialization data 112 may also include data describing the identity of the initial 3D facial model, such as information about the shape, size, etc. of the head. The initialization data 112 may also include data describing the posture of the initial 3D facial model, such as information about the rotation and translation of the head, neck, and eyes of the initial 3D facial model. In some implementations, a zero expression facial model is used as the initial facial model. In some implementations, a learning process (such as by using a transformer or a long short-term memory neural network) is used to generate the initial 3D facial model.

[0019] The instructions stored in the MMFMT program 108 also cause the processor 104 to receive data signals 114 from various external sensors. As described above, different types of data signals 114 can be received. The type of sensor implemented depends on the data signal for which the MMFMT program 108 is configured. In some example implementations, the implemented sensor includes a sensor located on a wearable device, such as an antenna, an eye camera, a microphone, etc. A capacitance value can be received from an antenna located on the wearable device. Depending on the application, different numbers of antennas can be utilized. In some implementations, a wearable device with at least eight antennas is utilized. In other examples, a wearable device with less than eight antennas can be used. Audio data can be received from a microphone or any other suitable type of transducer device. Image data of the user's eyes can be received from a camera or any other suitable type of optical recording device.

[0020] The MMFMT program 108 includes an MMFMT fitter module 116 that receives initialization data 112 and a data signal 114 as input. The received data signal 114 may be converted to a suitable data format before being fed to the MMFMT fitter module 116. For example, in some implementations, the data signal 114 includes image data of a user's eyes. A detector module for determining eye landmarks may be used to first determine the landmarks from the image data, and the landmarks are then fed to the MMFMT fitter module 116. In another example, an audio-to-face model module may be used to convert audio data into expression parameters, and the audio expression parameters are fed to the MMFMT fitter module 116.

[0021] The MMFMT fitter module 116 performs a parameter fitting process to find a set of parameters 118 that can be used to generate an expressive 3D face model. The MMFMT fitter module 116 receives parameters describing the initial state of the face model and simulates measurements associated with the utilized sensors. The fitting process uses an iterative loop to find a set of parameters that produces simulated measurements that are close to the actual measurements (true values) of the data signal. For example, given a sensor reading m and a deterministic function f:Φ→m, the fitting process attempts to find a set of parameters Φ * So that f(Φ * )≈m. In many implementations, a generative model is defined for each signal domain.

[0022] The difference between the simulated measurement and the actual measurement is called loss, and the fitting process usually reduces the loss until a threshold condition is reached. For example, the fitting process can be implemented as iteratively reducing the difference between the simulated measurement and the actual measurement, and adjusting the parameters accordingly in an attempt to find a parameter set with a loss below a given threshold. Different loss functions can be used, such as L1 and L2 loss functions. Depending on the parameter space and the implementation of the MMFMT fitter module, the result of the fit can correspond to a global minimum or a local minimum.

[0023] In some implementations, the fitting process includes a separate loss function for each data signal. In such an example, the fitting process can be performed until a threshold condition for the sum of the loss functions is reached. The fitting process can include constraints to prevent undesirable states of the face model. For example, the fitting process can include regularization and priors to prevent parameters that cause abrupt changes in mesh vertices between adjacent frames, skin intersecting with eyeballs, etc.

[0024] Depending on the application, the MMFMT fitter module 116 can be implemented in various ways. For example, the type of data signal utilized can depend on the available hardware, such as the available sensors on the wearable device. In some such examples, the wearable device is in the form of glasses with various sensors, including eye cameras, antennas, and microphones. A variety of each sensor can be implemented.

[0025] Figure 2 A diagram 200 of an example process of converting a multimodal data signal from a head mounted display (HMD) in the form of a pair of glasses 202 into an expressive 3D facial model 204 is schematically shown. In many implementations, the pair of glasses 202 is a smart device for VR / AR / MR / applications. The pair of glasses 202 includes various sensors 206 for providing multimodal data signals, which are fed to the MMFMT fitter module 208 along with initialization information 210 describing the initial state of the facial model. The MMFMT fitter module 208 can operate on a per-instance basis to determine a given expression 204 at a given time. As can be readily appreciated, different types of measurements and data formats can be utilized to determine the user's expression, depending on the application and available hardware. In some implementations, the MMFMT process includes at least the use of an eye camera, an antenna, and a microphone for providing a multimodal data signal.

[0026] Figure 3 Schematically illustrates an example process of determining expression parameters of a three-dimensional face at a given moment using data signals from an eye camera 302A, an antenna 304A, and a microphone 306A. As described above, these sensors 302A-306A can be implemented on a wearable device such as smart glasses. These sensors 302A-306A provide their corresponding data signals, which can be converted into a suitable format that can be fed to the MMFMT fitter module 308. For example, the eye camera 302A produces an image that can be fed to the eye landmark detector module 302B to estimate and determine the eye landmark y eye 302C. Measuring the capacitance 304B from the antenna 304A may produce a capacitance value C 304C. The audio data signal from the microphone 306A may be fed to the audio-to-facial expression module 306B to generate an audio expression parameter ψ audio 306C.

[0027] The measurements 302C-306C from these sensors 302A-306A are fed to the MMFMT fitter module 308 along with a set of parameters 310 describing the initial state of the facial model. In the depicted diagram 300, the set of parameters 310 describing the initial state of the facial model includes parameters describing the initial expression ψ0 310A, the initial pose θ 310B, and the initial identity β 310C of the facial model. The MMFMT fitter module 308 may operate on a per instance basis. For a given instance, the MMFMT fitter module 308 iteratively traverses the parameter space, attempting to find the expression ψ0 that produces the minimum amount of loss. * In the example depicted, the expression ψ * Determined as

[0028] ψ * =argmin ψ (λ1L eyecam +λ2L RF +λ3L audio +L regularization ),

[0029] Among them, λ1, λ2, λ3 are weights, L eyecam , L RF , and L audio is the loss function, and L regularization is the function used to enforce a priori constraints.

[0030] The MMFMT fitter module 308 simulates measurements using parameters 310 describing the initial state of the face model. The type of measurement simulated depends on the data signal utilized. The loss function in the MMFMT fitter module 308 receives the actual measurement (y eye 302C, C 304C, and ψ audio 306C) and simulated measurements as input, and compare the two sets to determine the loss. The smaller the difference between the actual measurement and the simulated measurement, the smaller the loss generated. Then, the MMFMT fitter module 308 updates the parameters 310 based on the calculated loss, and performs the fitting process again in an iterative loop. The fitting process can be iteratively performed until a predetermined criterion is met. For example, the iterative process can continue until the output of the loss function is below a loss threshold. In some implementations, the predetermined criterion is met after performing a predetermined number of iterations. Once the predetermined criterion is met, the MMFMT fitter module 308 outputs the expression parameters for generating an expressive 3D facial model. 312.

[0031] Different modalities utilized in generating expressive icons present different problems in processing different data signals. For example, given an MMFMT model, a device implementing such a model may lack one or more of the data signals utilized in the model, or data signals may be missing at times. In such cases, synthesis can be used to model the signals, and these signals can be inserted when real data is missing.

[0032] Another challenge includes modeling the measurements using antennas on wearable devices and their simulations. Typically, changes in facial expressions result in changes in the capacitive system between the antenna and the user, which is observed as changes in the measured capacitance values. For example, when facial muscles move, the capacitance measured by the antenna can change based on the proximity of the facial surface to the corresponding antenna. Figure 4 An example wearable device 400 including multiple antennas is shown. As shown, the wearable device 400 includes a left antenna array 402L formed on a left lens 404L of the wearable device 400, and a right antenna array 402R formed on a right lens 404R of the wearable device 400. Each of the left antenna array 402L and the right antenna array 402R includes multiple antennas, each configured to sense different areas of the user's face. Each antenna is located near the surface of the face and forms a capacitance based on the distance between the antenna and the surface of the face. In other examples, the wearable device 400 may alternatively or additionally include one or more antennas arranged on a frame 406 of the wearable device 400.

[0033] The left lens 404L and the right lens 404R are supported by the frame 406, which is connected to the side frames 408L, 408R by optional hinge joints 410L, 410R. The left antenna array 402L and the right antenna array 402R are schematically depicted by the dotted lines on the left lens 404L and the right lens 404R, respectively, which indicate any spatial arrangement of the antennas. Other layouts can be implemented. The term "lens" is used herein to refer to one or more optical components through which the real world environment can be viewed. The term "lens" can include an optical combiner that combines virtual and real images, and / or one or more transparent optical components other than the combiner, such as a separate lens with or without optical power.

[0034] Each lens 404L, 404R includes an at least partially optically transparent electrically insulating substrate. For example, the substrate may include glass, or an optically transparent plastic such as polycarbonate, polymethyl methacrylate (PMMA), polystyrene, polyethylene terephthalate (PET), cyclic olefin polymer, or other suitable material.

[0035] Antenna arrays 402L, 402R are formed of at least partially optically transparent conductive films. The thin films may include one or more conductive materials, such as indium tin oxide (ITO), silver nanowires, silver nanoparticles, carbon nanotubes, graphene, a mixture of two or more such materials (e.g., a silver nanoparticle-ITO mixture), and / or other suitable (multiple) materials. The (multiple) thin films may be formed by any suitable process, such as chemical vapor deposition, sputtering, atomic layer deposition, evaporation, or liquid phase application (e.g., spin coating, dip coating, blade application, etc.). The grooves formed between the antennas may be used to place conductive traces. Since the conductive film may not be completely optically transparent in some examples, using a relatively thin film for the antenna may provide greater transparency compared to a relatively thick coating.

[0036] The wearable device 400 also includes a plurality of charge sensing circuits, schematically shown at 412. Each of the plurality of charge sensing circuits 412 is connected to a corresponding antenna. Each charge sensing circuit 412 is configured to determine the capacitance of the corresponding antenna, for example by determining the amount of charge accumulated on the corresponding antenna due to application of a reference voltage.

[0037] Wearable device 400 also includes controller 414. Controller 414 includes, among other components, a logic subsystem and a storage subsystem that stores instructions executable by the logic subsystem to control various functions of wearable device 400, including, but not limited to, the facial tracking functions described herein.

[0038] Measurements from the simulation of the antenna can be calculated by modeling the antenna and the user's face as a capacitive system. Geometry, materials, sensor placement, etc. are all factors that affect the capacitance of the system. A relatively low complexity implementation of the finite element method can be used to measure the capacitance. However, this method is non-differentiable and can be slow, operating at, for example, approximately 1200 frames per hour. Therefore, in some implementations, the MMFMT model uses an approximation-based approach. One example approximation-based approach uses a parallel plate capacitor model. For a given antenna, the parallel plate capacitor model approach includes dividing the antenna into triangles. Using the triangular mesh of the 3D face model, antenna-face triangle pairs are formed by finding the closest triangle on the mesh for each antenna triangle. Each antenna-face triangle pair can be considered a capacitor placed in parallel. Then, the capacitance C of each pair of triangles △ can be determined as Where ε0 is the dielectric constant of free space, A is the area, and d is the distance between the triangle pairs. The capacitance values ​​may be summed to determine the effective capacitance of a given antenna. In some implementations, a weighted sum is used to determine the effective capacitance of a given antenna.

[0039] Since the forward model will be called in each iteration of the fitting process, the computational speed of the model is a factor that needs to be considered. However, the computations involved in the parallel plate capacitor model described above can present challenges. For example, wearable devices that implement this approach are typically small devices. The power and size constraints of the available hardware can present problems in terms of computational power. Techniques for simplifying computations and reducing the amount of memory utilized can be performed to accommodate this use case. For example, for a given antenna triangle, determining the closest facial triangle can be simplified by comparing only the distances of a candidate subset of facial triangles. A candidate subset of facial triangles can be pre-computed by finding a predetermined number K of closest candidate triangles for each antenna triangle under zero expression conditions. This reduces the search space for the closest triangle calculation.

[0040] Depending on the hardware and capacitance model implemented, the simulated capacitance value may not match the hardware measurement. In this case, a calibration step is performed to determine a mapping function that maps a given parallel plate capacitor simulated signal value to a hardware signal. In some implementations, a linear fit mapping function is utilized for each antenna (i.e., a linear regression is performed to map the parallel plate capacitor simulated signal value to the hardware signal). Other types of mapping functions may be utilized depending on the application. Example methods include min-max normalization, joint fitting, neural network-based fitting, etc.

[0041] As described above, the MMFMT fitter module can be implemented using a forward model. This model is typically a generative model specific to a given signal in a signal domain. The forward model receives parameters defining the face and simulates measurements, which in the case of a data signal from an antenna are simulated capacitance values. Figure 5 An example MMFMT fitter module 500 implementing a parallel plate capacitor forward model 502 is shown. As shown, the parallel plate capacitor forward model 502 is implemented as a module that receives initialization data 504 as input. The initialization data 504 includes parameters describing an initial expression ψ0 504A, an initial pose θ 504B, and an initial identity β 504C of a facial model. Based on the initial facial model and the antenna of the wearable device 506, the parallel plate capacitor forward model module 502 can simulate a capacitance measurement of the capacitive system and output a simulated capacitance value 508. When the user wears the wearable device, an actual measurement 510 is calculated by measuring the capacitance 512 across the antenna 506 of the wearable device. The simulated measurement 508 is compared to the actual measurement 510 using a loss function, and parameters are adjusted based on the comparison through back propagation. The process continues iteratively until a predetermined criterion is met. For example, the iterative process can continue until the output of the loss function is below a loss threshold. In some implementations, the predetermined criterion is met after performing a predetermined number of iterations. Once the predetermined criterion is met, the MMFMT fitter module 500 outputs expression parameters for generating an expressive 3D facial model. 514.

[0042] Another sensor set that can be used in the MMFMT process is an eye camera. Many wearable headphones or glasses include cameras facing the user's eyes. Typically, these cameras are used for gaze estimation and eye tracking for various applications. The MMFMT process according to the present disclosure can utilize such an eye camera to determine the expression in the eye region of the face. In addition, the eye camera can also provide reasonable priors for the expression in the lower region of the face. For example, a face performing a "surprised" expression will include a set of expressions near the eyes and mouth that are similar in several "surprised" expressions. Therefore, the expression near the eyes can be related to the expression near the mouth, and the expression in one area can be inferred by the expression in another area.

[0043] The eye camera can provide image data to the MMFMT model. In order to provide target metrics to the fitting process, eye landmarks are first determined from the image data. In some implementations, an eye landmark detector module is implemented to determine the eye landmarks. The eye landmark detector module can be developed through a training process using a synthetic training pipeline to regress many different landmarks on the eye. In some such examples, the training process regresses eighty landmarks on the eye. These landmarks can then be used in the fitting process to fit the parameters.

[0044] Figure 6An example MMFMT fitter module 600 implementing an eye camera forward model 602 is shown. As shown, the eye camera forward model 602 is implemented as a module that receives initialization data 604 as input. The initialization data 604 includes parameters describing an initial expression ψ0 604A, an initial posture θ604B, and an initial identity β604C of a facial model. Based on the initial facial model, the eye camera forward model 602 simulates eye landmarks 606. The actual eye landmarks 608 are determined using image data from one or more eye cameras 610 on a wearable device. An eye landmark detector module 612 is implemented to receive image data and output eye landmarks 608. The simulated eye landmarks 606 are compared with the actual eye landmarks 608 using a loss function, and parameters are adjusted based on the comparison by back propagation. The process is continued iteratively until a predetermined criterion is met. For example, the iterative process can continue until the output of the loss function is below a loss threshold. In some implementations, the predetermined criterion is met after performing a predetermined number of iterations. Once the predetermined criteria are met, the MMFMT fitter module 600 outputs expression parameters used to generate an expressive 3D facial model. 614.

[0045] Another modality of data signals that can be used in the MMFMT process includes the use of audio data. Figure 7 An example MMFMT fitter module 700 implementing a fitting process for an audio data signal is shown. An audio data signal is received from a microphone 702. The microphone 702 may be implemented on a wearable device. An audio-to-face model module 704 is implemented to receive the audio data signal. The audio-to-face model module 704 uses the audio data signal to generate an audio expression parameter ψ audio 706.

[0046] The MMFMT fitter module 700 receives initialization data 708 as input. The initialization data 708 includes parameters describing an initial expression ψ0 708A, an initial pose θ 708B, and an initial identity β 708C of the facial model. A mesh generation module 710 may be used to generate a simulated facial mesh 712 using the initialization data 708 including the initial expression ψ0 708A. Similarly, the mesh generation module 710 may be used to generate a simulated facial mesh 712 using the audio expression ψ audio706 and / or initialization data 708 to generate an actual facial mesh 714. In some implementations, the initial identity β708C parameters and their corresponding expression parameters are used to generate the simulated facial mesh 712 and the actual facial mesh 714. In further implementations, the initial pose θ708B may also be used to generate the simulated facial mesh 712 and the actual facial mesh 714. The simulated facial mesh 712 is compared to the actual facial mesh 714 using a loss function, and the parameters are adjusted based on the comparison. The process continues iteratively until a predetermined criterion is met. For example, the iterative process may continue until the output of the loss function is below a loss threshold. In some implementations, the predetermined criterion is met after performing a predetermined number of iterations. Once the predetermined criterion is met, the MMFMT fitter module 700 outputs the expression parameters used to generate an expressive 3D facial model. 716.

[0047] Combining multimodal data signals from various sensors (such as those described above) provides an improved framework for predicting a user's facial expressions. The framework is further improved by using sensors that have complementary characteristics to each other. For example, using a combination of antennas, eye cameras, and microphones provides lower errors (the distance between simulated and actual measurements) for most facial areas compared to using any single sensor. For certain areas of the face, certain sensors may be more reliable than other sensors. For example, the use of antennas performs well in the eye, cheek, and nose areas. On the other hand, the use of antennas, eye cameras, and / or microphones may be less predictive about the ear area.

[0048] Figure 8 A flow chart illustrating an example method 800 for generating an expressive facial model using a multimodal 3D facial modeling and tracking process is shown. At 802, the method 800 includes receiving initialization data describing an initial state of a facial model. Depending on the application, different types of initialization data may be implemented. In some implementations, the initialization data includes an initial parameter set describing the initial state of the facial model. For example, the initialization data may include a parameter ψ0 describing the expression of the initial state of the facial model. The initialization data may also include a parameter β describing the identity of the initial state of the facial model, such as information about the head shape, size, etc. The initialization data may also include a parameter θ describing the pose of the initial state of the facial model, such as information about the rotation and translation of the head, neck, and eyes of the facial model. The initial state utilized may depend on the application. For example, the initial state may be a zero expression state. In some implementations, the initial state is a previous state of the facial model in a real-time application. In other implementations, the initial state is generated by passing an audio data signal through an audio-to-facial model module.

[0049] At 804, method 800 includes receiving a plurality of multimodal data signals. Depending on the application, different types of data signals may be implemented. In some implementations, the plurality of multimodal data signals include a first data signal received from an eye camera, a second data signal received from a set of antennas, and a third data signal received from a microphone. The data signal from the eye camera may be received in the form of image data. In further implementations, the image data from the eye camera is used to obtain the eye landmark set y. eye The eye landmark y can be determined using the eye landmark detector module eye The data signal from the antenna set may include a capacitance measurement from the antenna set. The data signal from the microphone may be received in the form of audio data. In a further implementation, the audio data is used to obtain the expression ψ audio The audio-to-face model module can be used to determine the audio expression ψ audio .

[0050] At 806, method 800 includes performing a fitting process using the received initialization data and the received plurality of multimodal data signals. The fitting process may include solving ψ * =argmin ψ (λ1L eyecam +λ2L RF +λ3L audio +L regularization ), where λ1, λ2, λ3 are weights, L eyecam , L RF , and L audio is the loss function, and L regularization is the function used to enforce a priori constraints.

[0051] The fitting process can be performed using an iterative learning process. At sub-step 806A, iterations of the process can include simulating measurements using the received initialization data. Depending on the type of data signal utilized, different simulation techniques can be performed. For example, in an implementation where the multimodal data signal includes a data signal received from a set of antennas, a parallel plate capacitor model can be used to simulate capacitance values. Such a process can include dividing the antennas within the set of antennas into a plurality of antenna triangles. For each antenna triangle, the facial triangle closest to the antenna triangle is determined based on a predetermined distance metric. An example distance metric includes a Euclidean distance metric. A facial triangle is a triangle within a triangular mesh of an initial state of the facial model. For each antenna-face triangle pair, the capacitance value C △ is calculated as Where ε0 is the dielectric constant of free space, A is the area, and d is the distance between the triangle pairs. The capacitance value C of each antenna-face triangle pair in the calculated antenna-face triangle pair can be calculated based on △ To calculate the simulated capacitance C simulated In some implementations, the capacitance value C of each antenna-face triangle pair is calculated by △ Sum to calculate C simulated In some implementations, C is calculated using a weighted sum of the capacitance values ​​of each antenna-face triangle pair. simulated .

[0052] At 806B, the iteration includes comparing the simulated measurement with the actual measurement obtained from the plurality of multimodal data signals. The comparison may include using a loss function to find the difference between the two measurements.

[0053] At 806C, the iteration includes updating the initialization data based on the comparison of the simulated measurement with the actual measurement. The iterative process can continue until the comparison of the simulated measurement with the actual measurement reaches a predetermined threshold. For example, when the difference between the simulated measurement based on the loss function and the actual measurement is lower than a predetermined loss threshold, the iterative process can terminate to output a set of parameters. Various neural network architectures (including convolutional neural networks (CNNs), recurrent neural networks, bidirectional long short-term memory RNNs, encoder-decoder transformers, encoder-only transformers, Siamese networks, etc.) can be used to implement the fitting process. Additionally or alternatively, various nonlinear optimizers (including nonlinear optimizers using Hessian, quasi-Newton, gradient descent, and / or Levenberg-Marquardt type methods) can be used to implement the fitting process.

[0054] At 808, method 800 includes determining a parameter set based on the fitting process, wherein the determined parameter set describes an updated state of the face model. In some implementations, the determined parameter set includes identity parameters similar to identity parameters in the initial parameter set.

[0055] The methods and processes described herein may be closely related to the computing systems of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0056] Fig. 9A non-limiting embodiment of a computing system 900 is schematically illustrated that can implement one or more of the methods and processes described herein. The computing system 900 is illustrated in simplified form. The computing system 900 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices.

[0057] The computing system 900 includes a logic machine 902 and a storage machine 904. The computing system 900 may optionally include a display subsystem 906, an input subsystem 908, a communication subsystem 910, and / or Fig. 9 Other components not shown.

[0058] The logic machine 902 includes one or more physical devices configured to execute instructions. For example, the logic machine 902 may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise achieve a desired result.

[0059] The logic machine 902 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine 902 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic machine 902 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logic machine 902 may optionally be distributed between two or more separate devices that may be remotely located and / or configured for coordinated processing. Aspects of the logic machine 902 may be virtualized and may be executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0060] Storage machine 904 includes one or more physical devices configured to store instructions executable by logic machine 902 to implement the methods and processes described herein. When implementing such methods and processes, the state of storage machine 904 may be transformed—for example, to store different data.

[0061] Storage machine 904 may include removable and / or built-in devices. Storage machine 904 may include, among other things, optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.). Storage machine 904 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0062] It should be understood that storage device 904 includes one or more physical devices. However, various aspects of the instructions described herein may alternatively be propagated by a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not preserved by a physical device for a limited duration.

[0063] Aspects of the logic machine 902 and the storage machine 904 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0064] The terms "module," "program," and "engine" may be used to describe aspects of the computing system 900 that are implemented to perform specific functions. In some cases, a module, program, or engine may be instantiated by a logic machine 902 executing instructions stored by a storage machine 904. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may cover a single executable file or a group of executable files, a data file, a library, a driver, a script, a database record, etc.

[0065] It should be understood that a "service" as used herein is an application that can be executed across multiple user sessions. A service can be used for one or more system components, programs, and / or other services. In some implementations, a service can run on one or more server computing devices.

[0066] When included, the display subsystem 906 can be used to present a visual representation of the data stored by the storage machine 904. This visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the storage machine 904, and thus convert the state of the storage machine 904, the state of the display subsystem 906 can also be converted to visually represent the change of the underlying data. The display subsystem 906 may include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logic machine 902 and / or the storage machine 904 in a shared shell, or such a display device can be a peripheral display device.

[0067] When included, the input subsystem 908 may include or interface with one or more user input devices (such as a keyboard, mouse, touch screen, or game controller). In some embodiments, the input subsystem 908 may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be processed on or off the board. Example NUI components may include microphones for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.

[0068] When included, the communication subsystem 910 can be configured to communicatively couple the computing system 900 with one or more other computing devices. The communication subsystem 910 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem 910 can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem 910 can allow the computing system 900 to send messages to other devices and / or receive messages from other devices via a network (such as the Internet).

[0069] Another aspect includes a computer system for generating expressive icons using multimodal three-dimensional facial modeling and tracking. The computer system includes a processor coupled to a storage system storing instructions, which, when executed by the processor, causes the processor to receive initialization data describing an initial state of a facial model. The instructions also cause the processor to receive a plurality of multimodal data signals. The instructions also cause the processor to perform a fitting process using the received initialization data and the received plurality of multimodal data signals. The instructions also cause the processor to determine a set of parameters based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model. In this regard, additionally or alternatively, performing the fitting process includes iteratively performing simulated measurements using the initialization data, comparing the simulated measurements with actual measurements obtained from the plurality of multimodal data signals, and updating the initialization data based on the comparison of the simulated measurements with the actual measurements. In this regard, additionally or alternatively, the set of parameters is determined based on the updated initialization data of the iteration of the fitting process, wherein the comparison of the simulated measurements with the actual measurements satisfies a loss threshold. In this regard, additionally or alternatively, the plurality of multimodal data signals include a first data signal received from the eye camera, a second data signal received from the antenna, and a third data signal received from the microphone. In this regard, additionally or alternatively, performing the fitting process includes solving ψ * =argmin ψ (λ1L eyecam +λ2L RF +λ3L audio +L regularization ), where λ1, λ2, λ3 are weights, L eyecam , L RF , and L audio is the loss function, and L regularizationis a function for enforcing a priori constraints. In this regard, additionally or alternatively, the initialization data includes an initial parameter set describing the identity, expression, and posture of the facial model. In this regard, additionally or alternatively, the determined parameter set has identity parameters similar to the initial parameter set. In this regard, additionally or alternatively, the multiple multimodal data signals include data signals received from an antenna set, and performing the fitting process includes simulating capacitance values ​​using a parallel plate capacitor model. In this regard, additionally or alternatively, the storage system stores further instructions that, when executed by the processor, cause the processor to perform a calibration process to map the simulated capacitance values ​​to actual capacitance values. In this regard, additionally or alternatively, using a parallel plate capacitor model to simulate capacitance values ​​includes dividing the antennas within the antenna set into a plurality of antenna triangles, determining a plurality of antenna-face triangle pairs by determining a face triangle closest to each antenna triangle based on a distance metric, wherein the face triangles are part of a triangular mesh of an initial state of a face model, calculating capacitance of each of the plurality of antenna-face triangle pairs, and calculating simulated capacitance values ​​based on the calculated capacitance of each of the plurality of antenna-face triangle pairs.

[0070] Another aspect includes a method for generating an expressive icon using multimodal three-dimensional facial modeling and tracking. The method includes receiving initialization data describing an initial state of a facial model. The method also includes receiving a plurality of multimodal data signals. The method also includes performing a fitting process using the received initialization data and the received plurality of multimodal data signals. The method also includes determining a set of parameters based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model. In this regard, additionally or alternatively, performing the fitting process includes iteratively performing simulated measurements using the initialization data, comparing the simulated measurements with actual measurements obtained from the plurality of multimodal data signals, and updating the initialization data based on the comparison of the simulated measurements with the actual measurements. In this regard, additionally or alternatively, determining the set of parameters based on the updated initialization data of the iteration of the fitting process, wherein the comparison of the simulated measurements with the actual measurements satisfies a loss threshold. In this regard, additionally or alternatively, the plurality of multimodal data signals include a first data signal received from an eye camera, a second data signal received from an antenna, and a third data signal received from a microphone. In this regard, additionally or alternatively, performing the fitting process includes solving ψ * =argmin ψ (λ1L eyecam +λ2L RF +λ3L audio +L regularization ), where λ1, λ2, λ3 are weights, L eyecam, L RF , and L audio is the loss function, and L regularization is a function for enforcing a priori constraints. In this regard, additionally or alternatively, the initialization data includes an initial parameter set describing the identity, expression, and posture of the facial model. In this regard, additionally or alternatively, the determined parameter set has identity and posture parameters similar to the initial parameter set. In this regard, additionally or alternatively, the plurality of multimodal data signals include data signals received from an antenna set, and wherein performing the fitting process includes simulating capacitance values ​​using a parallel plate capacitor model. In this regard, additionally or alternatively, simulating capacitance values ​​using a parallel plate capacitor model includes dividing the capacitive antenna within the antenna set into a plurality of antenna triangles, determining a plurality of antenna-face triangle pairs by determining the face triangle closest to the antenna triangle for each antenna triangle based on a distance metric, wherein the face triangle is part of a triangular mesh of an initial state of the facial model, calculating the capacitance of each of the plurality of antenna-face triangle pairs, and calculating simulated capacitance values ​​based on the capacitance of each of the plurality of antenna-face triangle pairs calculated.

[0071] Another aspect includes a head-mounted display for generating expressive icons using multimodal three-dimensional facial modeling and tracking. The wearable device includes an antenna set, an eye camera set, a microphone, and a processor coupled to a storage system storing instructions, which, when executed by the processor, causes the processor to receive initialization data describing an initial state of a facial model. The instructions also cause the processor to receive a plurality of multimodal data signals, including a first data signal from the antenna set, a second data signal from the eye camera set, and a third data signal from the microphone. The instructions also cause the processor to perform a fitting process using the received initialization data and the received plurality of multimodal data signals, the fitting process being performed by iteratively using the initialization data to perform simulated measurements, comparing the simulated measurements with actual measurements obtained from the plurality of multimodal data signals, and updating the initialization data based on the comparison of the simulated measurements with the actual measurements. The instructions also cause the processor to determine a parameter set based on the fitting process, wherein the determined parameter set describes an updated state of the facial model.

[0072] It will be appreciated that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. Specific routines or methods described herein can represent one or more strategies in any number of processing strategies. Therefore, various actions shown and / or described can be performed in the order shown and / or described, in other orders, in parallel, or omitted. Similarly, the order of the process described above can be changed.

[0073] The subject matter of the present disclosure includes all novel and nonobvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computer system for generating expressive icons using multimodal three-dimensional facial modeling and tracking, the computer system comprising: A processor coupled to a storage system storing instructions that, when executed by the processor, cause the processor to: receiving initialization data describing an initial state of the facial model; receiving a plurality of multimodal data signals; performing a fitting process using the received initialization data and the received plurality of multimodal data signals; as well as A set of parameters is determined based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model.

2. The computer system of claim 1 , wherein performing the fitting process comprises iteratively performing: simulating a measurement using the initialization data; comparing the simulated measurements with actual measurements obtained from the plurality of multimodal data signals; and The initialization data is updated based on the comparison of the simulated measurements with the actual measurements.

3. The computer system of claim 2, wherein the set of parameters is determined based on the updated initialization data of an iteration of the fitting process, wherein the comparison of the simulated measurements to the actual measurements satisfies a loss threshold.

4. The computer system of claim 1, wherein the plurality of multimodal data signals comprises a first data signal received from an eye camera, a second data signal received from an antenna, and a third data signal received from a microphone.

5. The computer system of claim 4, wherein performing the fitting process comprises solving ψ * =argmin ψ (λ1L eyecam +λ2L RF +λ3L audio +L regularization ), where λ1, λ2, λ3 are weights, L eyecam , L RF , and L audio is the loss function, and L regularization is the function used to enforce a priori constraints.

6. The computer system of claim 1, wherein the initialization data comprises an initial set of parameters describing the identity, expression, and pose of the facial model. 7 . The computer system of claim 6 , wherein the determined parameter set has similar identity parameters as the initial parameter set.

8. The computer system of claim 1, wherein the plurality of multimodal data signals comprises data signals received from a set of antennas, and wherein performing the fitting process comprises simulating capacitance values ​​using a parallel plate capacitor model.

9. The computer system of claim 8, wherein the storage system stores further instructions that, when executed by the processor, cause the processor to: A calibration process is performed to map the simulated capacitance values ​​to actual capacitance values.

10. The computer system of claim 8, wherein simulating the capacitance value using the parallel plate capacitor model comprises: Dividing the antennas in the antenna set into a plurality of antenna triangles; Multiple antenna-face triangle pairs are determined by: For each antenna triangle, determining a face triangle that is closest to the antenna triangle based on a distance metric, wherein the face triangle is part of a triangle mesh of the initial state of the face model; calculating a capacitance for each triangle pair of the plurality of antenna-face triangle pairs; as well as The simulated capacitance value is calculated based on the calculated capacitance for each triangle pair in the plurality of antenna-face triangle pairs.

11. A method for generating expressive icons using multimodal three-dimensional facial modeling and tracking, the method comprising: receiving initialization data describing an initial state of the facial model; receiving a plurality of multimodal data signals; performing a fitting process using the received initialization data and the received plurality of multimodal data signals; as well as A set of parameters is determined based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model.

12. The method of claim 11, wherein performing the fitting process comprises iteratively performing: simulating a measurement using the initialization data; comparing the simulated measurements with actual measurements obtained from the plurality of multimodal data signals; and The initialization data is updated based on the comparison of the simulated measurements with the actual measurements.

13. The method of claim 11, wherein the plurality of multimodal data signals comprises a first data signal received from an eye camera, a second data signal received from an antenna, and a third data signal received from a microphone.

14. The method of claim 11, wherein the initialization data comprises an initial set of parameters describing the identity, expression, and pose of the facial model.

15. The method of claim 11, wherein the plurality of multimodal data signals include data signals received from a set of antennas, and wherein performing the fitting process includes simulating capacitance values ​​using a parallel plate capacitor model by: Dividing the capacitive antenna in the antenna set into a plurality of antenna triangles; Multiple antenna-face triangle pairs are determined by: For each antenna triangle, determining a face triangle that is closest to the antenna triangle based on a distance metric, wherein the face triangle is part of a triangle mesh of the initial state of the face model; calculating a capacitance for each triangle pair of the plurality of antenna-face triangle pairs; as well as The simulated capacitance value is calculated based on the capacitance calculated for each triangle pair in the plurality of antenna-face triangle pairs.

Images