Visual audio processing assembly, system and method

Abstract

A camera sensor system (212 / 306 / 308 / 318 / 320) generates RGB / IR images to track some challenging facial features such as the corners of the mouth and teeth during speech. The system also uses light intensity change signals from an event-driven sensor (EDS) (212 / 306 / 318) and also uses audio analysis using signals from a microphone (328). In this way, the camera sensor system allows for improved tracking performance (equivalent to using an ultra-high speed camera) with smaller bandwidth and less power consumption.

Description

[Technical field]

[0001] This application generally relates to technically inventive and unconventional solutions that are necessarily rooted in computer technology and result in concrete technical improvements. [Background technology]

[0002] When performing face tracking during speech for speaker identification in noisy environments, or for fake video detection, or for disambiguating speech recognition, or for animation, or for other purposes, dark areas in the mouth, parts of the face such as teeth, and fast movement of facial structures during speech are obstacles to accurate tracking. Summary of the Invention

[0003] The technical challenge posed by the above is that for better operation, high speed cameras may be required to reduce latency and improve tracking performance, which requires an increased frame rate of the camera data, but such high frame rates require greater bandwidth and processing, and therefore relatively large amounts of power consumption and heat generation.

[0004] To address the problems described herein, a camera sensor system is provided that includes a sensor cell equipped with both light intensity photodiodes under color filters and, if necessary, under infrared filters to capture RGB and IR images, as well as an EDS sensing cell that detects motion by the principle of event-driven sensors (EDS). EDS uses changes in light intensity sensed by one or more camera pixels as an indication of motion. EDS has high dynamic range (HDR), no motion blur, and low latency compared to RGB cameras. When EDS information is fused with RGB camera information and audio information, tracking becomes more robust. In situations of fast motion (e.g., mouth movement) or HDR, EDS information may be relatively more dependent on EDS information than in situations of slow motion and fine details (color, texture), which are situations that rely more on the camera image. Such fusion can also be applied to face tracking, eye tracking, and emotion recognition.

[0005] In this principle, raw event data from the EDS is used to fuse with RGB camera and audio data and then input to a classifier, which is trained using a training set of all three inputs from audio / camera / event data, in some implementations using a recurrent neural network with convolutional layers.

[0006] Thus, an assembly includes at least one camera unit configured to generate a red-green-blue (RGB) image of a face. The assembly further includes at least one event-driven sensor (EDS) configured to output a signal representative of a change in illumination of the face. At least one microphone is configured to output a signal representative of speech. Additionally, the assembly includes at least one processor configured with executable instructions to receive signals from the camera unit, the EDS, and the microphone. The instructions are executable to execute at least one neural network to generate at least one of an emotion prediction, a tracking of at least a portion of the face based on the signals from the camera unit, the EDS, and the microphone.

[0007] In some examples, the camera unit is configured to generate infrared (IR) images.

[0008] In an exemplary embodiment, the camera unit, processor, and EDS may be located on a single chip.

[0009] In a non-limiting embodiment, the portion of the face that is tracked may be one or more eyes, specifically one or more pupils, may be limited to the pupils, or may include other facial features, in other embodiments, the portion may include the corners of the mouth, may be limited to the corners of the mouth and / or the interior of the mouth including the teeth, or may include additional facial features.

[0010] In another aspect, a system includes at least one camera unit configured to generate a red-green-blue (RGB) image and / or an infrared (IR) image of a person. The system also includes at least one microphone and at least one event-driven sensor (EDS) configured to output a signal representative of the person. The system further includes at least one processor programmed with instructions to process an output of the microphone using a short-time Fourier transform (STFT) and to process an output of the STFT using at least one audio processing convolutional neural network (CNN). The instructions are executable to process at least features of an image from the camera unit using at least one visual processing CNN. Furthermore, the instructions are executable to process a representation of an output signal from the EDS using at least one event processing CNN. The instructions in the system can be executed to fuse outputs of the CNNs in a fully connected neural network layer to generate one or more of a prediction of an emotion of the person, tracking at least a portion of the face of the person, at least one virtual reality (VR) image of the person, and an identification of the person.

[0011] In one example of this latter aspect, the processor may be configured with instructions to process an output of the CNN using a recurrent neural network (RNN) and to process the output of the RNN using a fully connected neural network layer to generate a human mouth track.

[0012] In another aspect, a method includes receiving a signal from at least one camera unit, receiving a signal from at least one event-driven sensor (EDS), and receiving a signal from at least one microphone, and executing at least one neural network to generate at least one of emotion prediction, tracking at least a portion of a face, identifying a person, and generating a virtual reality (VR) image of the person based on the signals from the camera unit, the EDS, and the microphone.

[0013] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram of an exemplary system including an example according to the present principles; [Diagram 2] 1 shows a simplified sensor data flow. [Diagram 3] The sensors are shown in relation to the face of the person being tracked. [Figure 4] An example system is shown in block diagram form. [Diagram 5] Illustrates data flow from RGB input, audio input, and EDS input for emotion or speaker recognition. [Figure 6] 2 illustrates another classifier architecture for speech recognition. [Figure 7] Shows face tracking information from a camera on a head mounted display (HMD) or head mounted camera (HMC). [Figure 8]An example of the tracking logic is shown in the example flow chart format. [Figure 9] An example of training logic is shown in the example flow chart format. [Figure 10] An example of HMC is shown below. [Figure 11] FIG. 10 shows facial features imaged by HCM. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure generally relates to computer ecosystems including aspects of standalone computer simulation systems as well as consumer electronics (CE) device networks, such as, but not limited to, computer simulation networks, such as computer game networks. The systems herein may include a server component and a client component connected via a network such that data may be exchanged between the client component and the server component. The client components may include one or more computing devices including gaming consoles such as Sony PlayStation® or gaming consoles made by Microsoft® or Nintendo® or other manufacturers, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart televisions, Internet-enabled televisions), portable computers such as laptops and tablet computers, and smartphones and other mobile devices, including additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may use, by way of example, the Linux® operating system, Microsoft® operating system, or Unix® operating system, or operating systems made by Apple Computer® or Google®. These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft® or Google® or Mozilla®, or other browser programs that can access web sites hosted by the Internet servers described below. An operating environment according to the present principles may also be used to run one or more computer game programs.

[0016] The server and / or gateway may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the clients and servers may be connected via a local intranet or a virtual private network. The server or controller may be instantiated by a gaming console such as a Sony PlayStation®, a personal computer, or the like.

[0017] Information may be exchanged between the clients and the servers over a network. For this purpose and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus for implementing a method for providing a secure community, such as an online social website, for network members.

[0018] As used herein, instructions refer to computer-implemented steps for processing information in a system. Instructions may be implemented in software, firmware, or hardware and may include any type of programmed steps performed by components of the system.

[0019] The processor may be any conventional general purpose single-chip or multi-chip processor capable of implementing logic via various lines, such as address, data and control lines, as well as registers and shift registers.

[0020] The software modules described herein by flowcharts and user interfaces may include various subroutines, procedures, etc. Without limiting the disclosure, logic defined to be performed by a particular module may be redistributed among other software modules and / or aggregated together in a single module and / or made available in a shareable library.

[0021] The principles described herein may be implemented as hardware, software, firmware, or any combination thereof. Accordingly, the illustrative components, blocks, modules, circuits, and steps have been described in terms of their functionality.

[0022] Further to the above allusions, the logic blocks, modules, and circuits described below may be implemented or performed by a general purpose processor, a digital signal processor (DSP), other programmable logic devices such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may be implemented by a controller or state machine, or combination of computing devices.

[0023] The functions and methods described below, when implemented in software, can be written in a suitable language, such as, but not limited to, Java, C#, or C++, and can be stored in or transmitted through a computer-readable storage medium, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), or other optical disk storage, such as a digital versatile disk (DVD), a magnetic disk storage or other magnetic storage device, including a removable thumb drive, etc. A connection may establish a computer-readable medium. Such connections may include wired cables, including, by way of example, optical fiber, coaxial wire, digital subscriber line (DSL), and twisted pair wire. Such connections may also include wireless communication connections, including infrared and radio.

[0024] Components included in one embodiment may be used in other embodiments in any suitable combination, for example, any of the various components described herein and / or illustrated in the figures may be combined, interchanged, or excluded from other embodiments.

[0025] "A system having at least one of A, B, and C" (also "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.

[0026] Referring now specifically to FIG. 1, an exemplary system 10 is shown, which may include one or more of the exemplary devices described above and detailed below in accordance with the present principles. A first of the exemplary devices included in the system 10 is a consumer electronics (CE) device, such as an audio-video device (AVD) 12, such as, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). However, the AVD 12 may instead be an appliance or household item, such as a computer-controlled Internet-enabled refrigerator, washer, or dryer. Alternatively, the AVD 12 may also be a computer-controlled Internet-enabled ("smart") phone, a tablet computer, a notebook computer, an augmented reality (AR) headset, a virtual reality (VR) headset, Internet-enabled glasses or smart glasses, another type of wearable computer-controlled device, such as, for example, a computer-controlled Internet-enabled watch, a computer-controlled Internet-enabled bracelet, other computer-controlled Internet-enabled devices, a computer-controlled Internet-enabled music player, a computer-controlled Internet-enabled headphones, a computer-controlled Internet-enabled implantable device, such as an implantable skin device, and the like. In any event, it should be understood that AVD12 is configured to implement the present principles (e.g., to communicate with other CE devices to implement the present principles, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0027] Thus, to implement such principles, the AVD 12 may be established by some or all of the components shown in FIG. 1. For example, the AVD 12 may include one or more displays 14, which may be implemented by a high-resolution flat screen or an ultra-high-resolution flat screen of "4K" or higher, and may be touch-enabled for receiving user input signals via touch on the display. The AVD 12 may include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as, for example, an audio receiver / microphone, for inputting audible commands to the AVD 12 to control the AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. A graphics processor 24A may also be included. The interface 20 may thus be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It will be appreciated that processor 24 controls AVD 12 to implement the present principles, including, for example, controlling display 14 to present images, receiving input therefrom, and other elements of AVD 12 as described herein. It is further noted that network interface 20 may be, for example, a wired or wireless modem or router, or other suitable interface, such as, for example, a wireless telephony transceiver or a Wi-Fi transceiver as discussed above.

[0028] In addition to the above, ADV 12 may also include one or more input ports 26, such as, for example, a high-definition multimedia interface (HDMI) port or a USB port for physically connecting (e.g., using a wired connection) to another CE device, and / or a headphone port for connecting headphones to ADV 12 to provide audio from ADV 12 to the user through the headphones. For example, input port 26 may be connected, via wire or wirelessly, to a cable or satellite source 26a of audio-video content. Thus, source 26a may be, for example, a separate or integrated set-top box, or a satellite receiver. Alternatively, source 26a may be a game console or disc player that contains content that may be considered favorites by the user for channel allocation purposes, as described further below. When implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 44.

[0029] AVD 12 may further include one or more computer memories 28, such as disk-based or solid-state storage devices, that are not transitory and possibly embodied in the AVD's chassis as a stand-alone device, or as a personal video recording device (PVR) or video disk player for playing AV programs either inside or outside the AVD's chassis, or as a removable memory medium. Also, in some embodiments, AVD 12 may include a position or location receiver, such as, but not limited to, a cellular telephone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from at least one satellite or cellular telephone tower and provide that information to processor 24, and / or to determine the altitude at which AVD 12 is located in conjunction with processor 24. However, it should be understood that another suitable position receiver other than a cellular telephone receiver, a GPS receiver, and / or an altimeter may be used in accordance with the present principles, for example, to determine the location of AVD 12, e.g., in all three dimensions.

[0030] Continuing with the description of the AVD 12, in one embodiment, the AVD 12 may include one or more cameras 32, which may be, for example, a digital camera such as a thermal imaging camera, a webcam, and / or a camera integrated into the AVD 12 and controllable by the processor 24 to collect pictures / images and / or videos in accordance with the present principles. The AVD 12 may also include a Bluetooth transceiver 34 and other Near Field Communication (NFC) elements 36 for communication with other devices using Bluetooth and / or Near Field Communication (NFC) technologies, respectively. An exemplary NFC element may be a Radio Frequency Identification (RFID) element.

[0031] Furthermore, the AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., sensors for detecting gesture commands), etc.) that provide input to the processor 24. The AVD 12 may include an over-the-air (OTA) TV broadcast port 38 for receiving over-the-air (OTA) TV broadcasts that provide input to the processor 24. In addition to the above, it is noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an Infrared Data Association (IRDA) device. A battery (not shown) may be provided for powering the AVD 12.

[0032] 1, in addition to the AVD 12, the system 10 may include one or more other CE device types. In one embodiment, a first CE device 44 may be used to transmit audio and video of a computer game to the AVD 12 via commands sent directly to the AVD 12 and / or through a server, as described below, and a second CE device 46 may include similar components as the first CE device 44. In the illustrated example, the second CE device 46 may be configured as an AR headset worn by a player 47 as shown. In the illustrated example, only two CE devices 44, 46 are shown, but it will be understood that fewer or more devices may be used.

[0033] In the illustrated example, for purposes of illustrating the present principles, it is assumed that all three devices 12, 44, 46 are members of, for example, a home entertainment network, or at least are in close proximity to one another in a location such as a residence, but the present principles are not limited to the particular location indicated by dashed line 48 unless expressly asserted otherwise.

[0034] An exemplary, non-limiting first CE device 44 may be established by any one of the devices mentioned above, e.g., a portable wireless laptop or notebook computer, or a game controller, and therefore may have one or more of the components described below. The first CE device 44 may be, for example, a remote control (RC) for issuing AV play and pause commands to the AVD 12, or may be a more capable device, such as a tablet computer, a game controller, a personal computer, a wireless phone, etc., that communicates with the AVD 12 and / or a game console via a wired or wireless link. The second CE device 46 may be embodied by a head-mounted display (HMD) or a head-mounted camera (HMC).

[0035] Thus, the first CE device 44 may include one or more displays 50, which may be touch-enabled for receiving user input signals via touch on the displays 50. The first CE device 44 may include one or more speakers 52 for outputting audio in accordance with the present principles, and at least one additional input device 54, such as, for example, an audio receiver / microphone, for inputting audible commands to the first CE device 44 to control the device 44. The exemplary first CE device 44 may also include one or more network interfaces 56 for communicating over the network 22 under the control of one or more CE device processors 58. A graphics processor 58A may also be included. Thus, the interface 56 may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including a mesh network interface. It will be understood that the processor 58 controls the first CE device 44 to implement the present principles, including, for example, controlling the display 50 to present images and receiving input therefrom, as well as other elements of the first CE device 44 described herein. Further, it should be noted that network interface 56 may be, for example, a wired or wireless modem or router, or any other suitable interface, such as, for example, a wireless telephony transceiver or a Wi-Fi transceiver as described above.

[0036] In addition to the above, the first CE device 44 may also include one or more input ports 60, such as, for example, an HDMI port or a USB port for physically connecting to another CE device (e.g., using a wired connection) and / or a headphone port for connecting headphones to the first CE device 44 to provide audio from the first CE device 44 to a user through the headphones. The first CE device 44 may further include one or more tangible computer-readable storage media 62, such as disk-based storage or solid-state storage. Also, in some embodiments, the first CE device 44 may include a position or location receiver, such as, but not limited to, a cellular and / or GPS receiver and / or an altimeter 64, configured to receive geographic location information from at least one satellite and / or cellular tower using, for example, triangulation, and provide the information to the CE device processor 58, and / or to determine an altitude at which the first CE device 44 is located in conjunction with the CE device processor 58. However, it should be understood that another suitable position receiver other than a mobile phone and / or a GPS receiver and / or an altimeter can be used in accordance with the present principles to, for example, determine the position of the first CE device 44, for example in all three dimensions.

[0037] Continuing with the description of the first CE device 44, in one embodiment the first CE device 44 may include one or more cameras 66, which may be, for example, a digital camera such as a thermal imaging camera, a webcam, and / or a camera integrated into the first CE device 44 and controllable by the CE device processor 58 to collect pictures / images and / or videos in accordance with the present principles. The first CE device 44 may also include a Bluetooth transceiver 68 and other NFC elements 70 for communication with other devices using Bluetooth and / or Near Field Communication (NFC) technologies, respectively. An exemplary NFC element may be a Radio Frequency Identification (RFID) element.

[0038] Furthermore, the first CE device 44 may include one or more auxiliary sensors 72 (e.g., motion sensors such as an accelerometer, gyroscope, cyclometer, or magnetic sensor, infrared (IR) sensor, optical sensor, speed sensor and / or cadence sensor, gesture sensor (e.g., a sensor for detecting gesture commands), etc.) that provide input to the CE device processor 58. The first CE device 44 may include still other sensors, such as, for example, one or more climate sensors 74 (e.g., a barometer, humidity sensor, wind sensor, light sensor, temperature sensor, etc.) and / or one or more biometric sensors 76 that provide input to the CE device processor 58. In addition to the above, it is noted that in some embodiments the first CE device 44 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 78, such as an Infrared Data Association (IRDA) device. A battery (not shown) may be included to power the first CE device 44. The CE device 44 may communicate with the AVD 12 via any of the communication modes and associated components described above.

[0039] The second CE device 46 may include some or all of the components illustrated for CE device 44. One or both CE devices may be powered by one or more batteries.

[0040] Referring now to the at least one server 80 described above, the server 80 includes at least one server processor 82, at least one tangible computer-readable storage medium 84, such as a disk-based or solid-state storage device, and at least one network interface 86 that, under the control of the server processor 82, enables communication with other devices of FIG. 1 over the network 22, and may, in fact, facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 86 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephony transceiver.

[0041] Thus, in some embodiments, server 80 may be an entire Internet server or server "farm," and devices of system 10 may include and execute "cloud" functionality such that the devices may access a "cloud" environment via server 80, such as in an exemplary embodiment for a network gaming application. Alternatively, server 80 may be implemented by one or more gaming consoles or other computers in the same room or nearby as the other devices shown in FIG.

[0042] The methods herein may be implemented as software instructions executed by a processor, a suitably configured application specific integrated circuit (ASIC) or field programmable gate array (FPGA) module, or any other convenient manner as would be recognized by a person skilled in the art. When used, the software instructions may be embodied in a non-transitory device such as a CD ROM or flash drive. Alternatively, the software code instructions may be embodied in a transitory arrangement such as a radio or optical signal, or by downloading over the Internet.

[0043] 2 shows visual input 200, for example from an RGB camera, used for mouth tracking 202 from which visual features 204 including lips 206 and intra-mouth region 208 are extracted. The features 204 are augmented by keypoints from an event signal 210 from a dynamic vision sensor (DVS) 212 imaging the same person imaged by the RGB camera, implemented according to the principles of event-driven sensors (EDS). The event signal 210 can be used to generate an event-based optical flow 214 that can be used for a low-latency output 216 to augment output 218 that can be used for example for virtual animation of the person imaged in 200.

[0044] No. 7,728,269, and the “Dynamic Vision Platform” monochrome intensity and DVS sensor combination camera disclosed at https: / / inivation.com / dvp from iniVation AG of Zurich, Switzerland, are both incorporated herein by reference and may be referenced when implementing such sensors.

[0045] An EDS consistent with the present disclosure provides an output indicative of a change in light intensity sensed by at least one pixel of a light-sensing array. For example, if the light sensed by a pixel is decreasing, the output of the EDS may be -1, and if the light is increasing, the output of the EDS may be +1. No change in light intensity below a certain threshold may be indicated by an output binary signal of 0.

[0046] 3 further illustrates a virtual reality (VR) and / or augmented reality (AR) head mounted display (HMD) 300 that may be worn by a person 302 and may incorporate any of the components shown in FIG. 1 and described herein. In the illustrated example, the HMD 300 may include a display 304, such as a partially transparent AR display or an opaque VR display. The HMD 300 may also include left and right eye DVS cameras 306 and left and right eye RGB cameras 308 for imaging the person's 302 eyes 310 including pupils 312, as well as imaging the person's eyebrows 314 and nose 316.

[0047] Additionally, the HMD 300 may include a DVS 318 and an RGB camera 320 for mouth imaging oriented to image the mouth 322 of the person 302, including the tongue 324 and teeth 326. The HMD 300 may also include at least one microphone 328 to detect speech of the person 302.

[0048] As shown by representation 330, the above sensors output signals that can be time-stamped to output a representation 330 of the face of person 302. Display luminance information can be used to normalize the size of the imaged pupil 312.

[0049] FIG. 4 shows an exemplary implementation using the sensors shown in FIG. 3. In the illustrated example, the audio sensor 302, the RGB sensor 304, and the DVS 306 send signals to a processor, such as may be implemented by an artificial intelligence (AI) chip 400 that executes algorithms such as one or more neural networks (NNs) 402. The outputs of the sensors 306 / 308 / 318 / 320 are fused together and processed by the NN 402, and output labels representing, for example, emotions or speech are sent to a computer game or other application 404. It should be noted that the audio signal from the audio sensor 328 may be first processed by an audio digital signal processor (DSP) before input to the NN 402, and similarly, the output from the DVS 306 may be appropriately processed before input to the NN 402. Non-machine learning algorithms may also be executed.

[0050] In the illustrated example, the camera, EDS, and processor 400 are all implemented on a single chip 406, which may include local memory for storing images including the images. Processing of the components may be performed by a single digital signal processor (DSP). In either case, the processor 400 outputs the labels to one or more external applications 404, such as a VR object generation algorithm.

[0051] Figure 5 shows further details consistent with Figures 1-4. In Figure 5, audio input 500 from a microphone(s) as described herein is sent to a short-time Fourier transform (STFT) 502 for conversion to the frequency domain. The STFT 502 outputs signals to one or more audio processing convolutional neural networks (CNNs) 504.

[0052] Visual input 506 from the RGB camera(s) described herein is sent to an image recognition engine 508 to extract facial features, such as any of the features described above. The output of engine 508 is sent to one or more visual processing CNNs 510.

[0053] The EDS input from a DVS 512, such as any of the DVS(es) described herein, is sent to a frame generator 514 to generate low latency, high data rate virtual frames that are sent to an image filter 516 for filtering the virtual frames. The output 517 of the filter 516 is sent to one or more event processing CNNs 518.

[0054] As shown in Figure 5, the outputs of the CNNs 504, 510, 518 are fed into a fully connected layer 520 which, together with the CNNs, builds a classifier. The output of the classifier can be used to detect the emotion of an imaged person (e.g., person 302 in Figure 3), recognize the person, such as by speaker recognition.

[0055] The "fully connected layer" 520 is the part of the network where every neuron connects to every neuron in the next layer. The classifier includes a CNN for feature vector extraction. The "fully connected layer" comes at the very end to give the output label. In essence, the fully connected input layer takes the outputs of the CNN and "flattens" them into a single vector, which is input to the next stage. The fully connected layer applies weights to the features to predict the label and assigns a probability to each label.

[0056] 6 shows an alternative architecture that uses the components described in FIG. 5 through CNNs 504, 510, and 518 to track a target person's mouth 600, except that a recurrent neural network 602, which may include one or more long short-term memories (LSTMs) 604, receives the output of the CNNs and then outputs signals to a fully connected layer 520. The RNN 602 provides the encoding of temporal information.

[0057] The architecture in FIG. 6 fuses raw event data from the DVS(es) with RGB camera and audio data as input to a classifier to eliminate the effects of motion blur, inner-mouth areas that are difficult to image with RGB cameras, different lighting conditions, hard-to-image teeth, and fast mouth movements.

[0058] It should be noted that the architecture of Figure 5 is sufficient to recognize emotions by processing 1 / 2 frame images with short duration audio, making it particularly useful, but not exclusive, for human emotion recognition. On the other hand, Figure 6 adds an RNN 602 to provide temporary memory for the classifier, which is useful for audiovisual speech recognition. Thus, Figure 6 can provide the same functionality as Figure 5, but with the addition of an RNN for temporal information.

[0059] FIG. 7 provides further illustration of the embodiment of the CNN layer of FIG. 5 and FIG. 6 with respect to imaging the eye region. An eye image 700 is generated by either or both of the HMD 300 shown in FIG. 3 or the HMC described elsewhere herein. From the image, eyelid and eyebrow features 702 are extracted, as well as pupil segmentation 704 (normalized to display brightness) and gaze direction 706. Using DVS allows detection of otherwise difficult to detect structures in the mouth, and promotes high speed and high dynamic range for detecting eye movements, lip / tongue / teeth movements, etc., which may be important attributes in estimating human emotions. DVS images facilitate segmenting eye images to detect eye vibrations and mouth images to detect lips / tongue / teeth and their movements.

[0060] 8 illustrates an example of the overall logic in example flow chart form. At block 800, audio is received from the person being imaged. At block 802, an RGB image is received from any of the cameras described herein, and at block 804, event information is received from any of the DVSs described herein. All three are input at block 806 to any of the classifiers described herein, which at block 808 output information useful for detecting emotions, mouth movements, eye movements, etc., for rendering a virtual image of the person, or for other purposes including adjusting the context to the person's emotions.

[0061] FIG. 9 shows that the classifier herein may receive ground truth training set(s) of audio, RGB image, and event signals at block 900. The ground truth training set may include audio / visual clips and EDS data combined with labeled emotion output or spoken text output. Corresponding ground truth mouth tracking, emotion classification, eye tracking, etc. are also provided at block 902. Using the training set, the classifier learns the correct output from real data. The classifier can be trained using all three inputs (from audio / camera / event data), with the classifier including RNN and CNN in an exemplary embodiment.

[0062] 10 and 11 show that the HMC 1000 may be worn by a person 1002 with an inward-facing imaging assembly 1004 pointed toward the face of the person 1002 and spaced in front of the person, whose eyes are otherwise unobstructed from viewing the real world. The assembly 1004 generates RGB and DVS signals representing various parts of the face of the person 1002, including the mouth 1006 with tongue 1008 and teeth 1010, nose 1012, eyebrows 1014, and eyes 1016 with pupils 1018.

[0063] 11, the imaging assembly 1004 may include one or more DVS imagers 1100 consistent with the principles described herein, one or more RGB cameras 1102, and one or more microphones 1104. Signals from the various sensors in the imaging assembly 1004 are time-stamped so that they can be correlated together to generate an output 1106 representative of the face (and voice) of the person 1002.

[0064] While the present principles have been described with reference to certain illustrative embodiments, it will be understood that these are not intended to be limiting and that a variety of alternative configurations may be used to implement the subject matter claimed herein.

Claims

1. at least one camera unit configured to output a red-green-blue (RGB) image of a face; at least one event-driven sensor (EDS) configured to output a signal representative of a change in illuminance of the face, the EDS providing an output of −1 when a sensed light intensity is decreasing, providing an output of +1 when a sensed light intensity is increasing, and providing an output of 0 when there is no change in the light intensity; and at least one microphone configured to output a signal representative of speech; At least one processor, receiving signals from the camera unit, the EDS, and the microphone; Executing at least one neural network to generate at least one of a human emotion prediction, a tracking of at least a portion of the face based on the signals from the camera unit, the EDS, and the microphone; the at least one processor configured with executable instructions to perform An assembly comprising:

2. The assembly of claim 1 , wherein the camera unit is configured to output an infrared (IR) image of the face.

3. The assembly of claim 1 , wherein the camera unit, the processor, and the EDS are located on a single chip.

4. The instruction:

2. The assembly of claim 1, wherein the at least one neural network is executable to execute the at least one neural network to generate a human emotion prediction based on the signals from the camera unit, the EDS, and the microphone.

5. The instruction:

2. The assembly of claim 1, wherein the at least one neural network is executable to execute the at least one neural network to generate a tracking of at least a portion of the face based on the signals from the camera unit, the EDS, and the microphone.

6. The assembly of claim 5 , wherein the portion includes a pupil of at least one eye.

7. The assembly of claim 5 , wherein the portion includes a corner of the mouth.

8. The assembly of claim 5 , wherein the portion includes an interior of the mouth that includes the teeth.

9. at least one camera unit configured to output an image of a person; At least one microphone; at least one event-driven sensor (EDS) configured to output a signal representative of the person, the EDS providing an output of −1 when a sensed light intensity is decreasing, providing an output of +1 when a sensed light intensity is increasing, and providing an output of 0 when there is no change in the light intensity; and At least one processor, processing the output of said microphone using a Short-Time Fourier Transform (STFT); processing an output of the STFT using at least one audio processing convolutional neural network (CNN); processing at least features of images from said camera unit using at least one visual processing CNN; processing output signals from said EDS using at least one event-processing CNN; Fusing the outputs of the CNNs in a fully connected neural network layer, A prediction of the person's emotions; and tracking at least a portion of the person's face; at least one virtual reality (VR) image of the person; Identification of the person; and and generating one or more of: the at least one processor programmed with instructions to perform A system comprising:

10. The processor, processing the output of the CNN using a recurrent neural network (RNN); processing an output of the RNN using the fully connected neural network layer to generate a track of the person's mouth; 10. The system of claim 9, comprising instructions for:

11. The processor, 10. The system of claim 9, comprising instructions for generating a prediction of the person's emotion.

12. The processor, The system of claim 9 , comprising instructions for generating a tracking of at least a portion of the face of the person.

13. The processor, 10. The system of claim 9, further comprising instructions for generating at least one virtual reality (VR) image of the person.

14. The processor, The system of claim 9 comprising instructions for generating an identification of the person.

15. receiving a signal from at least one camera unit; receiving a signal from at least one event-driven sensor (EDS); Receiving a signal from at least one microphone; Executing at least one neural network to generate at least one of an emotion prediction of the person, tracking at least a portion of a face, identifying the person, and a virtual reality (VR) image of the person based on the signals from the camera unit, the EDS, and the microphone; presenting at least one of: emotion prediction of the person; tracking at least a portion of the face; identifying the person; and a VR image of the person on at least one display; Including, wherein the EDS provides an output of -1 when the sensed light intensity is decreasing, an output of +1 when the sensed light intensity is increasing, and an output of 0 when there is no change in the light intensity.

16. 16. The method of claim 15, comprising implementing at least one neural network to generate an emotion prediction of the person based on the signals from the camera unit, the EDS, and the microphone.

17. The method of claim 15 , comprising implementing at least one neural network to generate a tracking of at least a portion of the face based on the signals from the camera unit, the EDS, and the microphone.

18. The method of claim 15 , comprising implementing at least one neural network to generate an identification of the person based on the signals from the camera unit, the EDS, and the microphone.

19. 16. The method of claim 15, comprising executing at least one neural network to generate a virtual reality (VR) image of the person based on the signals from the camera unit, the EDS, and the microphone.

20. The method of claim 17 , wherein the portion of the face is the corners of the mouth and the inside of the mouth.

Images