Systems and methods for domain adaptation in neural networks using cross-domain batch normalization

Abstract

The domain adaptation module (1800) is used to optimize a first domain (1804) derived from a second domain (1802) using respective outputs of respective parallel hidden layers from the domains.

Description

Technical Field

[0001] The present application generally relates to technically inventive and unconventional solutions that are necessarily rooted in computer technology and result in specific technical improvements. Background Art

[0002] Machine learning (sometimes referred to as deep learning) can be used for a variety of useful applications related to data understanding, detection, and / or classification, including image classification, optical character recognition (OCR), object recognition, action recognition, speech recognition, and emotion recognition. However, as understood herein, a machine learning system may not be sufficient to recognize actions in one domain (such as computer games) using a training dataset from another domain (e.g., movie videos).

[0003] For example, in the computer game industry, video and audio are two separate processes. First, the game is designed and produced without audio, and then the audio team investigates the entire game video and inserts the corresponding SFX from a database of sound effects (SFX), which is extremely time-consuming. As we'll see in this article, machine learning can be used to accelerate this process, but current action recognition models are trained on real-world video datasets, making them susceptible to dataset shift or bias when used on game videos. Summary of the Invention

[0004] To overcome the domain mismatch problem, at least two common domains of training data (images, videos, or audio) are used to classify the target dataset. A pair of training data domains can be established, for example, using real-world video and computer game video, first and second speaker speech (for speech recognition), standard font text and cursive script (for handwriting recognition), and so on.

[0005] Therefore, a universal domain adaptation module established by the loss function and / or the actual neural network receives input from multiple output points from the two training domains of the deep learning and provides an output metric that allows optimization of one and possibly both of the two trajectories of the neural network. A universal cross-domain feature normalization module can also be used and inserted into any layer of the neural network.

[0006] Thus, in one aspect, an apparatus includes at least one processor and at least one computer storage device, the computer storage device being non-transient and comprising instructions executable by the at least one processor. The instructions are executable to: access a first neural network associated with a first data type; access a second neural network associated with a second data type different from the first data type; provide first training data as input to the first neural network; and provide second training data as input to the second neural network. The first training data is different from the second training data. The instructions are also executable to: identify a first output from an intermediate layer of the first neural network, wherein the first output is based on the first training data; and identify a second output from an intermediate layer of the second neural network, wherein the second output is based on the second training data. The respective intermediate layers of the first and second neural networks are parallel layers. The instructions are also executable to: identify a ratio for normalizing the first and second outputs; and apply an equation that takes the ratio into account to change one or more weights of the intermediate layer of the second neural network.

[0007] In some examples, the ratio can be related to the mean. In addition, both the mean and the variance between the first output and the second output can be analyzed to apply the equation. The ratio can be identified and the equation can be applied using Cross-Domain Batch Normalization (CDBN).

[0008] In some implementations, before providing the second training data to the second neural network, the second neural network can be established by a copy of the first neural network. In addition, the intermediate layers of the first neural network and the second neural network can be layers other than the output layer.

[0009] Furthermore, in some implementations, the first training data can be correlated with the second training data. Thus, for example, where the first neural network and the second neural network are related to action recognition, the first training data can be correlated with the second training data because both the first training data and the second training data can be related to the same action. As another example, where the first neural network and the second neural network are related to object recognition, the first training data can be correlated with the second training data because both the first training data and the second training data can be related to the same object.

[0010] In another aspect, a method includes accessing a first neural network associated with a first data type; accessing a second neural network associated with a second data type different from the first data type; providing first training data as input to the first neural network; and providing second training data as input to the second neural network. The first training data is different from the second training data. The method further includes identifying a first output from a hidden layer of the first neural network, wherein the first output is based on the first training data; and identifying a second output from a hidden layer of the second neural network, wherein the second output is based on the second training data. The respective hidden layers of the first neural network and the second neural network are parallel layers. The method then includes identifying a ratio for normalizing the first output and the second output; and applying the ratio to an output from the hidden layer of the second neural network to normalize the output from the hidden layer of the second neural network.

[0011] In yet another aspect, an apparatus includes at least one computer storage device that is not a transient signal and includes instructions executable by at least one processor. The instructions are executable to: access a first domain of training data associated with a first domain class; access a second domain of training data associated with a second domain class different from the first domain class; and classify a target dataset using the training data from the first domain and the second domain. The instructions are also executable to: output a classification of the target dataset, wherein the target dataset is classified by a domain adaptation module including a cross-domain batch normalization (CDBN) module to adaptively select domain statistics to normalize the input.

[0012] The details of the present application, both as to its structure and operation, may be best understood with reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example system consistent with the principles of the present invention;

[0014] Figure 2 、 Figure 3 、 Figure 5 、 Figure 7 、 Figure 9 、 Figure 10 、 Figure 14 and Figure 16 is a flow chart of example logic consistent with the principles of the present invention;

[0015] Figure 4 、 Figure 6 、 Figure 8 、 Figure 11 、 Figure 13 、 Figure 15 and Figure 18shows examples of various domain adaptation architectures according to the principles of the present invention; and

[0016] Figure 12 and Figure 17 is an example table illustrating the principles of the present invention. DETAILED DESCRIPTION

[0017] According to the principles of the present invention, a deep learning-based domain adaptation method can be used to overcome the domain mismatch problem for tasks related to images, videos, or audio, such as understanding / detection / classification given any source and target domain data. At least three general types of data (images, videos, or audio) can be used, and all types of neural network modules can be used to improve system performance.

[0018] As described herein, two tracks of a deep learning processing flow can be used for any particular input-to-output task. One track can be used for one data domain, while the other track can be used for another data domain, such that there can be at least two tracks of deep learning for two data domains. For example, a pair of domains could be two types of videos, such as real-world video and video game world video, speech from one speaker and speech from another speaker, standard font text and cursive script, speech recognition domains, text-to-speech, and speech-to-text.

[0019] The universal domain adaptation module is described below, sometimes using a loss function. The universal domain adaptation module can also use actual neural network connections, which take inputs from multiple output points from two deep learning tracks and provide output metrics, allowing optimization of both tracks of the neural network. The universal domain adaptation module can also use a universal cross-domain feature normalization module, which can be inserted into any layer of the neural network.

[0020] Therefore, the method described herein may involve multiple objects and multiple actions associated with the multiple objects. For example, an image text block of a plurality of texts may be an "object", and the type of the image block may be an "action".

[0021] The present disclosure also generally relates to computer ecosystems that include aspects of consumer electronics (CE) device networks, such as, but not limited to, distributed computer gaming networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, and artificial neural network and machine learning applications.

[0022] The system herein may include server and client components connected via a network so that data can be exchanged between the client and server components. The client component may include one or more computing devices, including AR headsets, VR headsets, game consoles (such as Sony ) and related motherboards, game controllers, portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers (such as laptop computers and tablet computers), and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some of the client computers may use, for example, Orbis or Linux operating systems, operating systems from Microsoft, or Unix operating systems, or operating systems produced by Apple or Google. These operating environments can be used to execute one or more programs / applications, such as browsers produced by Microsoft or Google or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. In addition, an operating environment according to the principles of the present invention can be used to execute one or more computer game programs / applications and other programs / applications that implement the principles of the present invention.

[0023] The server and / or gateway may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Additionally or alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be provided by a game console and / or one or more motherboards thereof (such as a Sony ), personal computers, etc.

[0024] Information can be exchanged between the client and the server over the network. To this end and for security purposes, the server and / or the client may include firewalls, load balancers, temporary storage devices and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements a method according to the principles of the present invention to provide network users, such as online social networking sites or video game sites, with a secure community for crowdsourced communication.

[0025] As used herein, instructions refer to computer-implemented steps for processing information in a system. Instructions can be implemented in software, firmware, or hardware, and include any type of programmed steps performed by components of the system.

[0026] The processor may be any conventional general purpose single chip processor or multi-chip processor capable of executing logic with the aid of various lines such as address, data and control lines, as well as registers and shift registers.

[0027] The software modules described by the flowcharts and user interfaces herein may include various subroutines, programs, etc. Without limiting the present disclosure, the logic recited as being performed by a particular module may be redistributed to other software modules and / or combined together in a single module and / or made available in a shareable library.

[0028] As indicated above, the principles of the present invention described herein may be implemented as hardware, software, firmware, or a combination thereof; accordingly, the illustrative components, blocks, modules, circuits, and steps are described in terms of their functionality.

[0029] In addition to the above, the logic blocks, modules, and circuits described below may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices designed to perform the functions described herein (such as an application-specific integrated circuit (ASIC), discrete gate or transistor logic, discrete hardware components, or any combination thereof). The processor may be implemented by a controller, a state machine, or a combination of computing devices.

[0030] Function and method described below can be realized with hardware circuit or software circuit.When being realized with software, function and method can be written with suitable language such as but not limited to Java, C# or C++, and can be stored on computer readable storage medium or transmitted by computer readable storage medium, described computer readable storage medium such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage devices (such as digital versatile disc (DVD)), magnetic disk storage device or other magnetic storage devices including removable thumb drive etc. are connected. Computer readable medium can be established. Such connection can include, for example, hard-wired cable, including optical fiber and coaxial line and digital subscriber line (DSL) and twisted pair. Such connection can include wireless communication connection, including infrared and radio.

[0031] The components included in one embodiment may be used in any appropriate combination in other embodiments. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0032] "A system having at least one of A, B, and C" (similarly, "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes the following systems: having only A; having only B; having only C; having A and B at the same time; having A and C at the same time; having B and C at the same time; and / or having A, B, and C at the same time, etc.

[0033] Now specifically refer to Figure 1 , an example system 10 is shown, which may include one or more of the example devices mentioned above and further described below in accordance with the principles of the present invention. The first of the example devices included in the system 10 is a consumer electronics (CE) device such as an audio video device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). However, the AVD 12 may alternatively be an appliance or household item, for example, a computerized internet-enabled refrigerator, washer, or dryer. Alternatively, the AVD 12 may also be a computerized internet-enabled ("smart") phone, a tablet computer, a laptop computer, an augmented reality (AR) headset, a virtual reality (VR) headset, internet-enabled or "smart" glasses, another type of wearable computerized device, such as a computerized internet-enabled watch, a computerized internet-enabled bracelet, a computerized internet-enabled music player, a computerized internet-enabled headset, a computerized internet-enabled implantable device (such as an implantable skin device), other computerized internet-enabled devices, etc. Regardless, it should be understood that the AVD 12 is configured to implement the present principles (e.g., communicate with other consumer electronics (CE) devices to implement the present principles, execute the logic described herein, and perform any other functions and / or operations described herein).

[0034] Therefore, in order to implement such a principle, the AVD 12 may be Figure 1 Some or all of the components shown may be established. For example, the AVD 12 may include one or more displays 14, which may be implemented as flat screens with high or ultra-high definition ("4K") or higher resolution and may be touch-enabled for receiving user input signals via touch on the display. The AVD 12 may include one or more speakers 16 for outputting audio in accordance with the principles of the present invention, and at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to the AVD 12 to control the AVD 12. The example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, a WAN, a LAN, etc.) under the control of one or more processors. Thus, the interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. Furthermore, it should be noted that the network interface 20 may be, for example, a wired or wireless modem or router, or other suitable interface (such as, for example, a wireless telephone transceiver or the Wi-Fi transceiver mentioned above).

[0035] It should be understood that one or more processors control AVD 12 to implement the principles of the present invention, including other elements of AVD 12 described herein, such as controlling display 14 to present images on the display and receive input from the display. The one or more processors may include a central processing unit (CPU) 24 and a graphics processing unit (GPU) 25 on a graphics card 25A.

[0036] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as, for example, a High-Definition Multimedia Interface (HDMI) port or a USB port for physically connecting (e.g., using a wired connection) to another consumer electronics (CE) device and / or a headphone port for connecting headphones to the AVD 12 for presenting audio from the AVD 12 to the user via the headphones. For example, the input port 26 may be connected to a cable or satellite source 26a of audio and video content via a wire or wireless connection. Thus, the source 26a may be, for example, a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or disk player containing content that the user may consider a favorite for channel allocation purposes. When implemented as a game console, the source 26a may include some or all of the components described below with respect to the CE device 44 and may implement some or all of the logic described herein.

[0037] The AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices, which in some cases are embodied as separate devices within the AVD's housing, or as a personal video recorder (PVR) or video disk player internal or external to the AVD's housing for playback of AV programs, or as removable storage media. In addition, in some embodiments, the AVD 12 may include a position or location receiver (such as, but not limited to, a cell phone receiver, a GPS receiver, and / or an altimeter 30) configured to receive geographic location information, for example, from at least one satellite or cell phone tower, and provide the information to the processor 24 and / or determine, in conjunction with the processor 24, the altitude at which the AVD 12 is located. However, it should be understood that another suitable position receiver other than a cell phone receiver, a GPS receiver, and / or an altimeter may be used in accordance with the principles of the present invention, for example, to determine the location of the AVD 12 in all three dimensions.

[0038] Continuing with the description of the AVD 12, in some embodiments, in accordance with the principles of the present invention, the AVD 12 may include one or more cameras 32, which may be, for example, thermal imaging cameras, digital cameras such as webcams, infrared (IR) cameras, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to generate pictures / images and / or videos. The AVD 12 may also include a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0039] Furthermore, the AVD 12 may include one or more auxiliary sensors 37 (e.g., a motion sensor such as an accelerometer, gyroscope, gyrometer, or magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and / or cadence sensor, a gesture sensor (e.g., for sensing gesture commands), etc.) that provide input to the processor 24. The AVD 12 may include a wireless TV broadcast port 38 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD 12.

[0040] Still refer to Figure 1 In addition to the AVD 12, the system 10 may also include one or more other consumer electronics (CE) device types. In one example, a first CE device 44 may be used to send computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through a server described below, while a second CE device 46 may include components similar to the first CE device 44. In the example shown, the second CE device 46 may be configured as an AR or VR headset worn by a user 47, as shown. In the example shown, only two CE devices 44, 46 are shown, and it should be understood that fewer or larger devices may also be used in accordance with the principles of the present invention.

[0041] In the example shown, it is assumed that all three devices 12, 44, 46 are elements of a network (such as a secure or encrypted network, an entertainment network, or Wi-Fi) in a home, for example, or at least are present in close proximity to each other at a particular location and are able to communicate with each other and with the server described herein. However, unless expressly required otherwise, the principles of the present invention are not limited to a particular location or network.

[0042] An example, non-limiting, first CE device 44 may be established by any of the above devices (e.g., a smartphone, a digital assistant, a portable wireless laptop or notebook computer, or a game controller (also referred to as a "console")), and thus may have one or more of the components described below. The second CE device 46 may be, but is not limited to, established by an AR headset, a VR headset, "smart" internet-enabled glasses, or even a video disc player (such as a Blu-ray player), a game console, or the like. Further, in some embodiments, the first CE device 44 may be a remote control device (RC) for, for example, issuing AV play and pause commands to the AVD 12, or it may be a more complex device, such as a tablet computer, a computer connected to the AVD 12 via a wired or wireless link. Figure 1 Another of the illustrated devices implements a game console, a game controller, a personal computer, a wireless telephone, or the like that communicates with and controls video games presented on the AVD 12 .

[0043] Thus, the first CE device 44 may include one or more displays 50, which may be touch-enabled for receiving user input signals via touch on the display 50. Additionally or alternatively, the display(s) 50 may be at least partially transparent displays configured for presenting AR and / or VR images, such as AR headset displays or “smart” glasses displays or “heads-up” displays, as well as VR headset displays or other displays.

[0044] The first CE device 44 may also include one or more speakers 52 for outputting audio in accordance with the principles of the present invention, as well as at least one additional input device 54, such as, for example, an audio receiver / microphone, for inputting audible commands to the first CE device 44 to control the device 44. The example first CE device 44 may also include one or more network interfaces 56 for communicating over the network 22 under the control of one or more CE device processors 58. Thus, the interface 56 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including a mesh network interface. It should be understood that the processor 58 controls the first CE device 44 to implement the principles of the present invention, including other elements of the first CE device 44 described herein, such as, for example, controlling the display 50 to present images on the display and receive input from the display. Furthermore, it should be noted that the network interface 56 may be, for example, a wired or wireless modem or router, or other suitable interface (such as a wireless telephone transceiver or, as mentioned above, a Wi-Fi transceiver).

[0045] Furthermore, it should be noted that the first CE device 44 may include a graphics processing unit (GPU) 55 on a graphics card 55A in addition to the processor(s) 58. The graphics processing unit 55 may be configured to, among other things, render AR and / or VR images on the display 50.

[0046] In addition to the foregoing, the first CE device 44 may also include one or more input ports 60 (such as, for example, an HDMI port or a USB port) for physically connecting (e.g., using a wired connection) to another CE device and / or a headphone port for connecting headphones to the first CE device 44 for presenting audio from the first CE device 44 to the user via the headphones. The first CE device 44 may also include one or more tangible computer-readable storage media 62, such as disk-based storage devices or solid-state storage devices. Furthermore, in some embodiments, the first CE device 44 may include a location or position receiver (such as, but not limited to, a cell phone and / or a GPS receiver and / or an altimeter 64) configured to receive geographic location information from at least one satellite and / or cell phone tower using triangulation and provide the information to the CE device processor 58 and / or determine, in conjunction with the CE device processor 58, the altitude at which the first CE device 44 is located. However, it should be understood that another suitable location receiver other than a cell phone and / or a GPS receiver and / or an altimeter may be used, for example, to determine the location of the first CE device 44 in all three dimensions, in accordance with the principles of the present invention.

[0047] Continuing with the description of the first CE device 44, in some embodiments, in accordance with the principles of the present invention, the first CE device 44 may include one or more cameras 66, which may be, for example, a thermal imaging camera, an IR camera, a digital camera such as a webcam, and / or another type of camera integrated into the first CE device 44 and controllable by the CE device processor 58 to generate pictures / images and / or videos. The first CE device 44 may also include a Bluetooth transceiver 68 and other near-field communication (NFC) components 70 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An example NFC component may be a radio frequency identification (RFID) component.

[0048] Furthermore, the first CE device 44 may include one or more auxiliary sensors 72 (e.g., motion sensors such as accelerometers, gyroscopes, gyrometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the CE device processor 58. The first CE device 44 may include other sensors that provide input to the CE device processor 58, such as, for example, one or more climate sensors 74 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 76. In addition to the foregoing, it should be noted that in some embodiments, the first CE device 44 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 78, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the first CE device 44. The CE device 44 may communicate with the AVD 12 via any of the above-described communication modes and associated components.

[0049] The second CE device 46 may include some or all of the components shown for the CE device 44. Either or both CE devices may be powered by one or more batteries.

[0050] Referring now to the aforementioned at least one server 80, it includes at least one server processor 82, at least one tangible computer readable storage medium 84 (such as a disk-based storage device or a solid-state storage device). In an implementation, the medium 84 includes one or more solid-state storage drives (SSDs). In accordance with the principles of the present invention, the server also includes at least one network interface 86, which allows communication with the server via the network 22. Figure 1 84. The server 80 may be a server or client device that is configured to communicate with other devices of the server and may in fact facilitate communication between the server and the client devices. It should be noted that the network interface 86 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as a wireless telephone transceiver). The network interface 86 may be a remote direct memory access (RDMA) interface that connects the medium 84 directly to a network such as a so-called "fabric" without going through the server processor 82. The network may include an Ethernet network and / or a Fibre Channel network and / or a wireless broadband network. Typically, the server 80 includes multiple processors in multiple computers, referred to as "blades" that may be arranged in a physical server "stack."

[0051] Thus, in some embodiments, the server 80 may be an Internet server or an entire "server farm" and may include and perform "cloud" functionality such that the devices of the system 10 may access the "cloud" environment via the server 80 in, for example, the domain-adapted exemplary embodiments disclosed herein. Additionally or alternatively, the server 80 may be managed by a server associated with Figure 1 The other devices shown are implemented in the same room or nearby one or more game consoles or other computers.

[0052] Before describing additional figures, it should be understood that, in accordance with the principles of the present invention, in order to optimize an artificial intelligence system, an optimized source domain / model of well-trained data can be copied to create a target domain / model, which is further refined for data of a different type than the source domain. For example, the source domain may be used for action recognition in real-world videos, while the target domain may be used for action recognition in video game videos. Due to differences in video type and visual effects, the source domain may not be sufficient to perform action recognition using video game data, but it may still provide a good starting point for adapting a sufficient target domain from the video game data for action recognition.

[0053] Thus, the present principles describe systems and methods for performing domain adaptation and optimization. According to the present disclosure, this can be performed not only by backpropagating from the output / activation layers of the neural network once a human supervisor or system administrator has identified an error, but also by running different but related training data for both the target domain and the source domain and selecting any given hidden or intermediate layers for each domain to be parallel to each other to determine whether the outputs are similar or even identical. If the outputs are statistically dissimilar (as defined by the supervisor or administrator), certain weight adjustments can be performed as described herein for the intermediate target layers to minimize the differences in the outputs from the parallel layers (e.g., to ensure that the abstractions for the parallel layers are similar / identical) and thereby further optimize the target domain for the different types of data. Then, after training, testing can also be performed to ensure that the optimization has proceeded to an acceptable degree.

[0054] In terms of different but related training data, the data may be different because it is data appropriate for a given domain, but related because the training data for each domain may be related to similar concepts or metaphors. For example, the training data fed into the source domain may be real-world videos of humans performing piercings, while the training data fed into the target domain may be video game videos of game characters performing piercings. As another example, this time for object recognition, the training data fed into the source domain may be real-world pictures of apples, while the training data fed into the target domain may be video game videos of digital apples.

[0055] Additionally, as used in the above references, parallel source and target intermediate / hidden layers refer to respective source and target intermediate layers that are initially identical due to the source domain being copied to initially establish the target domain, wherein the layers perform the same task(s) and / or have the same purpose. Thus, for example, intermediate source layer 500 may be parallel to intermediate target layer 500, wherein the target domain is copied from the source domain, both domains have the same number of intermediate layers, and target layer 500 is initially established from source layer 500.

[0056] In view of the foregoing, the principles of the present invention will now be described in more detail. Figure 2 Starting with the logic of FIG, as an example of a modification of a general trajectory of a neural network (NN) for video classification, the baseline architecture for video classification can be modified as follows. Starting at block 200, a modification of a common convolutional neural network (CNN) into a spatial region extraction network (SREN) can be performed so that feature vectors of the entire video scene and important spatial regions (e.g., objects, body parts, etc.) can be extracted. Then, Figure 2 The logic of may proceed to block 202 where the two types of outputs, region features and scene features, may be concatenated into a frame-level feature vector and then input into the video model at block 204 .

[0057] Then, Figure 2 The logic may proceed to block 206 where the frame-level feature vectors may be input into a recurrent neural network (RNN) including long short-term memory (LSTM) units to model temporal dynamic information. The logic may then proceed to block 208 where the final classifier may be modified to classify both (A) the entire scene and (B) all important regions in the video(s).

[0058] Figure 2 The logic can then proceed to block 210 where blocks 200-208 can be repeated for a second domain category to utilize and optimize the entire architecture with data from a different video type / category. Then, at block 212, the frame-level feature vector, the features after the RNN, and the classifier output can be input to the domain adaptation module as input. The domain adaptation module can use one or more of the following three methods, each of which is described in Figure 3 、 Figure 5 and Figure 7 The difference function method ( Figure 3 ), domain classifier method ( Figure 5 ) and cross-domain batch normalization methods ( Figure 7 ).

[0059] From reference Figure 3Starting with the difference function method, it should be understood that the difference function can be used to calculate the distance of the overall data distribution between the source data and the target data. The difference loss can be defined by different metrics from any subset of the layers of the source / target model, such as the probability-based distance between the source data and the target data extracted from multiple layers of the model (as further described below), or by normalizing the parameter differences between the source model and the target model (as further described below), or a weighted sum of these two types of losses (as further described below). By jointly training with the difference function, the model will be optimized to reduce the distribution difference, thereby improving the generalization ability.

[0060] Therefore, from block 212 as described above, Figure 3 You can start at box 300, where you can define and add another loss function (different from the overall loss function used when backpropagating from the output layer), where this additional loss function is a difference loss function that is calculated as the distance between features learned from the source data and the target data output by the corresponding parallel layers.

[0061] In the absence of a discrepancy loss, the overall loss function can be calculated using only labeled source data, so during optimization, the model will gradually fit the source data, which will increase the distribution difference between the two domains. Therefore, an unsupervised domain adaptation protocol can be used to reduce the overall distribution difference between the source data and the target data, where the training data used includes labeled data from the source domain and unlabeled data from the target domain (generally designated as box 302), and where the test data used is all from the target domain (generally designated as box 304).

[0062] exist Figure 3 At block 306, the logic may compute the distance between the features learned from the source and target data output from the corresponding parallel layers without labels. Then, at block 308, joint training with a discrepancy loss function may be used for the model to reduce the difference in the overall distribution between the source and target data. This may be accomplished at block 310 by computing the discrepancy loss using the feature vectors from the output of the temporal module and the last fully connected layer. Figure 4 Shows the combination Figure 3 and an example action recognition architecture for these principles as described.

[0063] Therefore, if Figure 4As shown, an apparatus embodying the principles of the present invention may access a first neural network / domain 400 associated with a first data type, which may be a source neural network / domain, access a second neural network / domain 402 associated with a second data type different from the first data type, which may be a target neural network / domain, and provide first training data as input to the first neural network. The apparatus may also provide second training data as input to the second neural network, where the first training data is different from but still related to the second training data.

[0064] For example, the first neural network / domain 400 may be related to object recognition using real-world videos, while the second neural network / domain 402 may be related to object recognition using video game videos. Thus, the first training data may be a video of a real-world apple from a real-life video recording, while the second training data may be a video of a video game-rendered graphical apple from a video game.

[0065] The apparatus may then identify a first output from a first layer, wherein the first layer is an output / activation layer of the first neural network, and wherein the first output is based on the first training data. The apparatus may also identify a second output from a second layer, wherein the second layer is an output / activation layer of the second neural network, and wherein the second output is based on the second training data. The apparatus may then determine a first adjustment to one or more weights of a third layer based on the first output and the second output, wherein the third layer is an intermediate layer of the second neural network. The first adjustment may be determined, for example, by backpropagation from a second layer of the second neural network (the output / activation layer of the second neural network) using a first difference / loss function.

[0066] Thereafter, a human supervisor may provide a command to manually select, or the device itself may (e.g., randomly) select the third and fourth layers (where the fourth layer is an intermediate layer of the first neural network). The third and fourth layers may be parallel intermediate / hidden layers. Thereafter, a second difference / loss function may be used to measure the third output from the third layer and compared to the fourth output from the fourth layer, the second difference / loss function being customized (e.g., by a human supervisor) to measure the similarity between the third and fourth outputs regardless of whether the object label (e.g., "apple") of the second neural network is available. The third and fourth outputs themselves may be respective vector outputs of the respective third and fourth layers, before being provided to the respective subsequent intermediate layers of the respective second and first neural networks, respectively, where the third and fourth outputs themselves are based on the second training data and the first training data, respectively.

[0067] The device may then determine a second adjustment to one or more weights of the third layer based on the comparison / second function, where the amount of the weight change is proportional to the magnitude of the second function. Thereafter, the device may subsequently adjust one or more weights of the third layer (and even one or all previous layers of the second neural network) based on consideration of both the first adjustment and the second adjustment. For example, one or more weights of the third layer may be adjusted by adding together the corresponding weight changes from the first adjustment and from the second adjustment. However, in some examples, only the weight change from one of the first adjustment or the second adjustment may be applied if determined by a human supervisor or the device to result in less loss than the sum of the weight changes from both the first adjustment and the second adjustment. In other examples, half of the weight change(s) from the first adjustment and half of the weight change(s) from the second adjustment may be added together if determined by a human supervisor or the device to result in less loss than the above alternatives.

[0068] Furthermore, it is noted that before providing the second training data to the second neural network, the second neural network can be established by a copy of the first neural network. The third layer and the fourth layer of the corresponding neural network can be layers other than the output layer, such as middle hidden layers of the corresponding neural network.

[0069] Additionally, the first training data may be related to the second training data, such as both being related to the same type of action during action recognition or the same type of object during object recognition.

[0070] Now refer to Figure 5 The domain classifier method referenced above is described to describe an exemplary adversarial-based domain adaptation. This method may use a gradient reversal layer (GRL) in the domain classifier to adjust weights and thereby confuse the entire architecture / domain classifier so that the domain classifier will gradually lose the ability to distinguish outputs from different domains. The domain classifier itself can be built at least in part by a third neural network separate from the source neural network and the target neural network.

[0071] In view of the foregoing, from block 212 as described above, by Figure 6 Block FC-2 600 is shown as adding an additional shallow binary classifier (referred to as a “domain classifier”) to identify or discern whether the data input to the domain adaptation module at block 212 is from the source domain or the target domain. Figure 5The logic may begin at block 500. Furthermore, before the device backpropagates the gradients to the main model (e.g., the main video model), at block 502, one or more domain classifiers 604 may use a gradient reversal layer (GRL) 602 to invert the gradients so that the video model can be optimized in the opposite direction and thus the domain classifier(s) will gradually lose the ability to distinguish between vectors from the two domains. Thus, the model will be generalized to both the source and target domains.

[0072] Then at block 504, one domain classifier 604 can be inserted immediately after the spatial module 605 of the architecture, and another domain classifier 606 can be inserted immediately after the temporal module 608 of the architecture to perform domain adaptation in both spatial and temporal directions. Then at block 506, the apparatus can backpropagate the gradients to the main model (which in this case can be a video model). Figure 6 An example architecture of the embodiment itself is shown.

[0073] Thus, a device implementing the principles of the present invention may have access to a first neural network / domain that is associated with a first data type and may be a source neural network / domain. The device may also have access to a second neural network / domain that is associated with a second data type that is different from the first data type and may be a target neural network / domain. The device may then provide the first training data as input to the second neural network.

[0074] For example, a first neural network / domain may be related to action recognition using real-world video, while a second neural network / domain may be related to action recognition using video game video. Thus, the first training data may be one frame of a video game rendered graphical punch action from a video game.

[0075] Thereafter, a human supervisor may provide a command to manually select, or the device itself may (e.g., randomly) select the first middle / hidden layer of the second neural network and then identify the first vector output from the first layer of the second neural network for the corresponding video frame. Then, using a third neural network, which may be a domain classifier, the device may determine whether the first vector output is from the first neural network or from the second neural network.

[0076] If the third neural network determines that the first vector output is from the second neural network (e.g., video game video domain), the third neural network is not confused, and therefore one or more weights of the first layer of the second neural network can be adjusted to subsequently confuse the third neural network when it is run again, so that the third neural network classifies the second vector output from the first layer of the second neural network as actually being the vector output from the first neural network rather than the vector output from the second neural network. However, if the second vector output is still classified as being the vector output from the second neural network, the adjusted weights of the first layer can be restored to their previous values, and another layer of the second neural network can be selected and the process repeated.

[0077] However, if, other than the immediately preceding paragraph, the third neural network classifies the first vector output from the first layer of the second neural network as actually being output from the first neural network (e.g., a real-world video domain), the apparatus may refuse to adjust one or more weights of the first layer of the second neural network because the first layer of the second neural network has been optimized to at least some degree (e.g., optimized sufficiently to confuse the third neural network into believing that the first vector output from the second neural network is actually from the first neural network). If desired, another hidden layer may be selected, and the process may be repeated for the other hidden layer of the second neural network.

[0078] Therefore, using the example of action recognition, if the game data output is classified as coming from the game domain by the domain classifier / third neural network, the weights of the hidden layer of the game domain can be adjusted using the "inverse" loss function via the gradient reversal layer of the domain classifier / third neural network to achieve the goal of causing the domain classifier / third neural network to classify subsequent game data output as coming from the real-life video domain.

[0079] It should also be noted that the foregoing with respect to the domain classifier approach can be performed after the third neural network itself (the domain classifier) ​​has been initially trained and optimized for accuracy. During this initial phase of training the third neural network, the third neural network may self-correct in an unsupervised manner when it incorrectly classifies a vector output of labeled data as being from one domain when it is actually from another domain for each label.

[0080] Thus, the weights of the third neural network may be initially random, and then during self-correction, backpropagation from the output layer of the third neural network may be performed to adjust the weights of the third neural network and thereby optimize the third neural network itself (which will build the domain classifier) ​​to correctly classify the output from the hidden layer or output layer as being from one domain or the other.

[0081] Now refer to Figure 7 The Cross-Domain Batch Normalization (CDBN) method referenced above is described to illustrate another version of domain adaptation in accordance with the principles of the present invention, again with reference to video data as an example. The present application recognizes that batch normalization (BN), originally used to improve optimization, can also be modified to benefit domain adaptation. To this end, the CDBN method can be used to modify the CBDN module 800 ( Figure 8 ) is applied to both the spatial region extraction network (SREN) 802 and the video model 804. Using CDBN, a mechanism can adaptively select domain statistics to normalize the input, which can reduce the distribution difference between different video types. Therefore, one of the differences between this CDBN and ordinary BN is that CDBN calculates two statistics: one for the source branch and the other for the target branch. Figure 8 As shown in the example architecture of , two statistics are calculated using a mixture of source and target data with a ratio of α (alpha).

[0082] The CDBN method is now described. Figure 7 Example logic, from block 212 as described above, the logic may be implemented at block 700 by Figure 8 The model begins by adding CDBN after the fully connected layer 806 in the spatial module shown. Then, during training at block 702, the model can learn the optimal ratio α (alpha) to normalize the data of both the source branch and the target branch. Then, during testing at block 704, the statistics for the source branch and the statistics for the target branch can be normalized using α (alpha) and the statistics for the target branch. Then, at block 706, an entropy loss 808 can be added to separate the unlabeled target data.

[0083] Thus, an apparatus embodying the principles of the present invention may access a first neural network associated with a first data type, access a second neural network associated with a second data type, and provide first training data as input to the first neural network. The apparatus may also provide second, different training data as input to the second neural network. The apparatus may then identify a first output from an intermediate layer of the first neural network based on the first training data, and a second output from a parallel intermediate layer of the second neural network based on the second training data. The apparatus may then identify a ratio for normalizing the first output and the second output, and apply an equation that takes the ratio into account to change one or more weights of the intermediate layer of the second neural network.

[0084] The ratio can be related to the mean, and in some examples, both the mean and variance between the first and second outputs can be analyzed to apply the equation. The ratio can be identified and the equation can be applied using Cross-Domain Batch Normalization (CDBN) to have similar means and variances between outputs from parallel intermediate layers.

[0085] As with other methods, the second neural network for CDBN can be built using a copy of the first neural network before providing the second training data to the second neural network. Furthermore, in some examples, the first and second neural networks can be related to action recognition, and the first training data can be correlated with the second training data because both the first and second training data can be related to the same action. In other examples, the first and second neural networks can be related to object recognition, and the first training data can be correlated with the second training data because both the first and second training data can be related to the same object.

[0086] Based on reference Figures 2 to 8 From the foregoing description, it should now be understood that the proposed framework(s) are both general and flexible. Many speaker / user adaptation algorithms can be applied to this framework with slight modifications to one or more of the domain loss or a portion of the source / target models. For example, in speaker adaptation, the adversarial loss can be defined as the speaker classification error so that the deep features learned by the source model will become discriminative for acoustic units (e.g., such as phonemes or words) and invariant to speakers.

[0087] Applications and examples incorporating the principles of the present invention will now be described.

[0088] The principles of the present invention can be used in all possible deep learning based methods for image, video and audio data processing, etc.

[0089] For game object and / or action detection, game videos can be collected and efficient data preparation tools developed to convert raw videos into processed datasets using another existing video dataset following a protocol. This can be combined with the real-world video dataset "Kinetics" to form the first action recognition dataset for domain adaptation. The principles of this invention can be used to recognize multiple objects and actions in both the real world and the game world, and can also be used to evaluate datasets and enhance dataset generation.

[0090] For optical character recognition, the principles of the present invention can be used to recognize different handwriting styles, including standard fonts, artistic text, fonts in games, etc.

[0091] For speech conversion, the principles of the present invention may be used to convert the speech of one speaker into the speech of another speaker.

[0092] To adapt speakers for speech recognition, the principles of the present invention can be applied to audio-related tasks by replacing the input with speech spectrograms. In speaker adaptation, the source model can be pre-trained using the speech of many speakers, and the target domain may only contain a few utterances from a new speaker. In this case, the target domain model can be initialized with the source model. During adaptation, a classification loss on the target domain data and a discrepancy loss between the source and target models can be jointly optimized. The discrepancy loss can be the parameter difference between the source and target models or the phonetic distribution distance between the source and target model outputs.

[0093] For multimodal user adaptation for emotion recognition (e.g., input is text, image, video, and speech with emotion output), given a user's speech or video clip (or both), the domain adaptation module can adapt one user's style to another user's. Therefore, user adaptation can improve the accuracy of emotion recognition for new speakers not in the training set. Furthermore, the spatial region extraction network can be used to detect multiple facial expressions, so emotions can be recognized from multiple people with different styles.

[0094] We will now discuss domain adaptation for action recognition between the game world and the real world in further detail. Figure 4 、 Figure 6 and Figure 8 An example architecture to be used according to this type of domain adaptation is shown in .

[0095] In the gaming industry, video and audio can be two separate processes. Games are often initially designed and produced without audio, and the audio team then reviews the entire game video and inserts corresponding SFX from the game's sound effects (SFX) database. Algorithms can be developed based on the principles of the present invention to enable machines to automatically analyze the visual content from game videos and then match the corresponding SFX with the analysis results to optimize the process.

[0096] Deep learning technology can also be used to analyze game video content. Action recognition is a crucial task for SFX matching, as most important sound effects are associated with character movements. Action recognition using deep learning methods can be applied to identify in-game actions and automatically identify and locate the SFX associated with those actions, accelerating the game production process.

[0097] Unfortunately, most (if not all) existing action recognition systems are used for real-world videos, which means that they can all show performance on real-world datasets. Those trained models cannot be directly used on game videos because there is a huge distribution difference, also known as dataset shift or dataset bias. Therefore, by using the principles of the present invention, models can be trained using data collected from game videos, using domain adaptation to reduce the impact of dataset shift for deep architectures for video tasks, including action recognition.

[0098] The following will refer to Figure 9 and Figure 10 The logic shown in the flowchart of describes the model as learning domain relations between game videos and real-world videos.

[0099] For real-action datasets, game videos can be collected and efficient data preparation tools can be developed to convert raw videos into processed datasets using another existing video dataset following a common protocol, e.g. Figure 9 This can then be combined with the real-world video dataset “Kinetics” to form the first action recognition dataset for domain adaptation, as shown in block 900. Figure 9 This is reflected in box 902.

[0100] Then, according to Figure 9 In block 904, a baseline method may be provided for action recognition, for example, without any domain adaptation techniques for fair comparison. Then, for video domain adaptation, a first action recognition architecture may be developed that integrates several domain adaptation techniques (e.g., difference-based, adversarial-based, and normalization-based) into a pipeline to improve performance over the baseline, such as Figure 9 This is reflected in box 906 .

[0101] Therefore, if Figure 11 As shown in , a baseline architecture for action recognition can be established. The input raw video can be fed forward to a 101-layer ResNet to extract frame-level feature vectors. The number of feature vectors can correspond to the number of video frames. The feature vectors can then be uniformly sampled and fed into the model. Figure 11As shown, the entire model can be divided into two parts - a spatial module 1100 and a temporal module 1102. The spatial module can include a fully connected layer 1104, a rectified linear unit (ReLU) 1106, and a dropout layer 1108. The spatial module can convert a general feature vector 1110 into a task-driven feature vector, which can be action recognition. The temporal module 1102 is designed to aggregate frame-level feature vectors to form a single video-level feature vector to represent each video. The average value can be calculated for all feature elements along the time direction to generate a video-level feature vector. This technique is sometimes called temporal pooling. The video-level feature vector can then be fed to the last fully connected layer 1112 as a classifier to generate a prediction 1114. The prediction can be used to calculate the classification loss and is then used to optimize the entire model.

[0102] Then, according to Figure 10 In block 1000 of FIGURE 1, one or more domain adaptation (DA) methods as described herein may be integrated into the baseline architecture: difference-based DA, adversarial-based DA, and normalization-based DA (e.g., Figure 4 、 Figure 6 and Figure 8 Then, an unsupervised domain adaptation protocol can be followed, where the training data includes labeled data from the source domain and unlabeled data from the target domain (according to Figure 10 1002), and the test data can all come from the target domain (according to Figure 10 For further details on the domain adaptation method according to this example for action recognition, please refer to the back Figures 2 to 8 and its corresponding description.

[0103] To evaluate the performance of various domain adaptation methods, the dataset can include data from both the virtual and real domains. Game videos can then be collected from several games to construct a dataset of game actions for the virtual domain. For example, the total length of the videos can be five hours and forty-one minutes. All original and untrimmed videos can be segmented into video clips based on the annotations. Each video clip can have a total length of 10 seconds, and the minimum length can be 1 second. The entire dataset can also be divided into training, validation, and test sets by randomly selecting videos from each category at a ratio of 7:2:1. For the real domain, Kinetics-600 can be used.

[0104] By following the closed-loop setting of domain adaptation, thirty overlapping categories can be selected between the virtual domain and the real domain. Categories may include, for example, resting, carrying, cleaning the floor, climbing, crawling, squatting, crying, dancing, drinking, driving, falling, fighting, hugging, jumping, kicking, turning on the light, news anchor, opening the door, paint brush, paragliding, falling, pushing, reading, running, shooting, staring, talking, throwing, walking, and washing dishes. Each category can correspond to multiple categories in Kinetics-600 or the virtual / game dataset. For example, the category "reading" can correspond to the categories reading a book and reading a newspaper in Kinetics-600.

[0105] Then, we can use both domains to construct a video game real-move dataset. For the virtual domain, we might have a total of 2,625 training videos and 749 validation videos. For the real-world domain, we can randomly select 100 videos per category to maintain a similar scale of training data between the real and virtual domains, and we can use all the validation videos from the original Kinetics-600 setup. This results in a total of 3,000 videos for training and 3,256 for validation. In addition, we might have 542 videos for pure testing.

[0106] The proposed domain adaptation method can then be evaluated on a self-collected virtual dataset. In some examples, the implementation can be based on the PyTorch framework. A ResNet-101 1116 model pre-trained on ImageNet raw videos 1118 can be utilized as a frame-level feature extractor. Before feeding into the model, a fixed number of frame-level feature vectors with equal space in the temporal direction for each video can be sampled. For adequate comparison, twenty-five frames can be sampled for testing by following the common protocol in action recognition. For training, only five frames can be sampled given any limitations on computing resources. For optimization, the initial learning rate can be 0.1, and a learning rate reduction strategy can be followed. Stochastic gradient descent (SGD) can be used as an optimizer with momentum and weight decay of 0.9 and 1×10-4. The batch size may be 512, half of which may come from labeled source data and half from unlabeled target data.

[0107] Then, the experimental protocol for unsupervised domain adaptation can be followed, with the following experimental settings (where all settings can be tested on a virtual validation set): Oracle, trained with a labeled virtual training set without any domain adaptation method; Source-only, trained with a labeled real-action training set without any domain adaptation method; Difference-based domain adaptation, trained with a labeled real-action training set and an unlabeled virtual training set with a difference-based domain adaptation method; Adversarial-based domain adaptation, trained with a labeled real-action training set and an unlabeled virtual training set with an adversarial-based domain adaptation method; and Normalization-based domain adaptation, trained with a labeled real-action training set and an unlabeled virtual training set with a normalization-based domain adaptation method.

[0108] Example results are shown in Figure 12 The difference between the Oracle 1200 and the source-only setup 1202 is the domain used for training. First, the Oracle setup can be considered an upper bound, free of domain shift issues, while the source-only setup represents a lower bound, directly applying a model trained using data from a different domain. As shown, the accuracy difference is 50 percent. Figure 12 It is also shown that each of the three domain adaptation methods 1204 disclosed herein can mitigate the domain shift problem. Among them, in this example, the normalization-based domain adaptation has the best performance, improving the accuracy by 9.2%.

[0109] Domain adaptation for emotion recognition will now be discussed in further detail. Given limited user-specific audio and video samples, multimodal emotion recognition accuracy can be improved. User adaptation can contribute to deep learning-based emotion recognition accuracy using, for example, audio-only, video-only, or both audio and video data, and a user adaptation structure adapted to a general domain adaptation framework according to the principles of the present invention.

[0110] Figure 13 Depicts the baseline model structure for this example, with further reference to Figure 14 The logic is reflected in the flowchart of Figure 2. The same model structure can be used for both audio and video emotion recognition.

[0111] First, a feature sequence 1300 can be extracted from the original data 1302 ( Figure 13 ),like Figure 14 Then, a speaker-independent (SI) model 1304 may be trained using a plurality of speaker training datasets, such as Figure 14 The model structure may include a stack of three bidirectional long short-term memory (BLSTM) layers 1306, and each layer 1306 may have 512 units in each direction. Features may be sent to the model frame by frame, and Figure 14 At block 1404, the temporal average layer 1308 can take the temporal average of the hidden states of the last LSTM layer and use it as the utterance embedding. The 1024-dimensional embedding can then be reduced to 256 dimensions using a fully connected layer 1310 at block 1406 and then passed through a softmax classifier 1312 at block 1408 to convert the embedding into a posterior sentiment probability. The model can be trained by minimizing the cross-entropy error.

[0112] Therefore, separate models can be trained using both audio and video data. During testing, each pair of audio and video test data can be aligned to the same utterance during a preprocessing step. For each pair, emotion posterior probabilities can be calculated from both models and averaged to obtain the final probability for the decision. This approach can be referred to as "decision fusion."

[0113] Figure 15 Describes the user adaptation structure of this example, with further reference to Figure 16 In order to adapt the pre-trained SI model to a new user using limited adaptation data from the new speaker, Figure 16 At block 1600 of FIGURE 16, the speaker-dependent (SD) model 1500 is initialized from the SI model 1502 (top branch). For user adaptation, practical applications may sometimes mean that only target (new user) adaptation data can be used during adaptation. Therefore, the source data (many speakers used to train the SI model) may not be used as in the general structure.

[0114] The loss function may include a sum of two terms, one of which is a cross entropy classification loss defined for the target domain data, and the other is the L2 distance of the model parameters between the source model and the target model, which may be similar to the difference loss in the general structure. By jointly optimizing these two terms at blocks 1602 and 1604, respectively, the target model can learn to correctly classify emotions for each new user at block 1606 and also avoid being adapted to be too far away from the source model. Since, for example, only target domain data is used, Figure 15 The user-adapted structure in

[15] can modify the general structure so that the classification error can be defined only for the target data. The user-adapted structure can also modify the general structure by adopting a specific form of difference loss, which can be the L2 norm between the source model and the target model.

[0115] As an example based on the principles of the present invention, audio emotion recordings of eighty-four speakers can be collected for training an audio SI model. For testing, five additional speakers that do not appear in the training set can be used. There may be ten emotion categories in the database. These can be merged into six categories, including happiness, anger, sadness, fear, surprise, and others (including, for example, excitement, boredom, neutrality, disappointment, and disgust), and unweighted accuracy can be reported, which can be calculated as the average of the individual accuracies of these six categories. For video data, 114 speakers can be collected for training. For testing, the same five-speaker test set can be used, where the audio and video have been aligned for each utterance.

[0116] Then, to perform user adaptation, up to 150 utterances can be randomly selected for each of the five test speakers as the maximum adaptation set. The remaining utterances can be used for testing. There are a total of 2661 possible utterances for the five test speakers, so after removing 150 adapted utterances for each speaker, there may still be 1911 utterances for testing, which may make the results statistically significant in this example.

[0117] The number of adaptation data for each speaker can also be varied from five to 150 utterances. To compare the results, all smaller adaptation sets can be selected from 150 utterances so that the test sets can be the same.

[0118] Using the adapted data, the audio and video models can be adapted separately, and at test time, the performance of the individual models as well as the decision fusion performance can be tested. Forty-dimensional log-mel filter bank audio features can be used, as well as additional frame energy, first-order and second-order deltas (a total of 123 dimensions). The audio frame length can be 25ms and shifted every 10ms. For each frame, video features can be extracted from the last layer (1024 dimensions) of the VGG model. The VGG model can be pre-trained on the FERPlus dataset, which is a dataset for facial expression recognition. 136-dimensional landmark facial points can also be attached to each frame.

[0119] For model training and adaptation, a mini-batch size of 40 utterances / videos can be used, with an Adam optimizer used to minimize the loss function. The initial learning rate when training the SI model can be set to 0.001, and when the classification accuracy on the development set decreases, it can be multiplied by 0.1. For adaptation, the learning rate can be fixed at 0.001, the audio model can be adapted for 5 epochs, and the video model can be adapted for 25 epochs on the adaptation set.

[0120] Figure 17A table showing the accuracy of six example emotion recognition classes on a test set before and after user adaptation is shown. SI_A, SI_V, and SI_AV refer to the performance of the SI model, using audio only, video only, and decision fusion. Similarly, SD_A, SD_V, and SD_AV show the results after adaptation. It should be understood that for each individual modality, user adaptation can improve baseline performance, and more adapted data yields better recognition accuracy. Similarly, decision fusion can provide better accuracy than using only a single modality.

[0121] Continue to Figure 18 , which shows all three domain adaptation methods used together in accordance with the principles of the present invention using a domain adaptation module 1800 to optimize a first (target) domain 1802 derived from a second (source) domain 1804.

[0122] From the foregoing detailed description it should be apparent that the inventive principles therefore improve the adaptation and training of neural networks through the technical solutions described herein.

[0123] It will be appreciated that while the principles of the invention have been described with reference to certain exemplary embodiments, these embodiments are not intended to be limiting and that various alternative arrangements may be used to implement the subject matter claimed herein.

Claims

1. A device for performing domain adaptation from a source domain to a target domain, the device comprising: at least one processor; as well as at least one computer storage device that is not a transient signal and that includes instructions executable by the at least one processor to: accessing a first neural network, the first neural network being associated with a first data type; accessing a second neural network associated with a second data type different from the first data type; providing first training data from the source domain as input to the first neural network, the first training data comprising image, video, or audio data; providing second training data from the target domain as input to the second neural network, the second training data comprising image, video, or audio data, the first training data being different from the second training data, wherein the first training data is related to the second training data, and wherein the first training data and the second training data are related to similar concepts or metaphors in the source domain and the target domain; identifying a first output from an intermediate layer of the first neural network, the first output being based on the first training data; identifying a second output from an intermediate layer of the second neural network, the second output being based on the second training data, the corresponding intermediate layers of the first neural network and the second neural network being parallel layers, the parallel layers performing the same one or more tasks and / or having the same purpose; identifying a ratio for normalizing the first output and the second output; Applying an equation that takes the ratio into account to change one or more weights of the intermediate layer of the second neural network to minimize the difference between the first output and the second output, thereby optimizing the second neural network for the second data type.

2. The apparatus of claim 1, wherein the ratio is related to an average value. 3 . The apparatus of claim 1 , wherein both a mean and a variance between the first output and the second output are analyzed to apply the equation. 4 . The apparatus of claim 1 , wherein Cross-Domain Batch Normalization (CDBN) is used to identify the ratio and apply the equation.

5. The apparatus of claim 1 , wherein the second neural network is established by a copy of the first neural network before providing the second training data to the second neural network.

6. The apparatus according to claim 1, wherein the intermediate layers of the first neural network and the second neural network are layers other than an output layer.

7. The apparatus of claim 1 , wherein the first neural network and the second neural network are related to action recognition, and wherein the first training data is correlated with the second training data in that both the first training data and the second training data are related to the same action.

8. The apparatus of claim 1 , wherein the first neural network and the second neural network are related to object recognition, and wherein the first training data is related to the second training data in that both the first training data and the second training data are related to the same object.

9. A method for performing domain adaptation from a source domain to a target domain, the method comprising: accessing a first neural network, the first neural network being associated with a first data type; accessing a second neural network associated with a second data type different from the first data type; providing first training data from the source domain as input to the first neural network, the first training data comprising image, video, or audio data; providing second training data from the target domain as input to the second neural network, the second training data comprising image, video, or audio data, the first training data being different from the second training data, wherein the first training data is related to the second training data, and wherein the first training data and the second training data are related to similar concepts or metaphors in the source domain and the target domain; identifying a first output from a hidden layer of the first neural network, the first output being based on the first training data; identifying a second output from a hidden layer of the second neural network, the second output being based on the second training data, the corresponding hidden layers of the first neural network and the second neural network being parallel layers, the parallel layers performing the same one or more tasks and / or having the same purpose; identifying a ratio for normalizing the first output and the second output; Applying an equation that takes the ratio into account to change one or more weights of the intermediate layer of the second neural network to minimize the difference between the first output and the second output, thereby optimizing the second neural network for the second data type.

10. The method of claim 9, wherein the ratio is related to an average value.

11. The method of claim 9, wherein the identifying and applying steps are performed using a Cross-Domain Batch Normalization (CDBN) module.

Images