Systems and methods for domain adaptation in neural networks using domain classifier

Abstract

The domain adaptation module (1800) is configured to optimize the first domain (1802) derived from the second domain (1804) using individual outputs from individual parallel hidden layers of the domains.

Description

Technical Field

[0001] This application generally relates to technically innovative and unconventional solutions that must be rooted in computer technology and produce specific technical improvements. Background Technology

[0002] Machine learning (sometimes called deep learning) can be used for a variety of useful applications related to data understanding, detection, and / or classification, including image classification, optical character recognition (OCR), object recognition, action recognition, speech recognition, and emotion recognition. However, as understood in this paper, machine learning systems may not be sufficient to recognize actions in a domain such as computer games using training datasets from another domain (e.g., movie videos).

[0003] For example, in the computer game industry, video and audio are two separate processes. First, a game without audio is designed and produced. Then, an audio team investigates the entire game video and inserts the corresponding SFX from a sound effects (SFX) database—a very time-consuming process. As understood in this paper, machine learning can be used to accelerate this process; however, current motion recognition models are trained on real-world video datasets, making them susceptible to dataset shifts or biases when used with game videos. Summary of the Invention

[0004] To overcome the aforementioned domain mismatch problem, at least two common domains of the training data (images, videos, or audio) are used to classify the target dataset. A pair of training data domains can be established, for example, real-world videos and computer game videos, first and second speaker speech (for speech recognition), standard font text and cursive script (for handwriting recognition), etc.

[0005] Therefore, a general domain adaptation module, built from the loss function and / or the actual neural network, receives input from multiple output points in both training domains of the deep learning and provides output measures, allowing optimization of one or both of the two trajectories of the neural network. Alternatively, a general cross-domain feature normalization module can be used and inserted into any layer of the neural network.

[0006] Therefore, in one aspect, the device includes at least one processor and at least one computer storage device, the computer storage device being not a transient signal and including instructions executable by the at least one processor. The instructions are executable to access a first neural network associated with a first data type, to access a second neural network associated with a second data type different from the first data type, and to provide first training data as input to the second neural network. The instructions are also executable to select a first layer, wherein the first layer is a hidden layer of the second neural network. The instructions are then executable to identify an output from the first layer generated based on the first training data, and to use a third neural network to determine whether the output from the first layer originates from the first neural network. The third neural network is different from the first and second neural networks. The instructions are also executable to adjust one or more weights of the first layer based on the determination that the output from the first layer does not originate from the first neural network.

[0007] In some examples, the instructions may be executable to initially build a second neural network by replicating the first neural network.

[0008] Similarly, in some examples, the instructions may be executable to reject adjusting one or more weights of the first layer based on determining that the output from the first layer is from the first neural network. In some implementations, the output may be a first output, and the instructions may be executable to select a second layer based on determining that the first output from the first layer is from the first neural network, wherein the second layer is also a hidden layer of the second neural network. The instructions may also be executable to identify a second output, wherein the second output is from the second layer, and to use a third neural network to determine whether the second output is from the first neural network. The instructions may then be executable to adjust one or more weights of the second layer based on determining that the second output is not from the first neural network. The first and second layers of the second neural network may be randomly selected.

[0009] Additionally, before using the third neural network to determine whether the output from the first layer originates from the first neural network, the instructions may be executable to adjust one or more weights of one or more layers of the third neural network, causing the third neural network to learn to correctly classify the outputs from layers of either the first or second neural network. The third neural network can even operate in an unsupervised mode to learn to correctly classify the outputs from layers of either the first or second neural network using labeled data.

[0010] In another aspect, a method includes: accessing a first neural network associated with a first data type; accessing a second neural network associated with a second data type different from the first data type; and providing first training data as input to the second neural network. The method further includes: selecting a first layer, wherein the first layer is a hidden layer of the second neural network; identifying an output generated from the first layer based on the first training data; and using a third neural network to determine whether the output from the first layer originates from the first neural network. The third neural network is different from the first and second neural networks. The method also includes adjusting one or more weights of the first layer based on the determination that the output from the first layer does not originate from the first neural network.

[0011] In another aspect, an apparatus includes at least one computer storage device, which is not a transient signal and includes instructions executable by at least one processor to: access a first domain associated with a first domain class; access a second domain associated with a second domain class different from the first domain class; and classify a target dataset using training data provided to the first and second domains. The instructions are also capable of executing to output a classification of the target dataset, wherein the target dataset is classified by a domain adaptation module including a domain classifier to invert gradients and backpropagate the gradients to the main model.

[0012] The details of both the structure and operation of this application can be best understood with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the drawings: Attached Figure Description

[0013] Figure 1 This is a block diagram of an example system that conforms to the principles of this invention;

[0014] Figure 2 , Figure 3 , Figure 5 , Figure 7 , Figure 9 , Figure 10 , Figure 14 and Figure 16 This is a flowchart illustrating example logic that conforms to the principles of this invention;

[0015] Figure 4 , Figure 6 , Figure 8 , Figure 11 , Figure 13 , Figure 15 and Figure 18 Examples of various domain adaptation architectures according to the principles of the present invention are shown; and

[0016] Figure 12 and Figure 17 This is an example table illustrating the principles of the present invention. Detailed Implementation

[0017] According to the principles of this invention, deep learning-based domain adaptation methods can be used to overcome the domain mismatch problem for tasks related to images, videos, or audio, such as understanding / detection / classification given any source and target domain data. At least three common data types (images, videos, or audio) can be used, and all types of neural network modules can be used to improve system performance.

[0018] As described in this paper, the two trajectories of a deep learning processing flow can be used for any specific input-to-output task. One trajectory can be used for one data domain, and the other trajectory can be used for another data domain, such that for each of the two data domains, there can be at least two deep learning trajectories. For example, a pair of domains could be two types of video, such as real-world video and video game world video, one speaker's speech and another speaker's speech, standard font text and cursive, speech recognition domain, text-to-speech, and speech-to-text.

[0019] The general domain adaptation module will be described below, and sometimes a loss function is used. The general domain adaptation module can also use actual neural network connections that take inputs from multiple output points and provide output measures from two trajectories of the deep learning network, allowing optimization of the two trajectories. The general domain adaptation module can also use a general cross-domain feature normalization module, which can be inserted into any layer of the neural network.

[0020] Therefore, the methods described in this paper can involve multiple objects and multiple actions associated with multiple objects. For example, a block of image text containing many texts can be an "object", and the type of the image block can be an "action".

[0021] This disclosure also relates generally to a computer ecosystem, which includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, distributed computer gaming networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, and artificial neural networks and machine learning applications.

[0022] The system described herein may include server and client components, which are connected via a network to enable data exchange between the client and server components. The client component may include one or more computing devices, including AR headsets, VR headsets, and game consoles (such as Sony PlayStation). ®This includes client devices such as motherboards, game controllers, portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers (such as laptops and tablets), and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some client computers may use operating systems such as Orbis or Linux, operating systems from Microsoft, or Unix, or operating systems manufactured by Apple or Google. These operating environments can be used to execute one or more programs / applications, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Furthermore, the operating environment according to the principles of the invention can be used to execute one or more computer game programs / applications and other programs / applications that implement the principles of the invention.

[0023] The server and / or gateway may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively or additionally, the client and server may connect via a local intranet or virtual private network. The server or controller may be instantiated from a game console and / or one or more of its motherboards (such as Sony PlayStation®), a personal computer, etc.

[0024] Information can be exchanged between clients and servers via a network. For this purpose, and for security reasons, servers and / or clients may include firewalls, load balancers, temporary storage devices, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device for implementing a method of providing secure communities, such as online social networking sites or video game websites, to network users through crowdsourced communication, according to the principles of the present invention.

[0025] As used herein, an instruction refers to a computer-implemented step for processing information within a system. Instructions can be implemented in software, firmware, or hardware, and include any type of programmed step implemented by components of the system.

[0026] A processor can be any conventional general-purpose single-chip or multi-chip processor capable of performing logic using various lines such as address lines, data lines, and control lines, as well as registers and shift registers.

[0027] The software modules described herein may include various subroutines, programs, etc. Without limiting this disclosure, logic stated to be performed by a particular module may be reallocated to other software modules and / or combined together in a single module and / or made available in a shareable library.

[0028] As indicated above, the principles of the invention described herein can be implemented in hardware, software, firmware, or a combination thereof; therefore, they are described in accordance with the functionality of the illustrative components, frames, modules, circuits, and steps.

[0029] In addition to the above, the logic blocks, modules, and circuits described below may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices (such as application-specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components, or any combination thereof) designed to perform the functions described herein. The processor may be implemented by a combination of a controller or state machine or computing device.

[0030] The functions and methods described below can be implemented using hardware or software circuitry. When implemented in software, the functions and methods can be written in a suitable language such as, but not limited to, Java, C#, or C++, and can be stored on or transmitted via a computer-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disc read-only memory (CD-ROM) or other optical disc storage devices (such as Digital Universal Optical Disc (DVD)), magnetic disk storage devices, or other magnetic storage devices including removable thumb drives. Connections can be established using computer-readable media. Such connections can include, for example, hardwired cables, including fiber optic and coaxial cables, as well as digital subscriber line (DSL) and twisted-pair cables. Such connections can include wireless communication connections, including infrared and radio.

[0031] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0032] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes the following systems: having only A; having only B; having only C; having both A and B; having both A and C; having both B and C; and / or having both A, B and C, etc.

[0033] Now for specific reference Figure 1An example system 10 is shown, which may include one or more of the example devices mentioned above and further described below according to the principles of the invention. The first example device included in system 10 is a consumer electronics (CE) device such as an audio-visual device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling the TV). However, the AVD 12 may alternatively be a home appliance or household item, such as a computerized internet-enabled refrigerator, washer, or dryer. Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) telephone, tablet computer, laptop computer, augmented reality (AR) headset, virtual reality (VR) headset, internet-enabled or “smart” glasses, another type of wearable computerized device, such as a computerized internet-enabled watch, a computerized internet-enabled bracelet, a computerized internet-enabled music player, a computerized internet-enabled headset, a computerized internet-enabled implantable device (such as an implantable skin device), other computerized internet-enabled devices, etc. In any case, it should be understood that AVD 12 is configured to implement the principles of the present invention (e.g., to communicate with other consumer electronics (CE) devices to implement the principles of the present invention, to perform the logic described herein, and to perform any other functions and / or operations described herein).

[0034] Therefore, in order to implement this principle, AVD 12 can be... Figure 1 Some or all of the components shown may be constructed. For example, AVD 12 may include one or more displays 14, which may be implemented as high-definition or ultra-high-definition (“4K”) or higher flat screens and may be touch-enabled for receiving user input signals via touch on the displays. AVD 12 may include: one or more speakers 16 for outputting audio according to the principles of the invention; and at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. Furthermore, it should be noted that network interface 20 may be, for example, a wired or wireless modem or router or other suitable interface (such as, for example, a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above).

[0035] It should be understood that one or more processors control AVD 12 to implement the principles of the present invention, including other elements of AVD 12 as described herein, such as controlling display 14 to present images on the display and receive input from the display. One or more processors may include a central processing unit (CPU) 24 and a graphics processing unit (GPU) 25 on a graphics card 25A.

[0036] In addition to the foregoing, AVD 12 may also include one or more input ports 26, such as, for example, a High Definition Multimedia Interface (HDMI) port or a USB port for physically connecting (e.g., using a wired connection) to another consumer electronics (CE) device and / or a headphone port for connecting headphones to AVD 12 to present audio from AVD 12 to a user via headphones. For example, input port 26 may be connected via wired or wireless connection to a wired or satellite source 26a of audio / video content. Thus, source 26a may be, for example, a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disk player containing content that the user may consider a favorite for channel allocation purposes. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 44 and may implement some or all of the logic described herein.

[0037] The AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. These computer memories may, in some cases, be embodied as stand-alone devices within the AVD's housing, or as personal video recording devices (PVRs) or video disk players for playing back AV programs, either inside or outside the AVD's housing, or as removable memory media. Furthermore, in some embodiments, the AVD 12 may include a location or position receiver (such as, but not limited to, a mobile phone receiver, a GPS receiver, and / or an altimeter 30), configured to receive geographic location information, for example, from at least one satellite or cell tower, and provide this information to the processor 24 and / or, in conjunction with the processor 24, determine the altitude at which the AVD 12 is set. However, it should be understood that, according to the principles of the invention, another suitable location receiver besides a mobile phone receiver, GPS receiver, and / or altimeter may be used, for example, to determine the location of the AVD 12 in all three dimensions.

[0038] Continuing the description of AVD 12, in some embodiments, according to the principles of the invention, AVD 12 may include one or more cameras 32, which may be, for example, thermal imaging cameras, digital cameras such as webcams, infrared (IR) cameras, and / or cameras integrated into AVD 12 and controllable by processor 24 to generate pictures / images and / or videos. AVD 12 may also include a Bluetooth transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0039] Furthermore, the AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, gyroscopes, or magnetometers, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the processor 24. The AVD 12 may include a wireless TV broadcast port 38 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12.

[0040] Still referencing Figure 1 In addition to AVD 12, system 10 may also include one or more other consumer electronics (CE) device types. In one example, a first CE device 44 may be used to send computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while a second CE device 46 may include components similar to the first CE device 44. In the example shown, the second CE device 46 may be configured as an AR or VR headset worn by user 47, as illustrated. In the example shown, only two CE devices 44, 46 are shown; it should be understood that fewer or larger devices may be used according to the principles of the invention.

[0041] In the example shown, it is assumed that all three devices 12, 44, and 46 are elements of a network (such as a secure or encrypted network, entertainment network, or Wi-Fi) in a home, or at least are located close to each other in a specific location and are capable of communicating with each other and with the server described herein. However, unless explicitly required otherwise, the principles of the invention are not limited to a specific location or network.

[0042] The example non-limiting first CE device 44 can be established by any of the aforementioned devices (e.g., smartphones, digital assistants, portable wireless laptops or notebook computers, or game controllers (also referred to as "consoles")), and therefore can have one or more of the components described below. The second CE device 46 can be established by, but is not limited to, AR headsets, VR headsets, "smart" internet-enabled glasses, or even video disc players (such as Blu-ray players), game consoles, etc. Furthermore, in some embodiments, the first CE device 44 can be a remote control (RC) for, for example, issuing AV play and pause commands to the AVD 12, or it can be a more complex device, such as a tablet computer, connected via a wired or wireless link to... Figure 1 The device shown communicates with and controls the game controller, personal computer, wireless phone, etc., that renders video games on the AVD12, as well as the game console.

[0043] Therefore, the first CE device 44 may include one or more displays 50, which may be touch-enabled for receiving user input signals via touch on the displays 50. Alternatively or additionally, the displays(one or more) 50 may be at least partially transparent displays configured to display AR and / or VR images, such as AR headset displays or "smart" glasses displays or "head-up" displays, as well as VR headset displays or other displays.

[0044] The first CE device 44 may further include one or more speakers 52 for outputting audio according to the principles of the invention, and at least one additional input device 54, such as an audio receiver / microphone, for inputting audible commands to the first CE device 44 to control the device 44. Example: The first CE device 44 may also include one or more network interfaces 56 for communicating over the network 22 under the control of one or more CE device processors 58. Thus, the interface 56 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including a mesh network interface. It should be understood that the processor 58 controls the first CE device 44 to implement the principles of the invention, including other elements of the first CE device 44 as described herein, such as, for example, controlling the display 50 to present images on the display and receive input from the display. Furthermore, it should be noted that the network interface 56 may be, for example, a wired or wireless modem or router or other suitable interface (such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above).

[0045] Furthermore, it should be noted that, in addition to processor(s) 58, the first CE device 44 may also include a graphics processing unit (GPU) 55 on a graphics card 55A. The GPU 55 may be configured, among other things, to render AR and / or VR images on the display 50.

[0046] In addition to the foregoing, the first CE device 44 may also include one or more input ports 60 (such as HDMI ports or USB ports, for example) for physically connecting (e.g., using a wired connection) to another CE device and / or a headphone port for connecting headphones to the first CE device 44 to present audio from the first CE device 44 to a user via headphones. The first CE device 44 may also include one or more tangible computer-readable storage media 62, such as disk-based storage devices or solid-state storage devices. Furthermore, in some embodiments, the first CE device 44 may include a location or place receiver (such as, but not limited to, a mobile phone and / or GPS receiver and / or altimeter 64), which is configured to receive geographic location information from at least one satellite and / or cell tower, for example using triangulation, and provide said information to the CE device processor 58 and / or in conjunction with the CE device processor 58 to determine the altitude at which the first CE device 44 is set. However, it should be understood that, according to the principles of the invention, another suitable location receiver besides a mobile phone and / or GPS receiver and / or altimeter can be used to determine, for example, the location of the first CE device 44 in all three dimensions.

[0047] Continuing the description of the first CE device 44, in some embodiments, according to the principles of the invention, the first CE device 44 may include one or more cameras 66, which may be, for example, thermal imaging cameras, IR cameras, digital cameras such as webcams, and / or another type of camera integrated into the first CE device 44 and controllable by the CE device processor 58 to generate pictures / images and / or videos. The first CE device 44 may also include a Bluetooth transceiver 68 and other near-field communication (NFC) elements 70 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0048] Furthermore, the first CE device 44 may include one or more auxiliary sensors 72 (e.g., motion sensors such as accelerometers, gyroscopes, gyroscopes, or magnetometers, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the CE device processor 58. The first CE device 44 may also include other sensors that provide input to the CE device processor 58, such as one or more climate sensors 74 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 76. In addition to the foregoing, it should be noted that in some embodiments, the first CE device 44 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 78, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the first CE device 44. The CE device 44 may communicate with the AVD 12 via any of the communication modes and associated components described above.

[0049] The second CE device 46 may include some or all of the components shown for CE device 44. Either or both CE devices may be powered by one or more batteries.

[0050] Referring now to the aforementioned at least one server 80, it includes at least one server processor 82 and at least one tangible computer-readable storage medium 84 (such as a disk-based storage device or a solid-state storage device). In an implementation, the medium 84 includes one or more solid-state storage drives (SSDs). According to the principles of the invention, the server also includes at least one network interface 86, which allows communication via network 22 with... Figure 1 The server communicates with other devices and can actually facilitate communication between the server and client devices. It should be noted that network interface 86 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as a wireless telephone transceiver). Network interface 86 can be a Remote Direct Memory Access (RDMA) interface, which connects media 84 directly to a network, such as a so-called "structure," without going through the server processor 82. The network can include an Ethernet network and / or a Fibre Channel network and / or a wireless bandwidth network. Typically, server 80 includes multiple processors in multiple computers, referred to as "blades" that can be arranged in a physical server "stack."

[0051] Therefore, in some implementations, server 80 may be an internet server or an entire "server farm" and may include and perform "cloud" functionality, enabling the apparatus of system 10 to access a "cloud" environment via server 80, such as in exemplary domain-adapted implementations disclosed herein. Additionally or alternatively, server 80 may be configured with... Figure 1 Other devices shown are implemented in the same room or nearby using one or more game consoles or other computers.

[0052] Before describing the additional figures, it should be understood according to the principles of the invention that, in order to optimize an artificial intelligence system, an optimized source domain / model can be built by replicating well-trained data to establish a target domain / model, which will be further refined for data of a different type than the source domain. For example, the source domain can be used for action recognition in real-world videos, while the target domain can be used for action recognition in video game videos. Due to differences in video type and visual effects, the source domain may be insufficient for performing action recognition using video game data, but it can still provide a good starting point for adapting a sufficient target domain from the video game data for action recognition.

[0053] Therefore, the principles of this invention describe systems and methods for performing domain adaptation and optimization. According to this disclosure, this can be performed not only by backpropagating from the output / activation layers of a neural network once a human supervisor or system administrator has identified an error, but also by running different but related training data on both the target and source domains and selecting any given hidden or intermediate layers parallel to each other for each domain to determine whether the outputs are similar or even identical. If the outputs are statistically dissimilar (as defined by the supervisor or administrator), certain weight adjustments can be performed on the intermediate target layers as described herein to minimize the differences in outputs from the parallel layers (e.g., to ensure that the abstractions for the parallel layers are similar / identical) and thereby further optimize the target domain for different types of data. Then, after training, testing can be performed to ensure that the optimization has been performed to an acceptable degree.

[0054] Regarding the different but related training data, the data may be different because it is data appropriate for a given domain, but related because the training data for each domain may be associated with similar concepts or metaphors. For example, the training data fed into the source domain could be a real-world video of a human undergoing piercing, while the training data fed into the target domain could be a video game video of a game character undergoing piercing. As another example, this time concerning object recognition, the training data fed into the source domain could be a real-world image of an apple, while the training data fed into the target domain could be a video game video of a digital apple.

[0055] Additionally, as used in the references above, parallel source and target intermediate / hidden layers refer to corresponding source and target intermediate layers that are initially identical because the source domain is copied to initially establish the target domain, wherein these layers perform the same(one or more) tasks and / or have the same purpose. Thus, for example, intermediate source layer 500 can be parallel to intermediate target layer 500, wherein the target domain is copied from the source domain, both domains have the same number of intermediate layers, and target layer 500 is initially established from source layer 500.

[0056] In view of the foregoing, the principles of the invention will now be described in more detail. (Refer to the flowchart shown below.) Figure 2 Starting with the logic, as an example of modifying the general trajectory of a neural network (NN) for video classification, the baseline architecture for video classification can be modified as follows. Starting at box 200, a common convolutional neural network (CNN) can be modified into a spatial region extraction network (SREN) to extract feature vectors from the entire video scene and important spatial regions (e.g., objects, body parts, etc.). Then, Figure 2 The logic can proceed to box 202, where the two types of outputs, region features and scene features can be concatenated into a frame-level feature vector, and then input into the video model at box 204.

[0057] Then, Figure 2 The logic can proceed to box 206, where frame-level feature vectors can be input into a recurrent neural network (RNN) including Long Short-Term Memory (LSTM) units to model temporal dynamics. The logic can then proceed to box 208, where the final classifier can be modified to classify both (A) the entire scene and (B) all important regions in (one or more) videos.

[0058] Figure 2 The logic can then proceed to box 210, where boxes 200-208 can be repeated for the second domain category to utilize and optimize the entire architecture with data from different video types / categories. Then, at box 212, the frame-level feature vectors, the features after the RNN, and the classifier output can be input into the domain adaptation module. The domain adaptation module can use one or more of the following three methods, each of which is described in detail below. Figure 3 , Figure 5 and Figure 7 Different flowcharts are shown, and the description is based on video data: Difference function method ( Figure 3 Domain classifier method Figure 5 ) and cross-domain batch normalization method ( Figure 7 ).

[0059] From reference Figure 3Starting with the difference function approach, it should be understood that the difference function can be used to calculate the distance between the overall data distribution of the source and target data. The difference loss can be defined by different measures from any subset of layers of the source / target model (such as a probability-based distance between the source and target data extracted from multiple layers of the model, as described further below), or by normalizing the parameter differences between the source and target models (also described further below), or by a weighted sum of these two types of losses (also described further below). By jointly training with the difference function, the model will be optimized to reduce distributional discrepancies, thereby improving generalization ability.

[0060] Therefore, from box 212 as described above, Figure 3 Starting at box 300, another loss function (different from the overall loss function used during backpropagation from the output layer) can be defined and added. This additional loss function is a difference loss function, which is calculated as the distance between features learned from the source data and target data output from the corresponding parallel layer.

[0061] Without difference loss, the overall loss function can be computed using only labeled source data. Therefore, during optimization, the model will progressively fit the source data, increasing the distributional difference between the two domains. Thus, an unsupervised domain fitting protocol can be used to reduce the overall distributional difference between the source and target data. This protocol uses training data consisting of labeled data from the source domain and unlabeled data from the target domain (typically designated as box 302), and test data entirely from the target domain (typically designated as box 304).

[0062] exist Figure 3 At box 306, the logic might compute the distance between features learned from the source and target data, output from the corresponding parallel layer, without labels. Then, at box 308, joint training with the difference loss function can be used for the model to reduce the difference in the overall distribution between the source and target data. This can be accomplished at box 310 by computing the difference loss using feature vectors from the temporal module and the output of the last fully connected layer. Figure 4 The combination is shown Figure 3 Examples of action recognition architectures that describe these principles.

[0063] Therefore, as Figure 4As shown, the apparatus implementing the principles of the present invention can access a first neural network / domain 400 associated with a first data type that may be a source neural network / domain, access a second neural network / domain 402 associated with a second data type that is different from the first data type that may be a target neural network / domain, and provide first training data as input to the first neural network. The apparatus can also provide second training data as input to the second neural network, wherein the first training data is different from the second training data but is still related.

[0064] For example, the first neural network / domain 400 may be related to object recognition using real-world videos, while the second neural network / domain 402 may be related to object recognition using video game videos. Therefore, the first training data could be a video of a real-world apple from a real-life video recording, while the second training data could be a video of a graphically rendered apple from a video game.

[0065] The device can then identify a first output from a first layer, where the first layer is the output / activation layer of a first neural network, and where the first output is based on first training data. The device can also identify a second output from a second layer, where the second layer is the output / activation layer of a second neural network, and where the second output is based on second training data. The device can then determine a first adjustment to one or more weights of a third layer, where the third layer is an intermediate layer of the second neural network, based on the first and second outputs. The first adjustment can be determined, for example, by using a first difference / loss function to perform backpropagation from the second layer (the output / activation layer of the second neural network).

[0066] Subsequently, a human supervisor can provide commands to manually select, or the device itself can (e.g., randomly) select the third and fourth layers (where the fourth layer is an intermediate layer of the first neural network). The third and fourth layers can be parallel intermediate / hidden layers. A second difference / loss function can then be used to measure the third output from the third layer and compare it with the fourth output from the fourth layer. This second difference / loss function is customized (e.g., by a human supervisor) to measure the similarity between the third and fourth outputs, regardless of whether object labels (e.g., "apple") are available for the second neural network. Before the third and fourth outputs are respectively fed to the corresponding intermediate layers of the respective second and first neural networks, the third and fourth outputs themselves can be the corresponding vector outputs of the respective third and fourth layers, where the third and fourth outputs are themselves based on the second and first training data, respectively.

[0067] The device can then determine a second adjustment to one or more weights of the third layer based on a comparison / second function, where the amount of weight change is proportional to the magnitude of the second function. Subsequently, the device can adjust one or more weights of the third layer (and even one or all previous layers of the second neural network) based on considerations of both the first and second adjustments. For example, one or more weights of the third layer can be adjusted by adding together the corresponding weight changes from the first and second adjustments. However, in some examples, if it is determined by a human supervisor or the device that this results in less loss than the sum of the weight changes from both the first and second adjustments, then only the weight change from either the first or second adjustment can be applied. In other examples, if it is determined by a human supervisor or the device that this results in less loss than the above alternatives, half of the weight change from one or more of the first adjustments and half of the weight change from one or more of the second adjustments can be added together.

[0068] Additionally, it should be noted that the second neural network can be constructed using a copy of the first neural network before the second training data is provided to it. The third and fourth layers of the corresponding neural network can be layers other than the output layer, such as the intermediate hidden layers of the corresponding neural network.

[0069] Additionally, the first training data can be related to the second training data, such as both being related to the same type of action during action recognition or the same type of object during object recognition.

[0070] Now refer to Figure 5 The above-referenced domain classifier method is described to illustrate an exemplary adversarial-based domain adaptation. This method may use a gradient inversion layer (GRL) in the domain classifier to adjust the weights and thus obfuscate the entire architecture / domain classifier, causing it to gradually lose its ability to distinguish outputs from different domains. The domain classifier itself may be at least partially constructed from a third neural network separate from the source and target neural networks.

[0071] Considering the foregoing, from box 212 as described above, via as Figure 6 The box shown, FC-2 600, adds an additional shallow binary classifier (called a "domain classifier") to identify or distinguish whether the data input to the domain adapter module at box 212 comes from the source domain or the target domain. Figure 5The logic can begin at box 500. Furthermore, before the device backpropagates the gradients to the main model (e.g., the main video model), at box 502, one or more domain classifiers 604 can use a gradient inversion layer (GRL) 602 to invert the gradients, allowing the video model to be optimized in the opposite direction, and thus (one or more) domain classifiers will gradually lose the ability to distinguish vectors from both domains. Therefore, the model will be generalized to both the source and target domains.

[0072] Then, at box 504, a domain classifier 604 can be inserted immediately after the spatial module 605 of the architecture, and another domain classifier 606 can be inserted immediately after the temporal module 608 of the architecture, to achieve domain adaptation in both the spatial and temporal directions. Then, at box 506, the device can backpropagate the gradient to the main model (which in this case could be the video model). Figure 6 An example architecture of this implementation is shown.

[0073] Therefore, the apparatus implementing the principles of the present invention can access a first neural network / domain, which is associated with a first data type and may be a source neural network / domain. The apparatus can also access a second neural network / domain, which is associated with a second data type different from the first data type and may be a target neural network / domain. The apparatus can then provide the first training data as input to the second neural network.

[0074] For example, the first neural network / domain could be associated with action recognition using real-world video, while the second neural network / domain could be associated with action recognition using video game video. Therefore, the first training data could be a frame of a graphical punch-hole action rendered from a video game.

[0075] Subsequently, a human supervisor can provide commands to manually select, or the device itself can (e.g., randomly) select the first intermediate / hidden layer of the second neural network, and then identify the first vector output from the first layer of the second neural network for the corresponding video frame. Then, using a third neural network, which can be a domain classifier, the device can determine whether the first vector output comes from the first neural network or the second neural network.

[0076] If the third neural network determines that the first vector output comes from the second neural network (e.g., the video domain of a video game), the third neural network will not be confused, and therefore one or more weights of the first layer of the second neural network can be adjusted to subsequently confuse the third neural network when it runs again, causing the third neural network to classify the second vector output from the first layer of the second neural network as a vector output that actually comes from the first neural network instead of a vector output from the second neural network. However, if the second vector output is still classified as a vector output from the second neural network, the adjusted weights of the first layer can be restored to their previous values, and conversely, another layer of the second neural network can be selected and the process repeated.

[0077] However, if, not immediately following the paragraph above, the third neural network classifies the first vector output from the first layer of the second neural network as actually being from the first neural network (e.g., a real-world video domain), then the device can refuse to adjust one or more weights of the first layer of the second neural network, because the first layer of the second neural network has already been optimized to at least some extent (e.g., optimized enough to confuse the third neural network into thinking that the first vector output from the second neural network actually comes from the first neural network). If necessary, another hidden layer can be selected, and the process can be repeated for another hidden layer of the second neural network.

[0078] Therefore, using the example of action recognition, if the game data output is classified as coming from the game domain by the domain classifier / third neural network, the weights of the hidden layers of the game domain can be adjusted via the gradient reversal layer of the domain classifier / third neural network using an "inverse" loss function, so that the domain classifier / third neural network can classify subsequent game data outputs as coming from the real-life video domain.

[0079] It should also be noted that the foregoing content related to the domain classifier method can be performed after the third neural network itself (the domain classifier) ​​has been initially trained and optimized for accuracy. During this initial phase of training the third neural network, it may self-correct unsupervised if the third neural network incorrectly classifies the vector output of labeled data as coming from one domain when it actually comes from another domain for each label.

[0080] Therefore, the weights of the third neural network can initially be randomized, and then during self-correction, backpropagation from the output layer of the third neural network can be performed to adjust the weights of the third neural network and thus optimize the third neural network itself (which will build the domain classifier) ​​to correctly classify the output from the hidden layer or the output layer as coming from one domain or another.

[0081] Now refer to Figure 7 The cross-domain batch normalization (CDBN) method described above is used to illustrate another version of domain adaptation according to the principles of the present invention, again referring to video data as an example. This application recognizes that the batch normalization (BN) itself, originally used to improve optimization, can also be modified to benefit domain adaptation. For this purpose, the CDBN method can use the CBDN module 800 ( Figure 8 This is applied to both the Spatial Region Extraction Network (SREN) 802 and the video model 804. Using CDBN, a mechanism adaptively selects domain statistics to normalize the input, which reduces distributional differences between different video types. Therefore, one difference between this CDBN and ordinary BN is that CDBN computes two types of statistics: one for the source branch and another for the target branch. As per this implementation... Figure 8 As shown in the example architecture, two types of statistics are calculated using a mixture of source and target data with a ratio of α (alpha).

[0082] Now, let's describe the methods used for CDBN. Figure 7 The example logic, from box 212 as described above, can be implemented at box 700 via, for example... Figure 8 The model begins by adding CDBN after the fully connected layer 806 in the spatial module shown. Then, during training at box 702, the model can learn the optimal ratio α (alpha) to normalize the data for both the source and target branches. Then, during testing at box 704, α (alpha) and statistics for the target branch are used to normalize the statistics for the source and target branches. Finally, at box 706, the entropy loss 808 is added to separate the unlabeled target data.

[0083] Therefore, the apparatus implementing the principles of the present invention can access a first neural network associated with a first data type, access a second neural network associated with a second data type, and provide first training data as input to the first neural network. The apparatus can also provide a second, different training data as input to the second neural network. The apparatus can then identify a first output from an intermediate layer of the first neural network based on the first training data, and identify a second output from a parallel intermediate layer of the second neural network based on the second training data. The apparatus can then identify a ratio for normalizing the first and second outputs, and apply an equation that takes said ratio into account to change one or more weights of the intermediate layers of the second neural network.

[0084] Ratios can be related to the mean, and in some examples, both the mean and variance between the first and second outputs can be analyzed to apply equations. Ratios can be identified, and cross-domain batch normalization (CDBN) can be used to apply equations to achieve similar means and variances between outputs from parallel intermediate layers.

[0085] Similar to other methods, the second neural network of the CDBN method can be built using a copy of the first neural network before feeding the second training data to the second neural network. Furthermore, in some examples, the first and second neural networks can be related to action recognition, and the first training data can be related to the second training data, since both the first and second training data can be related to the same action. In other examples, the first and second neural networks can be related to object recognition, and the first training data can be related to the second training data, since both the first and second training data can be related to the same object.

[0086] Based on reference Figures 2 to 8 From the foregoing description, it should now be understood that the proposed framework(s) is both general and flexible. Many speaker / user adaptation algorithms can be applied to this framework with slight modifications to one or more domain loss parameters or to a portion of the source / target model. For example, in speaker adaptation, adversarial loss can be defined as speaker classification error, such that the deep features learned by the source model become discriminative to acoustic units (e.g., phonemes or words) and speaker-invariant.

[0087] Applications and examples incorporating the principles of this invention will now be described.

[0088] The principles of this invention can be used in all possible deep learning-based methods for image, video, and audio data processing, etc.

[0089] For game object and / or motion detection, game videos can be collected and efficient data preparation tools can be developed to convert raw videos into processed datasets using another existing video dataset, following a protocol. This can be combined with the real-world video dataset "Kinetics" to form a first motion recognition dataset for domain adaptation. The principles of this invention can be used to identify multiple objects and actions in both the real-world and game worlds, and can also be used to evaluate datasets and enhance dataset generation.

[0090] For optical character recognition, the principles of this invention can be used to recognize different handwriting styles, including standard fonts, artistic text, and fonts used in games.

[0091] For speech conversion, the principles of this invention can be used to convert the speech of one speaker into the speech of another speaker.

[0092] To adapt speakers for speech recognition, the principles of this invention can be applied to audio-related tasks by replacing the input with a speech spectrogram. In speaker adaptation, the source model can be pre-trained using speech from many speakers, and the target domain may only contain some utterances from new speakers. In this case, the target domain model can be initialized from the source model. During adaptation, the classification loss for the target domain data and the difference loss between the source and target models can be jointly optimized. The difference loss can be the parameter difference between the source and target models, or the telephone distribution distance between the outputs of the source and target models.

[0093] For multimodal user adaptation for emotion recognition (e.g., inputs are text, images, videos, and speech with emotional output), given a user's speech or video clip (or both), the domain adaptation module can adapt one user's style to another user. Therefore, user adaptation can improve the accuracy of emotion recognition for new speakers not in the training set. Furthermore, spatial region extraction networks can be used to detect various facial expressions, thus enabling the identification of emotions from multiple individuals with different styles.

[0094] We will now discuss in more detail the domain adaptation used for action recognition between the game world and the real world, which has already been discussed in detail. Figure 4 , Figure 6 and Figure 8 The example schema shown is for adapting to this type of domain.

[0095] In the gaming industry, video and audio can often be two separate processes. Games are typically designed and produced initially without audio, and then an audio team investigates the entire game video and inserts corresponding SFX from the game's sound effects (SFX) database. Algorithms can be developed based on the principles of this invention to enable machines to automatically analyze the visual content from game videos and then match the corresponding SFX with the analysis results to optimize the process.

[0096] Deep learning technology can also be used to analyze game video content. Motion recognition is a crucial task in SFX matching because most important sound effects are related to character movements. For motion recognition using deep learning methods, these methods can be applied to identify actions in games and automatically identify and locate SFX associated with corresponding actions, thereby accelerating the game development process.

[0097] Unfortunately, most (if not all) existing action recognition systems are designed for real-world videos, meaning they all demonstrate performance on real-world datasets. Those trained models cannot be directly applied to game videos due to significant distributional discrepancies, also known as dataset shift or dataset bias. Therefore, by using the principles of this invention, models can be trained using data collected from game videos, employing domain adaptation to mitigate the impact of dataset shift on video tasks, including deep architectures for action recognition.

[0098] The following will refer to Figure 9 and Figure 10 The logic shown in the flowchart describes the model as learning the domain relationship between game videos and real-world videos.

[0099] For real-world action datasets, game videos can be collected, and efficient data preparation tools can be developed to convert raw videos into processed datasets using another existing video dataset, following public protocols. Figure 9 The 900-frame bounding box reflects this. This can then be combined with the real-world video dataset "Kinetics" to form the first action recognition dataset for domain adaptation, such as... Figure 9 The frame 902 reflects this.

[0100] Then, according to Figure 9 Box 904, for example, can provide a baseline approach for action recognition in the absence of any domain adaptation techniques for fair comparison. Then, for video domain adaptation, a first action recognition architecture can be developed that integrates several domain adaptation techniques (e.g., difference-based, adversarial-based, and normalization-based) into the pipeline to improve performance through the baseline, such as... Figure 9 The frame 906 reflects this.

[0101] Therefore, as Figure 11 As shown, a baseline architecture for action recognition can be established. The raw input video can be fed forward into a 101-layer ResNet to extract frame-level feature vectors. The number of feature vectors can correspond to the number of video frames. The feature vectors can then be uniformly sampled and fed into the model. Figure 11As shown, the entire model can be divided into two parts: a spatial module 1100 and a temporal module 1102. The spatial module may include a fully connected layer 1104, a rectified linear unit (ReLU) 1106, and a dropout layer 1108. The spatial module can transform the general feature vector 1110 into a task-driven feature vector, which could be for action recognition. The temporal module 1102 aims to aggregate frame-level feature vectors to form a single video-level feature vector representing each video. The average of all feature elements can be calculated along the temporal direction to generate the video-level feature vector. This technique is sometimes referred to as temporal pooling. The video-level feature vector can then be fed into a final fully connected layer 1112 as a classifier to generate a prediction 1114. The prediction can be used to calculate the classification loss and then used to optimize the entire model.

[0102] Then, according to Figure 10 Box 1000 allows for the integration of one or more domain adaptation (DA) methods, as described herein, into the baseline architecture: difference-based domain adaptation, adversarial domain adaptation, and normalization-based domain adaptation (e.g., Figure 4 , Figure 6 and Figure 8 (As shown separately). Then, an unsupervised domain adaptation protocol can be followed, where the training data includes labeled data from the source domain and unlabeled data from the target domain (according to...). Figure 10 (Box 1002), while the test data can all come from the target domain (according to...) Figure 10 (Box 1004). For further details on the domain adaptation method for this example of action recognition, please refer to the previous section. Figures 2 to 8 And its corresponding description.

[0103] To evaluate the performance of various domain adaptation methods, the dataset can include data from both the virtual and real domains. Game videos can then be collected from several games to construct a game action dataset for the virtual domain. As an example, the total length of the videos could be five hours and forty-one minutes. All raw and untrimmed videos can be segmented into video clips based on annotations. The total length of each video clip can be 10 seconds, with a minimum length of 1 second. The entire dataset can also be divided into training, validation, and test sets by randomly selecting videos from each category at a ratio of 7:2:1. For the real domain, Kinetics-600 can be used.

[0104] By adhering to the closed settings of domain adaptation, thirty overlapping categories can be selected between the virtual and real domains. Categories can include, for example, resting, carrying, cleaning the floor, climbing, crawling, crouching, crying, dancing, drinking, driving, falling, fighting, hugging, jumping, kicking, turning on the light, news anchoring, opening the door, paintbrush, paragliding, falling, pushing, reading, running, shooting, gazing, talking, throwing, walking, and washing dishes. Each category can correspond to multiple categories in the Kinetics-600 or virtual / game datasets. For example, the category "reading" could correspond to the categories of reading books and reading newspapers in Kinetics-600.

[0105] Then, a video game real-world action dataset can be constructed using two domains. For the virtual domain, there could be a total of 2625 training videos and 749 validation videos. For the real-world domain, 100 videos can be randomly selected for each category to maintain a similar scale of training data between the real and virtual domains, and all validation videos from the original Kinetics-600 setup can be used. In total, there could be 3000 videos for training and 3256 videos for validation. Additionally, there could be 542 videos for pure testing.

[0106] The proposed domain adaptation method can then be evaluated on a self-collected virtual dataset. In some examples, the implementation can be based on the PyTorch framework. A ResNet-101 1116 model pre-trained on the original ImageNet video 1118 can be used as a frame-level feature extractor. Before feeding the model, a fixed number of frame-level feature vectors with equal spatiality in the temporal direction can be sampled for each video. For sufficient comparison, twenty-five frames can be sampled for testing by following common protocols in action recognition. For training, given any limitations in computational resources, only five frames can be sampled. For optimization, the initial learning rate can be 0.1, and a learning rate reduction strategy can be followed. Stochastic gradient descent (SGD) can be used as the optimizer, with momentum and weight decay of 0.9 and 1×10⁻⁴, respectively. The batch size can be 512, with half potentially coming from labeled source data and half from unlabeled target data.

[0107] Then, an experimental protocol for unsupervised domain adaptation can be followed, with the following experimental setups (all of which can be tested on a virtual validation set): Oracle, trained on a labeled virtual training set without any domain adaptation method; Source only, trained on a labeled real action training set without any domain adaptation method; Difference-based domain adaptation, trained on a labeled real action training set and an unlabeled virtual training set with a difference-based domain adaptation method; Adversarial-based domain adaptation, trained on a labeled real action training set and an unlabeled virtual training set with an adversarial-based domain adaptation method; and Normalization-based domain adaptation, trained on a labeled real action training set and an unlabeled virtual training set with a normalization-based domain adaptation method.

[0108] Example results are shown in Figure 12 The difference between Oracle 1200 and Source Only 1202 lies in the domain used for training. First, the Oracle setting can be viewed as an upper bound without domain shift issues, while the Source Only setting represents a lower bound directly applied to models trained using data from different domains. As shown in the figure, the accuracy difference is fifty percent. Figure 12 It is also shown that each of the three domain adaptation methods 1204 disclosed herein can mitigate the domain shift problem. Among them, in this example, the normalization-based domain adaptation has the best performance, thereby improving the accuracy by 9.2%.

[0109] Domain adaptation for emotion recognition will now be discussed in more detail. Given a limited number of user-specific audio and video samples, the accuracy of multimodal emotion recognition can be improved. User adaptation structures, such as those using only audio, only video, or both audio and video data, and adapted to a general domain adaptation framework based on the principles of this invention, can contribute to the accuracy of deep learning-based emotion recognition.

[0110] Figure 13 The baseline model structure for this example is depicted, with further reference to... Figure 14 The logic is reflected in the flowchart. The same model structure can be used for audio and video emotion recognition.

[0111] First, feature sequence 1300 can be extracted from the original data 1302. Figure 13 ),like Figure 14 The bounding box 1400 reflects this. Then, a speaker-independent (SI) model 1304 can be trained using multiple speaker training datasets, such as... Figure 14 The model structure can be represented by a stack of three bidirectional long short-term memory (BLSTM) layers 1306, with each layer 1306 having 512 units in each direction. Features can be sent to the model frame by frame, and... Figure 14 At box 1404, the temporal averaging layer 1308 can use the temporal average of the hidden states of the last LSTM layer as the utterance embedding. Then, at box 1406, a fully connected layer 1310 can be used to reduce the 1024-dimensional embedding to 256 dimensions, and then at box 1408, it is passed through a softmax classifier 1312 to convert the embedding into a posterior sentiment probability. The model can be trained by minimizing the cross-entropy error.

[0112] Therefore, separate models can be trained using audio and video data. During testing, each audio and video test data pair can be aligned to the same utterance during the preprocessing step. For each pair, the posterior probability of emotion can be calculated from the two models and averaged to obtain the final probability for making a decision. This method can be referred to as "decision fusion".

[0113] Figure 15 The user adaptation structure of this example is depicted, with further references to Figure 16 The logic is reflected in the flowchart. To adapt the pre-trained SI model to the new user using limited adaptation data from the new speaker, it is possible to... Figure 16 At box 1600, the speaker-related (SD) model 1500 is initialized from the SI model 1502 (top branch). For user adaptation, practical applications may sometimes mean that only target (new user) adaptation data can be used during adaptation. Therefore, the source data (the many speakers used to train the SI model) may not be used as in the general structure.

[0114] The loss function can consist of the sum of two terms: a cross-entropy classification loss defined for the target domain data, and an L2 distance between the model parameters of the source and target models, which may be similar to the difference loss in a general structure. By jointly optimizing these two terms at boxes 1602 and 1604 respectively, the target model can learn to correctly classify sentiment for each new user at box 1606, and can also avoid being adapted to be too far removed from the source model. Since, for example, only target domain data is used, therefore... Figure 15 The user-adaptive structure can modify the general structure so that classification errors can be defined only for the target data. The user-adaptive structure can also modify the general structure by employing a specific form of difference loss, which can be the L2 norm between the source and target models.

[0115] As an example of the principles of the invention, audio emotion recordings of eighty-four speakers can be collected for training an audio SI model. For testing, five additional speakers not present in the training set can be used. The database may contain ten emotion categories. These can be merged into six categories, including happiness, anger, sadness, fear, surprise, and others (including, for example, excitement, boredom, neutrality, disappointment, and nausea), and an unweighted precision can be reported, which can be calculated as the average of the individual precisions of these six categories. For video data, 114 speakers can be collected for training. For testing, a test set of the same five speakers can be used, where the audio and video have been aligned for each utterance.

[0116] Next, to perform user adaptation, up to 150 utterances can be randomly selected for each of the five test speakers as the maximum adaptation set. The remaining utterances can be used for testing. The five test speakers could have a total of 2661 utterances, so after removing 150 adapted utterances for each speaker, there might still be 1911 utterances available for testing, which could make the result statistically significant in this example.

[0117] The number of fit data for each speaker can also vary from five to 150 utterances. To compare results, all smaller fit sets can be selected from the 150 utterances, making the test sets identical.

[0118] Using adaptive data, audio and video models can be adapted separately, and the performance of individual models and decision fusion can be tested during testing. Audio features can be used with a 40-dimensional log-mel filter bank, along with additional frame energy, first-order and second-order increments (a total of 123 dimensions). The audio frame length can be 25 ms, shifted every 10 ms. Video features can be extracted for each frame from the last layer (1024 dimensions) of the VGG model. The VGG model can be pre-trained on the FERPlus dataset, a dataset used for facial expression recognition. 136-dimensional landmark facial points can also be appended to each frame.

[0119] For model training and fitting, a mini-batch size of 40 utterances / videos can be used, with an Adam optimizer to minimize the loss function. The initial learning rate when training the SI model can be set to 0.001, and it can be multiplied by 0.1 as classification accuracy on the development set decreases. For fitting, the learning rate can be fixed at 0.001, allowing for fitting of the audio model for 5 epochs and the video model for 25 epochs on the fitting set.

[0120] Figure 17Tables show the example six-class emotion recognition accuracy on the test set before and after user adaptation. SI_A, SI_V, and SI_AV refer to the performance of the SI model, audio only, video only, and decision fusion. Similarly, SD_A, SD_V, and SD_AV show the results after adaptation. It should be understood that for each individual modality, user adaptation can improve baseline performance, and more adapted data yields better recognition accuracy. Likewise, decision fusion can provide better accuracy compared to using only a single modality.

[0121] Continue to Figure 18 It illustrates all three domain adaptation methods used together by the domain adaptation module 1800 according to the principles of the present invention to optimize the first (target) domain 1802 derived from the second (source) domain 1804.

[0122] As should be understood from the foregoing detailed description, the principles of this invention thus improve the adaptation and training of neural networks through the technical solutions described herein.

[0123] It should be understood that although the principles of the invention have been described with reference to some exemplary embodiments, these embodiments are not intended to be limiting, and various alternative arrangements can be used to achieve the subject matter claimed herein.

Claims

1. An apparatus for performing domain adaptation, the apparatus comprising: At least one processor; as well as At least one computer storage device, which is not a transient signal and includes instructions executable by the at least one processor to: Access a first neural network for action recognition, the first neural network being associated with a first data type, wherein the first data type is real-world video from real-life video recordings; Access a second neural network for action recognition, the second neural network being associated with a second data type different from the first data type, wherein the second data type is a rendered video game video from a video game; The first training data is provided as input to the second neural network; Select the first layer from the multiple hidden layers of the second neural network; Identify the spatial output from the first layer, which is generated based on the first training data; A third neural network is used to determine whether the spatial output from the first layer comes from the first neural network, the third neural network being different from both the first neural network and the second neural network; Based on the determination that the spatial output from the first layer does not originate from the first neural network, adjust one or more weights of the first layer; Select the second layer from the multiple hidden layers of the second neural network; Identify the time output from the second layer, which is generated based on the first training data; Using a fourth neural network, it is determined whether the time output from the second layer originates from the first neural network, wherein the fourth neural network is different from the first neural network, the second neural network, and the third neural network; and Based on the determination that the time output from the second layer does not originate from the first neural network, one or more weights of the second layer are adjusted.

2. The device of claim 1, wherein the instructions are executable by the at least one processor to: The second neural network is initially built by replicating the first neural network.

3. The device of claim 1, wherein the instructions are executable by the at least one processor to: Based on the determination that the spatial output from the first layer is from the first neural network, one or more weights of the first layer are not adjusted.

4. The device of claim 3, wherein the space output is a first space output, and wherein the instructions are executable by the at least one processor to: Based on the determination that the first spatial output from the first layer is from the first neural network, a third layer is selected from multiple hidden layers of the second neural network; Identify the second spatial output, which originates from the third layer; Using the third neural network, it is determined whether the second spatial output comes from the first neural network; as well as Based on the determination that the second spatial output does not originate from the first neural network, one or more weights of the third layer are adjusted.

5. The device of claim 4, wherein the first layer and the third layer of the second neural network are randomly selected.

6. The device of claim 1, wherein the instructions are executable by the at least one processor to: Before using the third neural network to determine whether the spatial output from the first layer comes from the first neural network, one or more weights of one or more layers of the third neural network are adjusted so that the third neural network learns to correctly classify the spatial output from layers of either the first neural network or the second neural network.

7. The device of claim 6, wherein the third neural network operates in an unsupervised mode to learn to correctly classify the spatial outputs of layers from either the first neural network or the second neural network using labeled data.

8. A method for performing domain adaptation, the method comprising: Access a first neural network for action recognition, the first neural network being associated with a first data type, wherein the first data type is real-world video from real-life video recordings; Access a second neural network for action recognition, the second neural network being associated with a second data type different from the first data type, wherein the second data type is a rendered video game video from a video game; The first training data is provided as input to the second neural network; Select the first layer from the multiple hidden layers of the second neural network; Identify the spatial output from the first layer, which is generated based on the first training data; A third neural network is used to determine whether the spatial output from the first layer comes from the first neural network, the third neural network being different from both the first neural network and the second neural network; Based on the determination that the spatial output from the first layer does not originate from the first neural network, adjust one or more weights of the first layer; Select the second layer from the multiple hidden layers of the second neural network; Identify the time output from the second layer, which is generated based on the first training data; Using a fourth neural network, it is determined whether the time output from the second layer originates from the first neural network, wherein the fourth neural network is different from the first neural network, the second neural network, and the third neural network; and Based on the determination that the time output from the second layer does not originate from the first neural network, one or more weights of the second layer are adjusted.

9. The method of claim 8, further comprising: Using the third neural network, at least in part by using the third neural network to identify the spatial output from the first layer as being related to the first data type, it is determined whether the spatial output from the first layer comes from the first neural network.

10. The method of claim 8, further comprising: Based on the determination that the spatial output from the first layer is from the first neural network, one or more weights of the first layer are not adjusted.

11. The method of claim 10, wherein the spatial output is a first spatial output, and wherein the method comprises: Based on the determination that the first spatial output from the first layer is from the first neural network, a third layer is selected from multiple hidden layers of the second neural network; Identify the second spatial output from the third layer; Using the third neural network, it is determined whether the second spatial output comes from the first neural network; as well as Based on the determination that the second spatial output does not originate from the first neural network, one or more weights of the third layer are adjusted.

12. The method of claim 8, wherein the first layer is selected based on commands from a human supervisor.

13. The method of claim 8, further comprising: Before using the third neural network to determine whether the spatial output from the first layer comes from the first neural network, one or more weights of one or more layers of the third neural network are adjusted so that the third neural network learns to correctly classify the spatial output from layers of either the first neural network or the second neural network.

14. The method of claim 8, wherein the third neural network operates in an unsupervised mode to learn to correctly classify the spatial outputs of layers from either the first neural network or the second neural network using labeled data.

15. The method of claim 8, further comprising: The second neural network is initially built by replicating the first neural network.

16. An apparatus for performing domain adaptation, the apparatus comprising: At least one computer storage device, which is not a transient signal and includes instructions executable by at least one processor to: A first domain classifier is used to determine whether the spatial output of the first hidden layer of the first model comes from the first model or from a second model that is different from the first model, the first model and the second model being associated with different data domains; Based on the determination that the spatial output comes from the second model, adjust one or more weights of the first hidden layer; A second domain classifier is used to determine whether the temporal output of the second hidden layer from the first model comes from the first model or the second model, and the second domain classifier is different from the first domain classifier. as well as Based on the determination that the time output comes from the second model, one or more weights of the second hidden layer are adjusted.

17. The device of claim 16, wherein the different data domains include a first domain relating to real-world video from a real-life video recording and a second domain relating to rendered computer game video from a video game.

18. The device of claim 16, wherein the different data fields include a first field relating to information derived from the first speech and a second field relating to information derived from the second speech.

19. The device of claim 16, wherein the different data fields include a first field relating to standard font text and a second field relating to cursive script.

20. The device of claim 16, wherein the first domain classifier and the second domain classifier use a gradient inversion layer (GRL) that receives data from the spatial model and the temporal model to invert the gradient.

Images