Synchronous Input Feedback for Machine Learning

Abstract

A method and system for providing synchronized input feedback includes: receiving an input event, encoding the input event in an output stream, wherein the encoded input event is synchronized with a specific event, and reproducing the output stream via an output device, wherein the encoded input event in the reproduced output stream is imperceptible to a user.

Description

Technical Field

[0001] This disclosure relates to generating datasets for machine learning. In particular, this disclosure relates to generating datasets with synchronous input feedback for training neural networks. Background Technology

[0002] Software quality assurance and release testing typically require several people spending many days to discover bugs and ensure content meets quality standards. Machine learning has been proposed as a way to reduce the number of staff and time required to perform quality assurance and release testing tasks. A current challenge in using machine learning in this field is that training the machine requires staff to repeat the testing process many times while the machine receives input from testers.

[0003] It is against this backdrop that the proposed implementation plan is put forward. Attached Figure Description

[0004] Various aspects of this disclosure will be readily understood by considering the following specific embodiments, taken in conjunction with the accompanying drawings, in which:

[0005] Figure 1 A block diagram depicts a method for synchronizing input feedback according to various aspects of this disclosure.

[0006] Figure 2A A block diagram of a method for synchronizing input feedback with an audio output stream according to various aspects of this disclosure is shown.

[0007] Figure 2B A block diagram depicts a method for synchronizing input feedback as a watermark in a video output stream, according to various aspects of this disclosure.

[0008] Figure 2C A block diagram is shown of a method for synchronizing input feedback as metadata in a video stream, according to various aspects of this disclosure.

[0009] Figure 3 A block diagram depicts a method for performing machine learning using an output stream of coded input events, according to various aspects of this disclosure.

[0010] Figure 4 Illustrations are shown of methods for synchronizing input feedback with an audio output stream according to various aspects of this disclosure.

[0011] Figure 5A It is a simplified node graph of a recurrent neural network used in a system with synchronous input feedback according to various aspects of this disclosure.

[0012] Figure 5BIt is a simplified node graph of an expanded recurrent neural network used in a system with synchronous input feedback according to various aspects of this disclosure.

[0013] Figure 5C This is a simplified diagram of a convolutional neural network used in a system with synchronous input feedback, according to various aspects of this disclosure.

[0014] Figure 5D This is a block diagram of a method for training a neural network in a system with synchronous input feedback, according to various aspects of this disclosure.

[0015] Figure 6 A block diagram of a system for implementing synchronous input feedback using a neural network according to various aspects of this disclosure is depicted. Detailed Implementation

[0016] While the following detailed description contains many specific details for illustrative purposes, those skilled in the art will understand that many variations and modifications of these details are within the scope of this disclosure. Therefore, the examples of embodiments of this disclosure described below are set forth without loss of generality and without imposing limitations on the claimed disclosure.

[0017] While numerous specific details have been set forth to provide a thorough understanding of embodiments of this disclosure, those skilled in the art will understand that other embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure this disclosure. Some portions of the description herein are presented as algorithms and symbolic representations of operations on data bits or binary digital signals within computer memory. These algorithmic descriptions and representations can be techniques used by those skilled in the art of data processing to convey the essence of their work to others skilled in the art.

[0018] The algorithms used in this paper are self-consistent sequences of actions or operations that lead to the desired result. These actions or operations include physical manipulation of physical quantities. Typically, although not necessary, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. It has been shown that, primarily for reasons of general use, these signals may sometimes be appropriately referred to as bits, values, elements, symbols, characters, items, numbers, etc.

[0019] Unless explicitly stated or apparent from the following discussion, it should be understood that throughout this specification, discussions using terms such as “processing,” “computing,” “converting,” “coordinating,” “determining,” or “identifying” refer to the actions and processes of a computer platform, an electronic computing device including a processor that manipulates and converts data represented as physical (e.g., electronic) quantities in processor registers and accessible platform memory into other data similarly represented as physical quantities in the computer platform memory, processor registers, or display screen.

[0020] Computer programs can be stored on computer-readable storage media, such as, but not limited to, any type of disk, including floppy disks, optical disks (e.g., optical disc read-only memory (CD-ROM), digital video discs (DVD), Blu-ray discs). TM (etc.), as well as magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards or optical cards, flash memory or any other type of non-transitory medium suitable for storing electronic instructions.

[0021] The terms “coupled” and “connected”, and their derivatives, are used herein to describe structural relationships between components of a device used to perform the operations described herein. It should be understood that these terms are not intended to be synonyms. Rather, in certain embodiments, “connected” can be used to indicate that two or more elements are in direct physical or electrical contact with each other. In some instances, “connected,” “connected,” and their derivatives are used to indicate logical relationships, such as between node layers in a neural network (NN). “Coupled” can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other (using other intermediary elements between them), and / or that two or more elements cooperate or communicate with each other (e.g., in causal relationships).

[0022] Currently, training artificial agents for testing video games and other software requires numerous workers to execute and repeat test protocols to generate a sufficient amount of training data. Synchronization is a major problem; for current methods, all training must be performed in real-time using a machine learning algorithm that receives tester input while the tester executes their test protocol. Recorded test performance cannot be tested because current machines cannot synchronize input from users with specific events occurring in the software environment. Therefore, aspects of this disclosure aim to create a system and method for encoding input events from users into an output stream so that machine learning will be able to synchronize the input with specific events occurring in the software environment. In some implementations, synchronizing the time of input with the resulting output can be greatly simplified by combining input and output timing information into a single output file.

[0023] Input feedback mechanism

[0024] Figure 1 A method for synchronizing input feedback according to various aspects of this disclosure is described. The method begins by receiving an input event 101 from an input device. The input event may be a button press, joystick input, analog gamepad input, steering wheel input, potentiometer input, etc. In some implementations, the input event is synchronized with a specific event occurring within the software environment. In some cases, the specific event is an output event or a series of output events in an output stream. For example, but not limited to, in video games, a button press (input event) may be synchronized prior to an action performed by an avatar within the game (specific event). As another example, in a music player, a potentiometer movement (input event) may precede an increase in amplitude (specific event). Synchronization of the input event with one or more specific events is important for testing certain applications (such as video games) and user interfaces, as malfunctioning applications may exhibit incorrect behavior, prolonged pauses, or unresponsive button presses. The input event is then encoded into an output stream 102. The output stream may include an audio stream, a video stream, etc. During encoding 102, synchronization between the input event and the specific event should be maintained within the output stream. As an example, and not a limitation of video games, the input event preceding the avatar's action in the video stream is placed once before the avatar's action occurs in the video game's audio stream. In this example, the audio and video streams are synchronized, which is common in most types of multimedia presentations. The output device can then utilize the encoded input to reproduce the output stream and transmit it, as indicated at 103. As an example, and not a limitation, the output device could be a speaker, a display screen, or a television screen. When reproduced by the output device, the encoded input embedded in the output stream during reproduction at 103 is undetectable or at least imperceptible to the user, who is a person with average visual and auditory abilities.

[0025] Figure 2A and Figure 4A method for synchronizing input feedback with an audio output stream is shown. Input events 401 and 402 are received from an input device. As described above, during most multimedia presentations, the audio stream 404 and the video stream are synchronized. This allows specific events occurring in either the video or audio output stream to be synchronized with encoded input events placed in either output stream at appropriate times. The advantage of encoding input events in the audio stream is that the audio output stream has a relatively high sampling rate compared to the video output stream. During encoding, the low-frequency portion 406 or the high-frequency portion of the audio stream 202 is removed by applying a high-pass or low-pass frequency filter. The low-frequency portion of the audio stream corresponds to infrasound, i.e., sound vibrations below 20 Hz. The high-frequency portion of the audio stream corresponds to ultrasound, i.e., sound vibrations above 20 kHz. Typically, infrasound and ultrasound are undetectable, or at least imperceptible, to adults. The input events can be converted to audio tones 403 and then mixed into audio streams 203 and 407 by summing them in the frequency domain and dividing the total by the total number of sound frequencies added together at each time step, as shown in Equation 1. Where f(a) is the frequency of the output stream at time step t, and f(b) 1…n ) represents the encoded pitch, and n represents the pitch number.

[0026] [(f(a)+f(b1)+…f(b n )) / (n+1)] t Formula 1.

[0027] like Figure 4 As shown, the lengths of button presses 401 and 402 can be encoded by the length of pitch 406. Each type of button press can be encoded as a different frequency pitch. The output audio stream with encoded pitches can be decomposed, for example, by Fourier transform, and analyzed to determine the presence of the input pitch. In the case of synchronized output audio and video streams, the synchronized audio and video streams can be saved to a single output file, which greatly simplifies the timing between input events and specific output events. In the example shown, the press of the down arrow button 401 can be encoded at a higher frequency 406 than the press of the right arrow button 402, and no pitch is encoded between button presses. During playback, the pitch encoded into the audio data is undetectable to the user but is decodeable for use by the neural network. Specifically, according to aspects of this disclosure, the neural network can use the information decoded from the pitch to determine which specific events occurring during gameplay are associated with input events associated with the encoded pitches.

[0028] Figure 2BA method for synchronizing input feedback as a watermark in a video output stream is described. Input events 204 are received from an input device. These input events can be synchronized with specific events occurring in the video or audio output stream. The input events are then encoded as a watermark in the video stream 205 while maintaining synchronization. For example, keystrokes can be encoded in the video output stream before movement in the video output stream (e.g., movement of an avatar in a video game) or before an increase in the amplitude of the output audio stream. The watermark can be, but is not limited to, located in an A-channel, a screen position, or an invisible transparent location.

[0029] Figure 2C A method for synchronizing input feedback as metadata in a video stream is illustrated. As previously described, a synchronization input event 206 is received from an input device. The synchronization input event is then encoded into the metadata of the video stream in sync with the video stream. The metadata can be the header of each frame of the video stream, supplementary enhancement information, etc. When the input event is encoded in the video output stream, the rate of the input event encoded in the output stream is locked to the frame rate of the video stream. Therefore, the rate of the encoded input can be limited to 60 to 120 Hz, which is a typical frame rate for video streams. Conversely, there are fewer restrictions on encoding the input into an audio stream because the sampling rate of the audio stream is much higher than that of the video stream.

[0030] Improvements in machine learning

[0031] Typically, large amounts of training data are needed to train machine learning systems (such as neural networks) for software testing. A key component of software testing is ensuring that input events correspond to correct software behavior and do not lead to crashes or other undesirable events that negatively impact the user experience. As mentioned above, current training methods for testing neural networks require testers to execute test protocols while the neural network is learning. This real-time test data serves as a labeled dataset. Testing must be performed in real-time because there is currently no easy way to provide synchronous input event information in a software environment that is not configured to provide such information to the neural network. Such software environments include video games and user interfaces.

[0032] According to various aspects of this disclosure, the input feedback mechanism can improve the training of a neural network (NN) by providing input event information that is synchronized with specific events occurring within the system's audio or video output stream. Figure 3The diagram below illustrates a method for improving neural network (NN) training using an output stream with encoded input events. The method begins by receiving an output stream with synchronized input events encoded within the stream 301. The received output stream is split, and filters are applied to one of the split output streams to separate the input events from the actual output 302. In some implementations, the NN can be trained to separate the input events from the output stream. Alternatively, a decoder application can pull out the input events and feed them separately to the NN. In some implementations, the output stream can be duplicated, with a low-pass filter applied to one duplicate and a high-pass filter applied to the other. In another implementation, the output stream can be split into channels, thus separating one channel from the others. In some implementations, frame metadata can be separated from the video data.

[0033] Once the encoded input events are separated from the output stream, the NN can be trained using the encoded input events to represent the real input events. This approach is advantageous because the encoded input events are synchronized with specific events. This synchronization can occur between events in the output stream or between events in an output stream different from the one that includes the encoded input events. For example, but not limited to, synchronization can occur between an input event encoded as pitch in an audio stream and the activation of a user interface element in a video stream. Maintaining this synchronization in an easily accessible medium, such as the output stream, improves the training of the NN because it allows training using a recorded dataset without significant changes to the configuration of the host system. As used herein, the term "host system" refers to the apparatus on which the output stream is reproduced and the NN is trained. As an example and not a limitation, the NN can be trained for software testing according to the methods discussed below.

[0034] Machine learning training

[0035] The neural networks, deep learning, or other machine learning used to implement software testing can include one or more of several different types of neural networks and can have many different layers. By way of example and not limitation, neural networks can include one or more of convolutional neural networks (CNNs), recurrent neural networks (RNNs), and / or dynamic neural networks (DNNs).

[0036] Figure 5A The basic form of an RNN with 520 layers is described, where each node is characterized by an activation function S, an input weight U, recursive hidden node transition weights W, and an output transition weight V. The activation function S can be any nonlinear function known in the art and is not limited to the hyperbolic tangent (tanh) function. For example, the activation function S can be sigmoid or ReLU. Unlike other types of neural networks, an RNN has a set of activation functions and weights throughout the layer. Figure 5B As shown, an RNN can be considered as a series of nodes 520 that move between times T and T+1 with the same activation function. Therefore, an RNN maintains historical information by feeding the results from the previous time T into the current time T+1.

[0037] The weights U, W, and V can be configured in several ways. For example, the input weight U can be applied based on video pixel values, audio sample values, input event values, etc. These different input weights can be stored in a lookup table and applied as needed. There may be default values ​​initially applied by the system. Users can then manually modify these values, or they can be automatically modified through machine learning.

[0038] In some implementations, a convolutional RNN (CRNN) can be used. Another type of RNN that can be used is a long short-term memory (LSTM) neural network, which adds storage blocks to RNN nodes with input gate activation functions, output gate activation functions, and forget gate activation functions to form a gated memory, which allows the network to retain some information for a longer period of time, as described in Hochreiter and Schmidhuber's "Long Short-term memory" (Neural Computation 9(8): 1735-1780(1997)), which is incorporated herein by reference.

[0039] Figure 5C An example layout of a convolutional neural network, such as a CRNN, according to various aspects of this disclosure is depicted. In this depiction, a convolutional neural network is generated for image 532, which has a height of 4 units and a width of 4 units, giving a total area of ​​16 units. The depicted convolutional neural network has a filter 533 with a height of 2 units and a width of 2 units, a jump value of 1, and a channel size of 9 for the channel 536. For clarity, in Figure 5C Only the connection 534 between the first column channel and its filtering window is depicted. However, aspects of this disclosure are not limited to this implementation. According to aspects of this disclosure, the convolutional neural network implementing software test 529 can have any number of additional neural network node layers 531 and can include any size of such layer type, such as additional convolutional layers, fully connected layers, pooling layers, max pooling layers, local contrast normalization layers, etc.

[0040] As in Figure 5D As seen in [the previous section], training a neural network (NN) begins with the initialization of the NN's weights

[541] . Typically, the initial weights should be randomly assigned. For example, an NN with a tanh activation function should have weights distributed as follows: and The random value between n and n is the number of inputs to the node.

[0041] After initialization, the activation function and optimization procedure are defined. Then, features or an input dataset are provided to the neural network (NN) 542. In some implementations, the NN can be provided with feature vectors corresponding to inputs with known labels. The NN then predicts the label or class of the features or input 543. The predicted label or class is compared to the known label or class (also called ground truth), and the loss function measures the total error between the prediction and the ground truth on all training samples 544. As an example and not a limitation, the loss function can be a cross-entropy loss function, a quadratic cost function, a triplet contrast function, an exponential cost function, etc. Multiple different loss functions can be used depending on the purpose. The results of the loss function are then used to optimize and train the NN using known neural network training methods (such as backpropagation with stochastic gradient descent, etc.) 545. At each training epoch, the optimization procedure attempts to select model parameters (i.e., weights) that minimize the training loss function (i.e., the total error). The data is divided into training samples, validation samples, and test samples.

[0042] During training, the optimization procedure minimizes the loss function of the training samples. After each training epoch, the pattern is evaluated on the validation samples by calculating the validation loss and accuracy. If there is no significant change, training may stop, and the resulting trained model can be used to predict the labels for the test data.

[0043] Therefore, a software testing neural network can be trained based on audio, video, and encoded input event streams with known labels or categories to identify and classify anomalous software activity by minimizing cross-entropy loss given a known target label.

[0044] Implementation

[0045] Figure 6 A system for providing synchronized input feedback according to various aspects of this disclosure is described. The system may include a computing device 600 coupled to a user input device 602. The user input device 602 may be a controller, touchscreen, microphone, keyboard, mouse, joystick, or other similar device.

[0046] The computing device 600 may include one or more processor units 603, which may be configured according to well-known architectures (such as, for example, single-core, dual-core, quad-core, multi-core, processor-coprocessor, unit processor, etc.). The computing device may also include one or more memory units 604 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), etc.).

[0047] Processor unit 603 can execute one or more programs, portions of which can be stored in memory 604, and processor 603 is operatively coupled to memory (e.g., accessed via data bus 605). The program can be configured to implement filter 608 as input events separate from the output stream. Additionally, memory 604 may contain a program implementing training of NN 621. Memory 604 may also contain software modules such as encoding module 610, which encodes input events from user input device 602 into an output stream. The output stream can be generated by executing program 617 in mass storage device 615, or received from network 620, and can be stored in a buffer in memory 604 before or after encoding the input events.

[0048] The overall structure and possibilities of the neural network (NN) can also be stored as data 618 in mass storage device 615. Processor unit 603 is further configured to execute one or more programs 617 stored in mass storage device 615 or memory 604, which cause the processor to execute method 100 to encode input events from input device 602 into output stream 610. Additionally, processor unit 604 can be configured to execute methods for training the NN according to methods 300 and 500 discussed above. The system can generate neural networks as part of the NN training process. These neural networks can be stored in memory 604 within the sound classification NN module 621. The complete NN can be stored in memory 604 or as data 618 in mass storage device 615. Program 617 (or portions thereof) can also be configured, for example, by appropriate programming, to apply appropriate filters 608 to the output stream to generate encoded input events and a separate output stream, train the NN 621 with the encoded input events, and synchronously encode the input events into output stream 610.

[0049] The computing device 600 may also include known support circuitry, such as input / output (I / O) 607, circuitry, power supply (P / S) 611, clock (CLK) 612, and cache 613, which may communicate with other components of the system, for example, via bus 605. The computing device may include a network interface 614. The processor unit 603 and network interface 614 may be configured to implement a local area network (LAN) or personal area network (PAN) via a suitable network protocol for the PAN (e.g., Bluetooth). The computing device may optionally include a mass storage device 615 (e.g., a disk drive, CD-ROM drive, tape drive, flash memory, etc.), and the mass storage device may store programs and / or data. The computing device may also include a user interface 616 for facilitating interaction between the system and a user. The user interface may include a monitor, television screen, speaker, headphones, or other means of conveying information to the user.

[0050] Computing device 600 may include a network interface 614 to facilitate communication via an electronic communication network 620. The network interface 614 may be configured to enable wired or wireless communication via a local area network (LAN) and a wide area network (WAN) such as the Internet. Device 600 may send and receive data and / or requests via one or more message packets through network 620. Message packets sent via network 620 may be temporarily stored in a buffer 609 in memory 604. Output streams may be received via network 620, or output streams may be generated locally from program 617.

[0051] While the foregoing provides a complete description of preferred embodiments of this disclosure, various alternatives, modifications, and equivalents are possible. Therefore, it should be understood that the foregoing description is intended to be illustrative and not restrictive. For example, although the flowcharts in the accompanying drawings illustrate a specific sequence of operations performed by certain embodiments of this disclosure, it should be understood that this sequence is not necessary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.). Furthermore, many other embodiments will be apparent to those skilled in the art upon reading and understanding the foregoing description. Although this disclosure has been described with reference to specific exemplary embodiments, it will be appreciated that this disclosure is not limited to the described embodiments but can be practiced with modifications and variations within the spirit and scope of the appended claims. Therefore, the scope of this disclosure should be determined by reference to the appended claims and the full scope of the authorized equivalents of such claims. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, The indefinite article "a" or "a kind""This refers to the quantity of one or more items following the article, unless otherwise expressly stated herein. The appended claims should not be construed as including means plus functional limitations, unless such limitations are expressly stated in a given claim using the phrase "means for..."

Claims

1. A method for providing synchronous input feedback: a) Receive input events; b) Encoding the input events in an output stream, wherein the encoding of the input events is synchronized with specific events occurring within the video game, wherein the output stream includes an audio stream, and wherein different types of input events are encoded by tones of different frequencies in the audio stream; and c) Couple the output stream to one or more neural networks, wherein the encoded input events in the output stream are imperceptible to the user when reproduced by an output device.

2. The method of claim 1, wherein the input event is encoded as a sub-tone.

3. The method of claim 1, wherein the input event is encoded as an ultrasonic tone.

4. The method of claim 1, wherein the output stream comprises a video stream.

5. The method of claim 4, wherein the input event is encoded as a watermark on the video stream.

6. The method of claim 4, wherein the input event is encoded as metadata in the video stream.

7. The method of claim 6, wherein the metadata is supplemental enhancement information for each video frame.

8. The method of claim 1, wherein the output stream comprises a series of output events, and the encoding of the input events is synchronized with the series of output events.

9. The method of claim 1, wherein the encoding of the input event is synchronized with an event occurring on a remote device via a network.

10. The method of claim 1, wherein the input event is a key press.

11. The method of claim 1, wherein the input event is a joystick control input.

12. The method of claim 1, wherein the input event is a steering wheel control input.

13. The method of claim 1, wherein the output stream having coded input is filtered to separate the coded input event from the output stream.

14. The method of claim 13, wherein the output stream and the encoded input event are provided to the neural network.

15. The method of claim 14, wherein the neural network is trained to associate the encoded input event with the specific event.

16. The method of claim 1, wherein the input event is configured to cause the specific event.

17. Instructions embedded in a non-transitory computer-readable medium, wherein the instructions, when executed, perform a method, the method comprising: a) Receive input events; b) Encoding the input events in an output stream, wherein the encoding of the input events is synchronized with a specific event occurring within the video game, wherein the output stream includes an audio stream, and wherein different types of input events are encoded by tones of different frequencies in the audio stream; as well as c) Couple the output stream to one or more neural networks, wherein the encoded input events in the output stream are imperceptible to the user when reproduced by an output device.

18. A system for providing synchronous input feedback, comprising: processor; Memory; as well as The non-transitory instructions in the memory, when executed, cause the processor to perform a method, the method comprising: a) Receive input events; b) Encoding the input events in an output stream, wherein the encoding of the input events is synchronized with specific events occurring within the video game, wherein the output stream includes an audio stream, and wherein different types of input events are encoded by tones of different frequencies in the audio stream; and c) Couple the output stream to one or more neural networks, wherein the encoded input events in the output stream are imperceptible to the user when reproduced by an output device.

19. A method for improved machine learning training, comprising: a) Receive an output stream comprising encoded input events synchronized with specific events occurring within a video game, wherein the encoded input events are undetectable by the user, wherein the output stream comprises an audio stream, and wherein different types of input events are encoded by tones of different frequencies in the audio stream; b) Filter the output stream to recover the encoded input event, wherein the encoded input event is synchronized with a specific event; as well as c) Train a neural network to associate the encoded input event with the specific event.

Images