Method and computing system for improved time processing of sequential data

Abstract

The invention relates to a method and a computing system for improved temporal processing of sequential data. Systems and methods may include or utilize a machine learning model (e.g., a convolutional neural network) that includes one or more temporal residual connections. In particular, each temporal residual connection may respectively supply one or more intermediate feature data sets generated from a current sequential input by a current instantiation of the model to one or more other instantiations of a machine learning model applied to process one or more other sequential inputs. For example, other instantiations of the machine learning model may include a subsequent instantiation of the machine learning model applied to one or more subsequent sequential inputs in the processing sequence that follow the current sequential input, and / or applied to previous instantiation of the machine learning model in the processing sequence that precedes one or more previous sequential inputs in the current sequential input.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 202180083068.9, filed on December 20, 2021.

[0003] Related applications

[0004] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 136,772, filed January 13, 2021. The entire contents of U.S. Provisional Patent Application No. 63 / 136,772 are incorporated herein by reference. Technical Field

[0005] This disclosure generally relates to machine learning for processing sequential data. More specifically, this disclosure relates to a machine learning model (e.g., a neural network, such as a convolutional neural network) characterized by one or more residual connections that pass intermediate feature data between different time instantiations of the model. Background Technology

[0006] Various machine learning tasks involve generating predictions or inferences from sequential data such as sequential image data, audio data (e.g., speech data), text data, sensor data, and so on. As an example, given a sequence of images from a video, the task of a video understanding model might be to predict frame-level or video-level outputs for tasks such as action recognition, object detection, or others.

[0007] In many cases, the ability to detect and understand temporal dependencies within sequential data itself is crucial for generating high-quality predictions. However, many typical existing model architectures do not contribute to this temporal understanding.

[0008] Other models that attempt to capture temporal information are inefficient or low-performance. For example, models based on 3D convolutions typically require a sequence of preprocessed image frames as input and attempt to process all images concurrently, resulting in large memory footprints (e.g., growing linearly with the number of input frames) and offline inference. As another example, some recursive models (e.g., those based on gated recursive units or long short-term memory units) are prone to training instability and poor performance. Summary of the Invention

[0009] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0010] One example aspect of this disclosure is a computational system for improved temporal processing of sequential data. The computational system includes one or more processors and one or more non-transitory computer-readable media collectively storing: a machine learning convolutional neural network including one or more temporal residual connections, which respectively supply one or more intermediate feature datasets generated from the current sequential input to one or more other instantiations of the machine learning convolutional neural network applied to process one or more other sequential inputs; and instructions that, when executed by the one or more processors, cause the computational system to perform operations. These operations include, for each of a plurality of sequential inputs included in the sequence: processing the current sequential input using at least a portion of the currently instantiated machine learning convolutional neural network to generate a current intermediate feature dataset; storing the current intermediate feature dataset to provide to one or more subsequent instantiations of the machine learning convolutional neural network applied to process one or more subsequent sequential inputs in the sequence; accessing one or more previous intermediate feature datasets generated by one or more previous instantiations of the machine learning convolutional neural network applied to process one or more previous sequential inputs preceding the current sequential input in the sequence; and generating a model output from the currently instantiated machine learning convolutional neural network based at least in part on the current intermediate feature dataset and the one or more previous intermediate feature datasets.

[0011] Another example aspect of this disclosure is directed to a computer-implemented method. The method includes, for each of a plurality of sequential inputs included in a sequence: processing the current sequential input using at least a portion of a currently instantiated machine learning convolutional neural network to generate a current intermediate feature dataset; storing the current intermediate feature dataset for provision to one or more subsequent instantiations of the machine learning convolutional neural network applied to process one or more subsequent sequential inputs in the sequence following the current sequential input, wherein the current instantiation of the machine learning convolutional neural network is connected to one or more subsequent instantiations of the machine learning convolutional neural network via one or more temporal residual connections; accessing one or more previous intermediate feature datasets generated by one or more previous instantiations of the machine learning convolutional neural network applied to process one or more previous sequential inputs preceding the current sequential input in the sequence, wherein the current instantiation of the machine learning convolutional neural network is connected to one or more previous instantiations of the machine learning convolutional neural network via one or more temporal residual connections; and generating a model output from the current instantiation of the machine learning convolutional neural network based at least in part on the current intermediate feature dataset and the one or more previous intermediate feature datasets.

[0012] Another example aspect of this disclosure relates to one or more non-transitory computer-readable media collectively storing: a machine learning convolutional neural network including one or more temporal residual connections, which respectively supply one or more intermediate feature datasets generated from the current sequential input to one or more other instantiations of the machine learning convolutional neural network applied to process one or more other sequential inputs; and computer-readable instructions for performing the machine learning convolutional neural network.

[0013] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0014] These and other features, aspects, and advantages of the various embodiments of this disclosure will become more readily understood upon reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0015] A detailed discussion of embodiments for those skilled in the art is set forth in the specification, with reference to the accompanying drawings, in which: Figure 1 A graph depicts a machine learning model according to an example embodiment of the present disclosure, the model including one or more temporal residual connections.

[0016] Figures 2A-2G A block diagram depicts an example machine learning model comprising one or more temporal residual connections according to an example embodiment of the present disclosure.

[0017] Figures 3A-3C A block diagram depicts an example time residual join operation according to an example embodiment of the present disclosure.

[0018] Figure 4A A block diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0019] Figure 4B A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0020] Figure 4C A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0021] Figure 5 A flowchart is depicted illustrating an example method employing a machine learning model according to an example embodiment of the present disclosure, the model including one or more temporal residual connections.

[0022] Repeated reference numerals across multiple figures are intended to identify the same features in different embodiments. Detailed Implementation

[0023] Overview

[0024] In general, this disclosure relates to systems and methods that include or fully utilize machine learning models (e.g., convolutional neural networks) comprising one or more temporal residual connections. Specifically, each temporal residual connection can supply one or more intermediate feature datasets generated from the current sequential input by the current model instantiation to one or more other instantiations of the machine learning model applied to process one or more other sequential inputs. For example, other instantiations of the machine learning model may include: subsequent instantiations of the machine learning model applied to process one or more subsequent sequential inputs following the current sequential input in a sequence, and / or previous instantiations of the machine learning model applied to process one or more previous sequential inputs preceding the current sequential input in a sequence. In this way, useful temporal information for modeling temporal dependencies between different parts of sequential data can be passed from one instantiation of the model to other instantiations, allowing the machine learning model to more effectively retain and process temporal information when generating predictions / inferences.

[0025] More specifically, the example machine learning models described in this paper can use temporal residual connections to receive and / or send intermediate feature data (e.g., intermediate activations from layers of a neural network—such as convolutional layers in a convolutional network) to neighboring instantiators of the model. Passing such information across temporal residual connections can generate effects that enhance or otherwise improve the feature extraction process for each instantiator of the model. In other words, temporal residual connections can be used to conjunct feature maps in time. Note that this differs from the standard residual skip connections widely used in deep neural networks, which connect feature maps along the depth of the neural network rather than across different temporal instantiators.

[0026] Therefore, in some example implementations, the model may include convolution operations. ,in Refers to time The input and the output of the convolution operation An example aspect of this disclosure is achieved by taking side input. and / or To extend this convolution operation, where Indicates time A set of one or more activation feature maps, and so on. The side inputs of this convolution operation, as OR, can be provided by residual connections in time. Such an output Not only includes time Information also includes information from time arrive The information, and in it It refers to combining in a certain way. and One or more operations. Furthermore, in some example implementations, such extended operations can be stacked to... Used in layers. In this case, the final feature map from the convolutional neural network will have a temporal receptive domain. .

[0027] Although the above examples are for convolution operations This is for discussion, but other operations can also be performed, which can be represented as ,in It is to apply the learned parameter values ​​to learn from generate Any operation. Furthermore, in some example implementations, the side input may also include subsequent feature data, which can be represented as... .

[0028] The information provided by temporal residual connections can be combined with existing information within the model in several different ways. The simplest residual connection method is addition, which can be achieved without rescaling if the input feature map at the current timestamp and the connection feature map at previous timestamps have the same shape.

[0029] Another form of residual connection is to concatenate feature maps along the channel dimension and then apply one or more convolutions to the concatenated feature maps. In some implementations, standard residual connections can also be used to stabilize training.

[0030] Another example of temporal concatenation is to first concatenate feature maps along the channel dimension, and then apply convolutions with different filter sizes and / or dilation rates in parallel on the concatenated feature maps. Standard residuals can also be used.

[0031] The systems and methods disclosed herein offer numerous technical effects and benefits. As an example, the proposed models can be easily deployed on CPUs and accelerators. The proposed temporal residual connections can also facilitate improved or competitive performance, even with significantly smaller models (e.g., with a significantly smaller number of parameters). Smaller models execute faster, require less storage, and consume less bandwidth during transmission. Therefore, by enabling smaller models to achieve improved performance, the techniques described herein achieve savings in computational resources, such as processor usage, memory usage, and network bandwidth usage. As an example, extensive experiments demonstrate that a MobilenetV3-based example model with temporal residual connections is 15 times smaller and 7 times more efficient than ResNet50-3D, offering competitive performance. In addition to saving computational resources, the proposed techniques can also improve the performance of the computing system itself when performing various sequential data processing tasks.

[0032] The sequential data processed by the model can include any form of sequential data. As examples, multiple sequential inputs included in a sequence can be or include: multiple image frames included in a video, multiple LiDAR datasets included in a light detection and ranging LiDAR data sequence, multiple parts of audio (e.g., represented by raw audio data or by spectrograms), multiple parts of text data (e.g., tags, sentences, etc.), multiple sensor datasets, and / or various other forms of sequential data. The tasks performed by the machine learning model can include a wide variety of tasks, including, for example, action recognition, object detection, translation, speech recognition, encoding / embedding, anomaly detection, time-event prediction, and / or various other tasks.

[0033] Example embodiments of this disclosure will now be discussed in further detail with reference to the figures.

[0034] Example machine learning models featuring temporal residual connections

[0035] Figure 1 An illustration depicts an example machine learning model including one or more temporal residual connections according to an example embodiment of this disclosure. In particular, Figure 1An example temporal residual MobileNet is shown, which includes a MobileNet backbone with temporal residual connections. For example, a temporal residual can be added in each inverted bottleneck block to achieve a strong speed-accuracy tradeoff. For a description of MobileNets, see Sandler et al., MobileNetV2: Inverted Residuals and Linear Bottlenecks, arXiv:1801.04381. MobileNet is just one example backbone that can be used; many other backbones can be used alternatively. Figure 1 The example model is illustrated to perform an example task, namely video analysis (e.g., action recognition). Other tasks can be performed alternatively.

[0036] More notably, temporal residuals can be conceptually analogous to residual connections widely used in deep neural networks, the difference being the connection along the time axis. Temporal residuals are more flexible than 3D convolutions because they can be applied to 2E convolutions, which are well optimized for devices.

[0037] Figure 1 An example model is shown where the temporal residual connections are unidirectional (e.g., in the illustrated example, the model only includes forward propagation temporal residual connections, which pass information to subsequent instantiations of the model). This example implementation characterized by unidirectional temporal connections does not require access to future frames to aggregate information. Therefore, the model can process streaming data in an online manner. Thus, unidirectional temporal residual connections can provide robust performance for online tasks—such as online video understanding. In particular, the model does not require access to future information and can be executed in real time, which is required by some example applications—such as robotics, autonomous driving, surveillance, etc.

[0038] As a further example, Figure 1 The example model illustrated in the diagram further includes a final Long Short-Term Memory (LSTM) layer. Therefore, in the last layer, a temporally enhanced feature map with temporal residual connections from MobileNet is input into the convolutional LSTM. The combination of MobileNet with temporal residual connections and LSTM provides the following benefits: the temporal residual connections aggregate short-range temporal dependencies, while the LSTM memorizes long-term context. This stacked structure expands the temporal receptive domain, which is crucial for video action recognition tasks.

[0039] Empirical analysis shows that adding a temporal residual immediately after strided convolution in one layer results in the optimal speed-accuracy tradeoff. An example implementation of MobileNetV2 with temporal residual connections is shown in Table 1 below.

[0040]

[0041] Table 1. Specifications for Temporal Residual MobilenetV3-Large. The T-Res Bneck represents the temporal residual enhancement bottleneck. By default, parallel atrous convolutions are used as the fusion operator. Empirical evidence suggests that inserting the T-Res Bneck (starting from C2) immediately after the strided convolution achieves the optimal speed-accuracy tradeoff.

[0042] Extensive experiments on multiple datasets demonstrate that the proposed mobile model achieves highly competitive performance compared to 3D convolution-based models, with a 10x reduction in memory footprint and computational cost. On the Kinetics600 dataset, the state-of-the-art MobileNetV2-based model with temporal residual connections achieves 74.57% Top-1 accuracy, compared to 74.6% for ResNet50-3D. On the Moments-in-Time dataset, the proposed model achieves 30.25% Top-1 accuracy, compared to 31.41% for the state-of-the-art AssembleNet-50.

[0043] Figure 1 An example model is illustrated, characterized by temporal residual connections. However, the techniques disclosed herein are not limited to these. Figure 1 The specific example model shown is illustrated below. As a further example, Figures 2A-2G A block diagram depicts an example machine learning model comprising one or more temporal residual connections according to an example embodiment of the present disclosure.

[0044] like Figure 2A As shown, in some implementations, one or more temporal residual connections may include only a single temporal residual connection in each instantiation. However, as Figure 2B As shown, in some implementations, multiple temporal residual connections can exist at different corresponding depths within the machine learning model. Any number of temporal residual connections can be used.

[0045] like Figure 2A and Figure 2BAs shown, in some implementations, the machine learning model may only include a forward propagation temporal residual connection that feeds one or more intermediate feature datasets generated from the current sequential input to subsequent instantiations of a machine learning convolutional neural network applied to process subsequent sequential inputs following the current input. However, as Figure 2C As shown, in some implementations, the machine learning model may include both: a forward propagation temporal residual connection that supplies one or more intermediate feature datasets generated from the current sequential input to subsequent instantiations of the machine learning convolutional neural network; and a backward propagation temporal residual connection that supplies one or more intermediate feature datasets generated from the current sequential input to a previous instantiation of the machine learning convolutional neural network applied to process previous sequential inputs preceding the current sequential input. Bidirectional connections can be in different layers, such as... Figure 2C As shown. As another example, such as Figure 2D As shown, time connections can be bidirectional on the same layer.

[0046] like Figures 2A-2D As shown, in some implementations, some or all of one or more temporal residual connections can be configured to feed one or more intermediate feature datasets to one or more other instantiated identical layers of a machine learning convolutional neural network. However, as Figure 2E As shown, in some implementations, some or all of one or more temporal residual connections can be configured to feed one or more intermediate feature datasets to one or more other instantiated different layers of a machine learning convolutional neural network.

[0047] like Figures 2A-2E As shown, in some implementations, each time residual connection can be connected to a unique location in the model. However, as Figure 2F As shown, in some implementations, one or more temporal residual connections can be connected to shared locations in the model. For example, as Figure 2F As shown, each instantiation of the model has two forward propagation temporal residual connections that start from a shared location but arrive at different locations in the next sequential instantiation of the model.

[0048] like Figures 2A-2F As shown, in some implementations, each temporal residual connection can be connected to an adjacent instantiation of the model (e.g., the next sequential instantiation of the model applied to the next sequential input in the processing sequence and / or the previous sequential instantiation of the model applied to the previous sequential input in the processing sequence). However, as Figure 2GAs shown, in some implementations, each temporal residual connection can be connected to a non-adjacent instantiation of the model (e.g., an instance of the model that is greater than the next sequential input in the processing sequence and / or an instance of the model that is less than the previous sequential input in the processing sequence).

[0049] The information provided by temporal residual connections can be combined with existing information within the model in a variety of different ways. As an example, Figures 3A-3C A block diagram is described for an example time residual join operation according to an example implementation of this disclosure.

[0050] The simplest form of residual connection is addition, which can be implemented without readjustment if the input feature map at the current timestamp and the connection feature map from previous timestamps have the same shape. Figure 3A An example addition join is shown.

[0051] Another form of residual connection involves concatenating feature maps along the channel dimension and then applying one or more convolutions to the concatenated feature maps. In some implementations, standard residual connections can also be used to stabilize training. Figure 3B The example connection and convolution are illustrated in the diagram.

[0052] Another example of temporal concatenation is to first concatenate feature maps along the channel dimension, and then apply convolutions with different filter sizes and / or dilation rates in parallel on the concatenated feature maps. Standard residuals can also be used. Figure 3C An example illustration of one implementation of this technology is shown. Figure 3C The multiple convolutions shown can have different corresponding filter sizes and / or different corresponding dilation rates. Having different filter sizes and / or different dilation rates can help the model understand and handle dependencies in data whose size changes over time. For example, from the... t Frame to the t +1 frame, the size of the person walking towards the camera will increase (e.g., in terms of the corresponding number of pixels). Having different filter sizes and / or different dilation rates can help the model handle changes in the size of the person.

[0053] Example devices and systems

[0054] Figure 4A A block diagram of an example computing system 100 according to an exemplary embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0055] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0056] User computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118, which are executed by processor 112 to cause user computing device 102 to perform operations.

[0057] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be, or may otherwise include, various machine learning models such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some example machine learning models may fully utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., transformer models). (See reference...) Figures 1-3C Example machine learning model 120 is discussed.

[0058] In some implementations, one or more machine learning models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instantiations of a single machine learning model 120 (e.g., performing parallel processing of multiple sequential inputs).

[0059] Additionally or alternatively, one or more machine learning models 140 may be included in, or otherwise stored in and implemented by, a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 140 as part of a web service. Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0060] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which the user can provide input.

[0061] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138, which are executed by processor 132 to cause server computing system 130 to perform operations.

[0062] In some implementations, the server computing system 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. In an instantiation of the server computing system 130 including multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0063] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be, or may otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may fully utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., transformer models). (See reference...) Figures 1-3C Example model 140 was discussed.

[0064] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130, or it may be part of server computing system 130.

[0065] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158, which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0066] The training computing system 150 may include a model trainer 160 that trains machine learning models 120 and / or 140 stored at the user computing device 102 and / or server computing system 130 using various training or learning techniques, such as backpropagation of errors. For example, a loss function can be used to update one or more parameters of the model through model backpropagation (e.g., gradients based on the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or other various loss functions. Gradient descent techniques can be used to iteratively update parameters over a number of training iterations.

[0067] In some implementations, performing backpropagation of errors may include performing truncated backpropagation over time. The model trainer 160 may perform generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0068] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. In some implementations, training examples can be provided by user computing device 102 if the user has provided consent. Therefore, in such an implementation, model 120 provided to user computing device 102 can be trained by training computing system 150 based on user-specific data received from user computing device 102. In some instances, this process can be referred to as model personalization.

[0069] Model trainer 160 includes computer logic for providing the desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more computer-executable instruction sets stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0070] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication over Network 180 can be conducted via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0071] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0072] In some implementations, the input to the machine learning model of this disclosure can be image data. The machine learning model can process the image data to generate output. As an example, the machine learning model can process image data to generate image recognition output (e.g., image data identification, latent embedding of image data, encoded representation of image data, hashing of image data, etc.). As another example, the machine learning model can process image data to generate image segmentation output. As another example, the machine learning model can process image data to generate image classification output. As another example, the machine learning model can process image data to generate image data modification output (e.g., image data alteration, etc.). As another example, the machine learning model can process image data to generate encoded image data output (e.g., encoded and / or compressed representation of image data, etc.). As another example, the machine learning model can process image data to generate scaled-up image data output. As another example, the machine learning model can process image data to generate predictive output.

[0073] In some implementations, the input to the machine learning model of this disclosure can be text or natural language data. The machine learning model can process the text or natural language data to generate output. As an example, the machine learning model can process natural language data to generate language-encoded output. As another example, the machine learning model can process text or natural language data to generate latent text embedding output. As another example, the machine learning model can process text or natural language data to generate translation output. As another example, the machine learning model can process text or natural language data to generate classification output. As another example, the machine learning model can process text or natural language data to generate text segmentation output. As another example, the machine learning model can process text or natural language data to generate semantic intent output. As another example, the machine learning model can process text or natural language data to generate scaled-up text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model can process text or natural language data to generate predictive output.

[0074] In some implementations, the input to the machine learning model of this disclosure may be speech data. The machine learning model can process the speech data to generate an output. As an example, the machine learning model can process speech data to generate a speech recognition output. As another example, the machine learning model can process speech data to generate a speech translation output. As another example, the machine learning model can process speech data to generate a latent embedding output. As another example, the machine learning model can process speech data to generate an encoded speech output (e.g., encoded and / or compressed representations of speech data, etc.). As another example, the machine learning model can process speech data to generate an upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model can process speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model can process speech data to generate a predicted output.

[0075] In some implementations, the input to the machine learning model of this disclosure can be latent encoded data (e.g., a latent space representation of the input). The machine learning model can process the latent encoded data to generate an output. As an example, the machine learning model can process the latent encoded data to generate an identification output. As another example, the machine learning model can process the latent encoded data to generate a reconstruction output. As another example, the machine learning model can process the latent encoded data to generate a search output. As another example, the machine learning model can process the latent encoded data to generate a re-aggregation output. As yet another example, the machine learning model can process the latent encoded data to generate a prediction output.

[0076] In some implementations, the input to the machine learning model of this disclosure may be statistical data. The machine learning model can process the statistical data to generate output. As an example, the machine learning model can process the statistical data to generate an identification output. As another example, the machine learning model can process the statistical data to generate a prediction output. As another example, the machine learning model can process the statistical data to generate a classification output. As another example, the machine learning model can process the statistical data to generate a segmentation output. As another example, the machine learning model can process the statistical data to generate a visualization output. As yet another example, the machine learning model can process the statistical data to generate a diagnostic output.

[0077] In some implementations, the input to the machine learning model of this disclosure can be sensor data. The machine learning model can process the sensor data to generate output. As an example, the machine learning model can process sensor data to generate an identification output. As another example, the machine learning model can process sensor data to generate a prediction output. As another example, the machine learning model can process sensor data to generate a classification output. As another example, the machine learning model can process sensor data to generate a segmentation output. As another example, the machine learning model can process sensor data to generate a visualization output. As another example, the machine learning model can process sensor data to generate a diagnostic output. As another example, the machine learning model can process sensor data to generate a detection output.

[0078] In some cases, machine learning models can be configured to perform tasks involving encoding (and / or corresponding decoding) input data to achieve reliable and / or efficient transmission or storage. For example, this task could be an audio compression task. The input might include audio data, and the output might include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In yet another example, the task could include generating embeddings for input data (e.g., input audio or visual data).

[0079] In some cases, the input includes visual data, and the task is a computer vision task. In other cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, an image processing task could be image classification, where the output is a set of scores, each corresponding to a different object class, and representing the likelihood that one or more images depict objects belonging to that object class. An image processing task could be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, the region depicts the likelihood of an object of interest. As another example, an image processing task could be image segmentation, where the image processing output defines a corresponding likelihood for each class in a predetermined set of classes for each pixel in one or more images. For example, this set of classes could be foreground and background. As another example, the set of classes could be object classes. As another example, an image processing task could be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, an image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at pixels between the images in the network input for each pixel in one of the input images.

[0080] In some cases, the input includes audio data representing spoken utterance, and the task is a speech recognition task. The output may include text output mapped to the spoken utterance. In some cases, the task involves encrypting or decrypting the input data. In some cases, the task involves microprocessor performance tasks such as branch prediction or memory address translation.

[0081] Figure 4A The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems may also be used. For example, in some embodiments, user computing device 102 may include a model trainer 160 and a training dataset 162. In such embodiments, model 120 can be both trained on user computing device 102 and used locally. In some such embodiments, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0082] Figure 4B A block diagram depicts an example computing device 10 implemented according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0083] The computing device 10 includes several applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0084] like Figure 4B As shown, each application can communicate with several other components of the computing device, such as, for example, one or more sensors, a scene manager, a device status component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0085] Figure 4C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0086] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0087] The central intelligence layer includes several machine learning models. For example, such as... Figure 4C As shown, a corresponding machine learning model can be provided for each application and managed by a central intelligent layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligent layer can provide a single model for all applications. In some embodiments, the central intelligent layer is included within the operating system of computing device 50 or otherwise implemented by the operating system of computing device 50.

[0088] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing device 50. For example... Figure 4C As shown, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a field manager, a device status component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a proprietary API).

[0089] Example Method

[0090] Figure 5 A flowchart depicts an example method for improving time processing of sequential data according to an example embodiment of this disclosure. Although Figure 5 For illustrative and discussion purposes, the steps performed in a particular order are described, but the method of this disclosure is not limited to the specifically described order or arrangement. Various steps of method 500 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of this disclosure.

[0091] In some implementations, method 500 can be performed iteratively on several sequential inputs. In some instances, if multiple temporal residual connections exist, steps 504 and 506 can be performed multiple times in each iteration.

[0092] In 502, the computing system can utilize at least a portion of the currently instantiated machine learning model to process the current sequential input in order to generate the current intermediate feature dataset.

[0093] In 504, the computing system may store the current intermediate feature dataset to provide one or more previous and / or subsequent instantiations of a machine learning model that is applied to the current sequential input preceding one or more previous sequential inputs and / or the current sequential input following one or more subsequent sequential inputs in the processing sequence.

[0094] In 506, the computing system can access one or more previous and / or subsequent intermediate feature datasets generated by one or more previous and / or subsequent instantiations of a machine learning model applied to one or more previous sequential inputs preceding the current sequential input in the processing sequence and / or one or more subsequent sequential inputs following the current sequential input.

[0095] In 508, the computing system can generate model output from the current instantiation of the machine learning model, at least in part, based on the current intermediate feature dataset and one or more previous and / or subsequent intermediate feature datasets.

[0096] After 508, method 500 may optionally return to 502 to process the next sequential input. Alternatively, multiple instances of method 500 may be executed in parallel to process multiple sequential inputs concurrently.

[0097] Additional description

[0098] This article discusses technologies including servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from these systems. The inherent flexibility of computer-based systems allows for a vast number of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed in this article can be implemented using a single device or component, or a combination of multiple devices or components. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0099] While this subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation and not as a limitation of this disclosure. Those skilled in the art, upon understanding the foregoing, can readily generate changes, variations, and equivalents to such embodiments. Therefore, this disclosure does not preclude modifications, variations, and / or additions to this subject matter, as such modifications, variations, and / or additions will be apparent to those skilled in the art. For example, features illustrated or described as part of an embodiment may be used with another implementation to produce further embodiments. Therefore, this disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computing system for improved time processing of sequential data, the computing system comprising: One or more processors; as well as One or more non-transitory computer-readable media, wherein the one or more non-transitory computer-readable media centrally store: A machine learning convolutional neural network, comprising one or more temporal residual connections, each of which supplies one or more intermediate feature datasets generated from the current sequential input to one or more other instantiations of the machine learning convolutional neural network applied to process one or more other sequential inputs, wherein the machine learning convolutional neural network comprises one or more convolutional layers followed by long short-term memory layers; and Instructions, which, when executed by the one or more processors, cause the computing system to perform operations, the operations including: For each of the multiple sequential inputs included in the sequence: The current sequential input is processed using at least a portion of the currently instantiated machine learning convolutional neural network to generate the current intermediate feature dataset; Access to one or more previously intermediate feature datasets generated by one or more previously instantiated machine learning convolutional neural networks applied to process one or more previously sequential inputs preceding the current sequential input in the sequence; and The model output is generated from the current instantiation of the machine learning convolutional neural network, based at least in part on the current intermediate feature dataset and the one or more previous intermediate feature datasets.

2. The computing system according to claim 1, further comprising: The current intermediate feature dataset is stored for use in one or more subsequent instantiations of the machine learning convolutional neural network applied to process one or more subsequent sequential inputs following the current sequential input.

3. The computing system according to claim 2, wherein, The machine learning convolutional neural network consists only of forward propagation temporal residual connections, which feed the one or more intermediate feature datasets generated from the current sequential input to the subsequent instantiation of the machine learning convolutional neural network.

4. The computing system according to claim 2, wherein, The machine learning convolutional neural network includes the following two: A forward propagation temporal residual connection, which supplies the one or more intermediate feature datasets generated from the current sequential input to the subsequent instantiation of the machine learning convolutional neural network; as well as A backward propagation temporal residual connection that supplies the one or more intermediate feature datasets generated from the current sequential input to the previously instantiated machine learning convolutional neural network.

5. The computing system according to claim 1, wherein, At least one of the one or more temporal residual connections is configured to feed the one or more intermediate feature datasets to the one or more other instantiated identical or different layers of the machine learning convolutional neural network.

6. The computing system according to claim 1, wherein, The one or more temporal residual connections include multiple temporal residual connections at different corresponding depths existing within the machine learning convolutional neural network.

7. The computing system according to claim 1, wherein, Generating the model output from the current instantiation of the machine learning convolutional neural network, at least in part based on the current intermediate feature dataset and the one or more previous intermediate feature datasets, includes: Combine at least one of the previous intermediate feature datasets with at least one existing feature dataset to form a combined feature dataset; and The model output is generated from the current instantiation of the machine learning convolutional neural network, based at least in part on the feature dataset of the combination.

8. The computing system according to claim 7, wherein, Combining at least one of the previous intermediate feature datasets with at least one existing feature dataset to form the combined feature dataset includes summing at least one of the previous intermediate feature datasets with at least one existing feature dataset.

9. The computing system according to claim 7, wherein, The combination of at least one of the previous intermediate feature datasets with at least one existing feature dataset to form the combined feature dataset includes: Link at least one of the previous intermediate feature datasets to at least one existing feature dataset; and Apply one or more convolutions to the connected data.

10. The computing system according to claim 7, wherein, The combination of at least one of the previous intermediate feature datasets with at least one existing feature dataset to form the combined feature dataset includes: Link at least one of the previous intermediate feature datasets with at least one existing feature dataset; Multiple convolutional filters are applied in parallel to the concatenated data; and Combine the outputs of the plurality of convolutional filters.

11. The computing system according to claim 10, wherein, The plurality of convolutional filters have different corresponding filter sizes and / or different corresponding dilation rates.

12. The computing system according to claim 7, wherein, The at least one existing feature dataset includes the current intermediate feature dataset.

13. The computing system according to claim 2, wherein, The one or more subsequent instantiations of the machine learning convolutional neural network applied to process the one or more subsequent sequential inputs include the next sequential instantiation of the machine learning convolutional neural network applied to process the next sequential input in the sequence.

14. The computing system according to claim 2, wherein, The one or more subsequent instantiations of the machine learning convolutional neural network applied to process the one or more subsequent sequential inputs include the next-next-sequence instantiations of the machine learning convolutional neural network applied to process the sequence of inputs greater than the next sequential input.

15. The computing system according to claim 1, wherein, The current intermediate feature dataset includes activation maps of the convolutional layers used in the machine learning convolutional neural network.

16. The computing system according to claim 1, wherein, The one or more temporal residual connections exist in the one or more convolutional layers.

17. The computing system according to claim 1, wherein, The plurality of sequential inputs included in the sequence include a plurality of image frames included in the video and / or a plurality of LiDAR datasets included in the light detection and ranging LiDAR data sequence.

18. The computing system according to any one of claims 1-17, wherein, The machine learning convolutional neural network is configured to perform tasks, including action recognition or object detection.

19. A computer-implemented method, the method comprising: For each of the multiple sequential inputs included in the sequence: At least a portion of the current instantiation of the machine learning convolutional neural network is used to process the current sequential input to generate the current intermediate feature dataset, wherein the current instantiation of the machine learning convolutional neural network is connected to one or more subsequent instantiations of the machine learning convolutional neural network via one or more temporal residual connections, and the machine learning convolutional neural network includes one or more convolutional layers followed by long short-term memory layers. Access one or more previously intermediate feature datasets generated by one or more previously instantiated instances of the machine learning convolutional neural network applied to process one or more previously sequential inputs preceding the current sequential input in the sequence, wherein the current instantiation of the machine learning convolutional neural network is connected to the one or more previously instantiated instances of the machine learning convolutional neural network via the one or more temporal residual connections; and The model output is generated from the current instantiation of the machine learning convolutional neural network, based at least in part on the current intermediate feature dataset and the one or more previous intermediate feature datasets.

20. One or more non-transitory computer-readable media, wherein the one or more non-transitory computer-readable media centrally store: A machine learning convolutional neural network, comprising one or more temporal residual connections, wherein the one or more temporal residual connections respectively supply one or more intermediate feature datasets generated from the current sequential input to one or more other instantiations of the machine learning convolutional neural network applied to process one or more other sequential inputs, wherein, The machine learning convolutional neural network includes one or more convolutional layers, followed by long short-term memory layers; and Computer-readable instructions for executing the machine learning convolutional neural network.

Images