Computing system, semiconductor apparatus, computer readable storage medium, and method for mage sequence or video analysis

Abstract

Techniques for performing image sequence / video analysis may include a processor and a memory coupled to the processor, the memory storing a neural network comprising a plurality of convolutional layers, a network depth relay structure comprising a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and a feature dimension relay structure comprising a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers. Each network depth calibration layer is coupled to the previous network depth calibration layer via a first hidden state signal and a cell state signal, and each feature dimension calibration slice is coupled to the previous feature dimension calibration slice via a second hidden state and a cell state signal.

Description

Technical Field

[0001] Embodiments in the field of invention generally relate to computing systems. More specifically, embodiments relate to performance-enhancing deep learning techniques that utilize convolutional neural networks for image sequence / video analysis. Prior Technology

[0002] Background of the Invention: Deep learning networks, such as convolutional neural networks (CNNs), have become an important candidate technology for use in image sequence / video analysis tasks, including graphics-related tasks such as video rendering, video action recognition, video ray tracing, and so on. Unlike two-dimensional (2D) CNNs, which perform convolution and pooling operations only in space, three-dimensional (3D) CNNs are constructed using 3D convolution and 3D pooling operations performed in spacetime. However, the use of 3D CNNs presents significant challenges in applications. For example, on the one hand, the increase in the dimension of the input data significantly leads to more complex feature distribution variations. On the other hand, compared to 2D CNNs, the model size of 3D CNNs has the potential to increase by one cubic degree. These factors result in huge memory and computational requirements for 3D CNN architectures (from both the data and model perspectives), making the use of 3D CNNs more difficult than for tasks based on 2D CNNs, thus effectively hindering the use of generalized 3D CNN architectures for high-performance image sequence / video analysis. Summary of the Invention

[0003] According to one embodiment of the present invention, a computing system for image sequence or video analysis is specifically proposed, comprising: a processor; and a memory coupled to the processor, the memory storing a neural network comprising: a plurality of convolutional layers; a network depth relay structure comprising a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and a feature dimension relay structure comprising a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers. Simple Explanation of the Diagram

[0004] The various advantages of these embodiments will become apparent to those skilled in the art from the following description and appended claims, and from the following accompanying drawings, in which: Figures 1A-1B provide schematic diagrams according to one or more embodiments, illustrating an overview of an example system for image sequence / video analysis; Figures 2A-2D provide example diagrams of neural network structures according to one or more embodiments; Figure 3A provides a block diagram according to one or more embodiments of an example network depth calibration structure for a neural network; Figure 3B is a schematic diagram illustrating an example of a network depth calibration layer for a neural network according to one or more embodiments; Figures 3C-3D are schematic diagrams illustrating an example of a unary gated relay (MGR) unit for a network depth calibration layer for a neural network according to one or more embodiments; Figure 4A provides a block diagram according to one or more embodiments of an example feature dimension calibration structure for a neural network; Figure 4B is a schematic diagram illustrating an example of a feature dimension calibration slice for a neural network according to one or more embodiments; Figures 4C-4D are schematic diagrams illustrating, according to one or more embodiments, an example of an MGR unit for calibrating a feature dimension slice of a neural network; Figures 5A-5B are flowcharts illustrating, according to one or more embodiments, an example of a method for constructing a neural network; Figures 6A-6F are illustrations of, according to one or more embodiments, inputting image sequences and corresponding activation maps in a system for image sequence / video analysis; Figure 7 is a block diagram illustrating, according to one or more embodiments, an example of a computing system for image sequence / video analysis; Figure 8 is a block diagram illustrating, according to one or more embodiments, an example of a semiconductor device; Figure 9 is a block diagram illustrating, according to one or more embodiments, an example of a processor; and Figure 10 is a block diagram illustrating, according to one or more embodiments, an example of a multiprocessor-based computing system. Implementation

[0005] Detailed Description of Preferred Embodiments A performance-enhancing computing system, as described herein, improves the performance of CNNs for image sequence / video analysis, particularly 3D CNNs. This technique improves the overall performance of the deep learning computing system from the perspective of feature representation calibration and fusion through Sample Adaptive Feature Calibration and Joint Agent (SA-FCAA). The SA-FCAA technique described herein can be applied to any deep CNN—particularly 3D CNNs—to provide a significant performance boost for image sequence / video analysis tasks in at least two ways. First, the SA-FCAA technique described herein is sample-specific; it calibrates a given 3D feature map not only using statistics conditioned on a current input instance but also using statistics from feature maps from adjacent convolutional layers and adjacent feature slices along an additional dimension—which may typically be a temporal dimension. Second, the SA-FCAA technique fusion of the calibrated 3D feature maps along two orthogonal dimensions via a shared lightweight meta-gated relay unit. By employing these dynamic learning and cross-layer relay capabilities—including the joint integration of calibrated features along a network depth and a feature dimension—this technique amplifies the joint spatiotemporal feature learning capabilities of 3D CNNs, thereby significantly improving the inference accuracy and training speed of 3D CNNs.

[0006] Figures 1A-1B provide schematic diagrams illustrating an overview of an example of a system 100 for image sequence / video analysis according to one or more embodiments. Figures 1A-1B reference the components and features described herein, including but not limited to the figures and associated descriptions. The system 100 includes a neural network 110 arranged as described herein, which incorporates a sample adaptation mechanism that dynamically generates calibration parameters conditioned on an input feature map to overcome potentially inaccurate calibration statistic estimations in CNNs such as 3D CNNs under limited batch size settings. The neural network 110 may be a CNN, such as a 3D CNN, which includes a plurality of convolutional layers 120. In some embodiments, the neural network 110 may include other types of neural network architectures. As shown in Figure 1A, the neural network 110 further includes a unary gated relay (MGR) structure 130 to jointly link the calibrated feature maps across two orthogonal dimensions, such as time and network depth, to amplify the spatiotemporal dependency modeling of 3D features in the 3D CNN. The MGR structure 130 may include a network depth relay structure 132 and a feature dimension relay structure 134, each of which will be further described below.

[0007] The neural network 110 receives an image sequence 140 as input. The image sequence 140 may include, for example, a video consisting of an image sequence associated with a time period. The neural network 110 generates an output feature map 150. The output feature map 150 represents the results of processing the input image sequence 140 by the neural network 110, and the results may include classification, detection, and / or segmentation of objects, features, etc., from the input image sequence 140.

[0008] As shown in Figure 1B, the convolutional layers 120 and the MGR structure 130 of the neural network 110 can be (at least partially) arranged in modules. Figure 1B depicts three modules: module (k-1), module (k), and module (k+1). Although three modules are shown in Figure 1B, it will be understood that the convolutional layers 120 and the MGR structure 130 of the neural network 110 can be (at least partially) arranged in one or fewer modules. Further details about the neural network 110 will be provided herein with reference to Figures 2A-2D, 3A-3D, 4A-4D, and 5A-5B.

[0009] Figure 2A provides a schematic diagram of an example of a neural network structure 200 according to one or more embodiments. Figure 2A refers to the components and features described herein, including but not limited to the figures and associated descriptions. The neural network structure 200 can be used in the neural network 110 (Figures 1A-1B, already discussed). The neural network structure 200 may include a plurality of modules, including a module 210, a module 220, and a module 230. Modules 210, 220, and 230 are indicated by module numbers ranging from (k-1), to (k), and to (k+1), respectively. Each module may include multiple layers, including one or more convolutional layers, a network depth calibration layer (denoted as "FCAA-D"), and a feature dimension calibration layer (denoted as "FCAA-T"). Furthermore, one or more modules in the neural network structure 200 may include one or more selectable activation layers (shown in dashed lines), and / or one or more additional / selectable layers such as convolutional layers, regularization layers, etc. (shown in dashed lines); other selectable neural network layers may also be included in a module.

[0010] Each network depth calibration layer (FCAA-D) typically follows a convolutional layer, and similarly, each feature dimension calibration layer (FCAA-T) typically follows another convolutional layer. Furthermore, the network depth calibration layers are arranged in a cross-module network depth relay structure, such that a network depth calibration layer in one module receives a hidden state signal and a cell state signal from a network depth calibration layer in the preceding module. Thus, for example, the network depth calibration layer in module (k+1) receives a hidden state signal hk and a cell state signal ck from a network depth calibration layer in module (k), the network depth calibration layer in module (k) receives a hidden state signal hk-1 and a cell state signal ck-1 from a network depth calibration layer in module (k-1), and so on, extending back to the initial module in the neural network that has a network depth calibration layer (for such an initial module, there will be no preceding module with a network depth calibration layer).

[0011] Although three modules are illustrated in Figure 2A, it will be understood that the number of modules in the neural network structure 200 may be greater than or less than three. The neural network structure 200 can be inserted into any neural network (such as neural network 110), particularly into a 3D CNN, at virtually any location within that neural network. The neural network structure 200 receives input (not shown in Figure 2A), which can, for example, come from any part of neural network 110, and provides output that will be used at any part of neural network 110. In some embodiments, the neural network structure 200 can be inserted at multiple points within the neural network. In some embodiments, the neural network structure 200 may include residual modules for use in a neural network. Further details regarding modules such as module 210, module 220, and / or module 230 are provided herein with reference to Figures 2B-2D.

[0012] Figure 2B provides a schematic diagram 240 of an example module 220 for use in the neural network structure 200 according to one or more embodiments. The diagram references the components and features described herein, including but not limited to the figures and associated descriptions. Module 220 represents a module (k) and corresponds to module 220 (Figure 2A). The structure shown for module 220 can also be applied to other modules (such as modules 210 and / or 230 in Figure 2A). Module 220 includes a first convolutional layer 221, a network depth calibration layer (FCAA-D) 222, a second convolutional layer 224, and a feature dimension calibration layer (FCAA-T) 225. The network depth calibration layer 222 follows the first convolutional layer 221, and the feature dimension calibration layer 225 follows the second convolutional layer 224. In some embodiments, the order of the network depth calibration layer 222 and the feature dimension calibration layer 225 can be reversed, such that the feature dimension calibration layer 225 follows the first convolutional layer 221 and the network depth calibration layer 222 follows the second convolutional layer 224.

[0013] The network depth calibration layer 222 used in module (k) receives a hidden state signal hk-1 and a cell state signal ck-1 from a network depth calibration layer in the previous module (k-1), and passes a hidden state signal hk and a cell state signal ck to a network depth calibration layer in a subsequent module (k+1). Module 220 may also include one or more optional activation layers, such as activation layer 223 following the network depth calibration layer 222, and / or activation layer 226 following the feature dimension calibration layer 225. Each of the activation layers 223 and / or 226 may include an activation function useful for CNNs, such as, for example, a rectified linear unit (ReLU) function, a SoftMax function, etc. Module 220 may also include other additional, optional layers such as, for example, additional convolutional, normalization, and / or activation layers (in whole labeled 227 in FIG. 2B). The module 220 receives input from a preceding module or another part of the neural network 110 and provides output to a subsequent module or another part of the neural network 110.

[0014] Figure 2C provides a schematic diagram 260 of another example module 270 for use in the neural network structure 200 according to one or more embodiments. The figure refers to the components and features described herein, including but not limited to the figures and associated descriptions. Module 270 represents a module (k) and may replace module 220 (Figures 2A-2B). The structure shown for module 270 may also replace other modules (such as modules 210 and / or 230 in Figure 2A). Module 270 includes a convolutional layer 271 and a network depth calibration layer (FCAA-D) 272 following the convolutional layer 271. The network depth calibration layer 272 for module (k) receives a hidden state signal and a cell state signal from a network depth calibration layer in the preceding module (k-1) and transmits a hidden state signal and a cell state signal to a network depth calibration layer in a subsequent module (k+1). The module 270 may also include an optional activation layer, such as activation layer 273, which follows the network depth calibration layer 272. Activation layer 273 may include an activation function useful for CNNs, such as, for example, a rectified linear unit (ReLU) function, a softmax function, etc. The module 270 may also include other additional, optional layers such as, for example, additional convolutional, normalization, and / or activation layers (labeled 274 in Figure 2C). The module 270 receives input from a preceding module or another part of the neural network 110 and provides output to a subsequent module or another part of the neural network 110.

[0015] Figure 2D provides a schematic diagram 280 of another example module 290 for use in the neural network structure 200 according to one or more embodiments. The diagram references the components and features described herein, including but not limited to the figures and associated descriptions. Module 290 represents a module (k) and may replace module 220 (Figures 2A-2B). The structure shown for module 290 may also replace other modules (such as modules 210 and / or 230 in Figure 2A). Module 290 includes a convolutional layer 291 and a feature dimension calibration layer (FCAA-T) 292 following the convolutional layer 291. Module 290 may also include an optional activation layer, such as activation layer 293, following the feature dimension calibration layer 292. Activation layer 293 may include an activation function useful for CNNs, such as, for example, a rectified linear unit (ReLU) function, a softmax function, etc. The module 290 may also include other additional, optional layers such as, for example, additional convolutional, normalization, and / or activation layers (labeled 294 in Figure 2D). The module 290 receives input from a preceding module or another part of the neural network 110 and provides output to a subsequent module or another part of the neural network 110.

[0016] Figure 3A is a block diagram illustrating an example of a network depth calibration structure 300 according to one or more embodiments. The diagram references the components and features described herein, including but not limited to the figures and associated descriptions. The network depth calibration structure 300 can be used in all or part of the neural network 110 (Figures 1A-1B, already discussed). The network depth calibration structure 300 includes a plurality of convolutional layers, including a convolutional layer 302 (representing a module k-1), a convolutional layer 304 (representing a module k), and a convolutional layer 306 (representing a module k+1). The convolutional layer 302 operates to provide an output feature map xk-1. Similarly, the convolutional layer 304 operates to provide an output feature map xk, and the convolutional layer 306 operates to provide an output feature map xk+1. These convolutional layers (such as convolutional layer 302, convolutional layer 304, and convolutional layer 306) correspond to convolutional layers 120 (Figures 1A-1B, already discussed) and / or correspond to one or more convolutional layers shown in Figure 2A, and have parameters and weights determined through a neural network training process. Convolutional layer 304 corresponds to convolutional layer 221 in Figure 2B.

[0017] The network depth calibration structure 300 further includes a plurality of network depth calibration layers (FCAA-D) arranged in a cross-module network depth relay structure 310, including a network depth calibration layer 312 (for module k-1), a network depth calibration layer 314 (for module k), and a network depth calibration layer 316 (for module k+1). Each network depth calibration layer is coupled to and follows a separate convolutional layer among the plurality of convolutional layers, such that each network depth calibration layer receives an input from the separate convolutional layer and provides an output to a subsequent layer. Each network depth calibration layer (i.e., each network depth calibration layer after an initial network depth calibration layer in the neural network) is also coupled to the network depth calibration layer in the respective preceding module via a hidden state signal and a cell state signal received from a network depth calibration layer of a respective preceding module. Therefore, as shown in this example of Figure 3A, the cross-module relay structure includes a network depth calibration layer for each module (k) that is coupled to a network depth calibration layer for the preceding module (k-1). The network depth relay structure 310 corresponds to the network depth relay structure 132 (shown in Figure 1, already discussed).

[0018] For example, the network depth calibration layer 312 (for module k-1) receives the feature map xk-1 as input from the convolutional layer 302. The network depth calibration layer 312 also receives a hidden state signal and a cell state signal from a network depth calibration layer in the previous module (not shown in FIG. 3A), unless the network depth calibration layer 312 is the initial network depth calibration layer in the neural network (in which case there would be no network depth calibration layer in the previous module). The network depth calibration layer 312 produces an output feature map yk-1. As shown in this example of FIG. 3A, the output yk-1 can be fed into a subsequent module (e.g., module (k)) or another neural network layer.

[0019] Similarly, the network depth calibration layer 314 (for module k) receives the feature map xk as input from the convolutional layer 304, and also receives a hidden state signal hk-1 and a cell state signal ck-1 from the network depth calibration layer 312 in the previous module (k-1), and generates an output feature map yk. As shown in this example of FIG3A, the output yk can be fed to a subsequent module (e.g., module (k+1)) or another neural network layer. For the next module, the network depth calibration layer 316 (for module k+1) receives the feature map xk+1 as input from the convolutional layer 306, and also receives a hidden state signal hk and a cell state signal ck from the network depth calibration layer 314 in the previous module (k), and generates an output feature map yk+1. As shown in this example of FIG3A, the output yk+1 can be fed to a subsequent module (not shown in FIG3A) or another neural network layer. The network depth calibration structure 300 shown in Figure 3A can be repeatedly used for all or part of the remaining portion of the neural network.

[0020] The network depth calibration structure 300 may include one or more selectable activation layers, such as activation layers 303, 305, and / or 307. Each of the activation layers 303, 305, and / or 307 may include activation functions useful for CNNs, such as, for example, a rectified linear unit (ReLU) function, a softmax function, etc.

[0021] The activation layers 303, 305, and / or 307 can receive the outputs of the respective adjacent network depth calibration layers 312, 314, and / or 316 as inputs. For example, as shown in FIG. 3A, activation layer 303 receives the output yk-1 from network depth calibration layer 312 as input, and the output of activation layer 303 is fed into a subsequent module or another neural network layer. Similarly, as shown in FIG. 3A, activation layer 305 receives the output yk from network depth calibration layer 314 as input, and the output of activation layer 305 is fed into a subsequent module or another neural network layer. Likewise, as shown in FIG. 3A, activation layer 307 receives the output yk+1 from network depth calibration layer 316 as input, and the output of activation layer 256 is fed into a subsequent module or another neural network layer (if present).

[0022] In some embodiments, the activation functions of the activation layers 303, 305, and / or 307 may be incorporated into the respective adjacent network depth calibration layers 312, 314, and / or 316. In some embodiments, each of the activation layers 303, 305, and / or 307 may be arranged between a respective convolutional layer and the subsequent network depth calibration layer. The network depth calibration structure 300 may include one or more additional / optional neural network layers, such as convolutional layers (not shown in FIG. 3A).

[0023] The implementation of some or all of the components and features of the network depth calibration structure 300 may be achieved using a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator, a field-programmable gate array (FPGA) accelerator, an application-specific integrated circuit (ASIC), and / or via one or more of a processor with software, or a combination of a processor with software and an FPGA or ASIC. More specifically, the components and features of the network depth calibration structure 300 can be implemented in one or more modules as a set of logic instructions, which are stored in a non-transitory machine or computer-readable storage medium such as random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc.; stored in combinable logic such as, for example, programmable logic (PLA), FPGA, complex programmable logic device (CPLD); and stored in fixed-function logic hardware that uses circuit technologies such as, for example, ASIC, complementary metal-oxide-semiconductor (CMOS), or transistor-transistor logic (TTL) technology, or any combination thereof.

[0024] Figure 3B provides a schematic diagram illustrating an example of a network depth calibration layer (FCAA-D) 350 for a neural network according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The network depth calibration layer 350 may correspond to any of the network depth calibration layers 222 (Figure 2B, already discussed), 272 (Figure 2C, already discussed), and / or 312, 314, and / or 316 (Figure 3A, already discussed). As shown in Figure 3B, the network depth calibration layer 350 will be described with reference to a module (k) (e.g., the network depth calibration layer 314 corresponding to Figure 3A). The network depth calibration layer 350 receives the output feature map xk of the convolutional layer (e.g., the convolutional layer 304 shown in Figure 3A, already discussed) for module k as an input. For example, the feature map xk can represent a video (or image sequence) feature map, which is a feature tensor with a time dimension T and other dimensions associated with an image: Equation (1) where N, C, T, H, and W represent the batch size, number of channels, time length, height, and width of the tensor xk, respectively.

[0025] The network depth calibration layer 350 may include a first global average pooling (GAP) function 352, a first meta-gate control relay (MGR) unit 354, a first standardization (STD) function 356, and a first linear transformation (LNT) function 358. The GAP function 352 is a known function used in CNNs. The GAP function 352 operates on the feature map xk by calculating the average output of the feature map xk (e.g., the feature map xk generated by the convolutional layer 304 used for module (k) in FIG. 3A) to produce an output x̅k: x̅k = GAP(xk) Equation (2) which represents a spatiotemporal aggregation of the input feature map xk. For an input feature map having dimensions (N×C×T×H×W), the GAP function 352 produces a result output of dimension (N×C×1).

[0026] The output, x̅k, of the GAP function 352 is fed into the first MGR unit 354. The first MGR unit 354 is a shared lightweight structure capable of dynamically generating feature calibration parameters and relaying these parameters between coupling layers along the neural network depth. The first MGR unit 354 of the network depth calibration layer 350 receives additional input from the previous module (k-1) of the network depth calibration layer in the form of a hidden state signal hk-1 and a cell state signal ck-1, and generates an updated hidden state signal hk and an updated cell state signal ck. Equation (3)

[0027] The updated hidden state signal hk and the updated cell state signal ck are fed into the LNT function 358, and also into a network depth calibration layer of a subsequent module (k+1). Further details regarding the first MGR unit 354 will be provided herein with reference to Figures 3C-3D.

[0028] The STD function 356 operates on the input feature map xk by calculating a standardized feature as follows: Equation (4) where μ and σ are the mean and standard deviation calculated in the non-overlapping subset of the input feature map, and This is a small constant that maintains numerical stability. The output of the STD function 356... This is a standardized characteristic, which is expected to lie in a distribution with zero mean and unit variation. This standardized characteristic... Feed 358 into the LNT function.

[0029] The LNT function 358 has this standardized feature The operation is performed to calibrate and combine the feature representation capability of the feature map. The LNT function 358 uses the hidden state signal hk and the cell state signal ck (which, as described herein, is generated by the first MGR unit 354) as scaling and shifting parameters to calculate an output yk, as shown below: Equation (5) where yk is the output of the network depth calibration layer of module (k), hk and ck are the hidden state signal and cell state signal generated by the first MGR unit 354, respectively, and The standardized feature is generated by the STD function 356. In this way, the calibrated 3D feature yk receives the feature distribution dynamics of the previous layer and relays its calibration statistics to the next layer via the shared network depth relay structure.

[0030] Some or all of the components and features of the network depth calibration layer 350 may be implemented using one or more of a CPU, a GPU, an AI accelerator, an FPGA accelerator, an ASIC, and / or via a processor with software, or in a combination of a processor with software and an FPGA or ASIC. More specifically, the components and features of the network depth calibration layer 350 may be implemented in one or more modules as a set of logic instructions stored in a non-transitory machine or computer-readable storage medium such as RAM, read-only memory ROM, PROM, firmware, flash memory, etc.; stored in combinable logic such as, for example, PLA, FPGA, CPLD; or stored in fixed-function logic hardware that uses circuit technologies such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0031] Figure 3C provides a schematic diagram illustrating, according to one or more embodiments, an example of a single gated relay (MGR) unit 360 for a network depth calibration layer (module k) of a neural network. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The MGR unit 360 may correspond to the first MGR unit 354 (Figure 3B, already discussed). The MGR unit 360 includes a modified long short-term memory (LSTM) unit 370. The modified LSTM cell 370 can be generated from an LSTM cell used in a neural network; an example of a modified LSTM cell is provided herein with reference to Figure 3D. The modified LSTM cell 370 receives the spatiotemporal aggregation x̅k (Equation 2) from the previous module (k-1) of the network depth calibration layer, as well as the hidden state signal hk-1 and the cell state signal ck-1 as inputs to generate an updated hidden state signal hk and an updated cell state signal ck.

[0032] Figure 3D provides a schematic diagram illustrating an example of an MGR unit 380 for a network depth calibration layer (module k) of a neural network, according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The MGR unit 380 may correspond to the first MGR unit 354 (Figure 3B, already discussed) and / or the MGR unit 360 (Figure 3C, already discussed). In particular, the MGR unit 380 includes an example of a modified LSTM cell, such as the modified LSTM cell 370 (Figure 3C, already discussed). The MGR unit 380 provides a gate control mechanism, which can be represented as: Equation (6) where This is a bottleneck unit used to process the spatiotemporal aggregation x̅k (Equation 2) and the hidden state signal hk-1 from the network's deep calibration layer (k-1), while b is a deviation. For example, this bottleneck unit It can be a shrink-expand bottleneck unit having a fully connected (FC) layer that maps the input to a low-dimensional space with the shrinkage ratio r, a ReLU activation layer, and another FC layer that maps the input back to the original dimensional space. In some embodiments, the bottleneck unit It can be implemented with a reduction ratio of 4. In some embodiments, the bottleneck unit It can be implemented as any form of linear or nonlinear mapping. These dynamically generated parameters... A set of gates is formed to adjust the update of the cell state signal ck and the hidden state signal hk of the MGR unit 380 in module (k) as follows: Equation (7) and Equation (8) where ck is the updated cell state signal, hk is the updated hidden state signal, and ck-1 is the cell state signal from module (k-1) of the previous network depth calibration layer. This refers to the sigmoid function, and This is the Hadamard product operator.

[0033] Some or all of the components and features of the MGR unit 360 and / or the MGR unit 380 may be implemented using one or more of a CPU, a GPU, an AI accelerator, an FPGA accelerator, an ASIC, and / or via a processor with software, or in a combination of a processor with software and an FPGA or ASIC. More specifically, the components and features of the MGR unit 360 and / or the MGR unit 380 may be implemented in one or more modules as a set of logic instructions stored in a non-transitory machine or computer-readable storage medium such as RAM, read-only memory ROM, PROM, firmware, flash memory, etc.; stored in combinable logic such as, for example, PLA, FPGA, CPLD; or stored in fixed-function logic hardware using circuit technologies such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0034] Figure 4A is a block diagram illustrating an example of a feature dimension calibration structure 400 according to one or more embodiments. The diagram references the components and features described herein, including but not limited to the figures and associated descriptions. The feature dimension calibration structure 400 can be used in all or part of the neural network 110 (Figures 1A-1B, already discussed). The feature dimension calibration structure 400 includes a convolutional layer 402 (representing layer n). The convolutional layer 402 operates to provide an output feature map xn. The convolutional layer 402 corresponds to one or more convolutional layers shown in Figure 2A and to convolutional layer 224 shown in Figure 2B, and has parameters and weights determined through a neural network training process. The feature map xn can represent, for example, a video (or image sequence) feature map, similar to the feature map xk described above with reference to Figures 3A-3D.

[0035] The feature map xn, as the output of the convolutional layer 402, can be segmented along the time dimension into a set of T slices 404 {xn,1, xn,2, ..., xn,t, ..., xn,T}, such that each slice xn,t represents a feature slice corresponding to one or more frames (e.g., one or more input frames for a slice t). In some embodiments, these feature slices 404 {xn,1, xn,2, ..., xn,t-1, xn,t, xn,t+1, ..., xn,T} can represent a feature map segmented along a feature dimension other than the time dimension.

[0036] The feature dimension calibration structure 400 includes a plurality of feature dimension calibration slices (e.g., FCAA-T(slice t)) arranged in a feature dimension relay structure 410. The feature dimension relay structure 410 includes a feature dimension calibration slice 412 (for slice t-1), a feature dimension calibration slice 414 (for slice t), and a feature dimension calibration slice 416 (for slice t+1), and so on. Each feature dimension calibration slice receives an input (e.g., xn,t) from its respective feature slice and produces an output slice (e.g., yn,t). This output is a set of T slices 406 {yn,1, yn,2, ..., yn,t-1, yn,t, yn,t+1, ..., yn,T}.

[0037] Each feature dimension calibration slice (i.e., every feature dimension calibration slice other than the initial slice t=1) is also coupled to the feature dimension calibration slice of the preceding slice via a hidden state signal and a cell state signal received from the feature dimension calibration slice of the respective preceding slice. Therefore, as shown in the example of FIG4A, the feature dimension relay structure 410 includes arranging a feature dimension calibration slice for each slice (t) to be coupled to a feature dimension calibration slice for the preceding slice (t-1). The feature dimension relay structure 410 corresponds to the feature dimension relay structure 134 (shown in FIG1, already discussed). The feature dimension relay structure 410 also corresponds to the feature dimension calibration layer 225 (FIG. 2B, already discussed) and / or the feature dimension calibration layer 292 (FIG. 2D, already discussed).

[0038] For example, the feature dimension calibration slice 412 (for slice t-1) receives input from slice xn,t-1 and also receives a hidden state signal and a cell state signal from a feature calibration slice in the previous slice (not shown in FIG. 4A), unless slice t-1 is the initial slice (in which case there will be no previous feature calibration slice). The feature dimension calibration slice 412 (for slice t-1) produces an output slice yn,t-1.

[0039] Similarly, the feature dimension calibration slice 414 (for slice t) receives input from slice xn,t and also receives a hidden state signal ht-1 and a cell state signal ct-1 from the feature dimension calibration slice 412 (for slice t-1), and produces an output slice yn,t. For the next slice, the feature dimension calibration slice 416 (for slice t+1) receives input from slice xn,t+1 and also receives a hidden state signal ht and a cell state signal ct from the feature dimension calibration slice 414 (for slice t), and produces an output slice yn,t+1. These output slices 406 {yn,1, yn,2, ... yn,t-1, yn,t, yn,t+1, ..., yn,T} can be combined into a feature map yn and, as shown in the example of Figure 4A, provided to another layer or part of the neural network. The feature dimension calibration structure 400 shown in Figure 4A can be repeated in one or more modules of the neural network.

[0040] The feature dimension calibration structure 400 may include one or more selectable activation layers, such as activation layer 408. Each activation layer 408 may include an activation function useful for CNNs, such as, for example, a rectified linear unit (ReLU) function, a softmax function, etc. In some embodiments, such activation functions of the activation layer 408 may be incorporated into such feature dimension calibration slices 412, 414, and / or 416. The feature dimension calibration structure 400 may include one or more additional / selectable neural network layers, such as convolutional layers (not shown in FIG. 4A).

[0041] The implementation of some or all of the components and features of the feature dimension calibration structure 400 may be carried out using a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator, a field-programmable gate array (FPGA) accelerator, an application-specific integrated circuit (ASIC), and / or via one or more of a processor with software, or a combination of a processor with software and an FPGA or ASIC. More specifically, the components and features of the feature dimension calibration structure 400 can be implemented in one or more modules as a set of logic instructions, which are stored in a non-transitory machine or computer-readable storage medium such as random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc.; stored in combinable logic such as, for example, programmable logic (PLA), FPGA, complex programmable logic device (CPLD); or stored in fixed-function logic hardware that uses circuit technologies such as, for example, ASIC, complementary metal-oxide-semiconductor (CMOS), or transistor-transistor logic (TTL) technology, or any combination thereof.

[0042] Figure 4B provides a schematic diagram illustrating an example of a feature dimension calibration slice (FCAA-T) 450 for a neural network according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The feature dimension calibration slice 450 may correspond to any of the feature dimension calibration slices 412, 414, and / or 416 (Figure 4A, already discussed). As shown in Figure 4B, the feature dimension calibration slice 450 will be described with reference to a slice(t) (e.g., corresponding to feature dimension calibration slice 414 in Figure 4A). The feature dimension calibration slice 450 receives a slice xn,t of a feature map xn (e.g., the slice xn,t of the feature map xn shown in Figure 4A, already discussed) as an input.

[0043] The feature dimension calibration slice 450 may include a second GAP function 452, a second MGR unit 454, a second STD function 456, and a second LNT function 458. The GAP function 452 is a known function used in CNNs and has the same form as the GAP function 352 (Figure 3B, already discussed). The GAP function 452 operates on the feature slice xn,t by calculating the average output of the feature slice xn,t to produce an output x̅n,t: x̅n,t = GAP(xn,t) Equation (9) which represents a spatial aggregation of the input feature slice xn,t. For an input feature map with dimensions (N×C×T×H×W), the GAP function 452 produces a result output with dimensions (N×C×1).

[0044] The output, x̅n,t, of the GAP function 452 is fed into the second MGR unit 454. The second MGR unit 454 is a shared lightweight structure capable of dynamically generating feature calibration parameters and relaying these parameters along the time dimension between coupled slices. The second MGR unit 454 of the feature dimension calibration slice 450 receives additional input from the feature dimension calibration slice of the previous slice (t-1) in the form of a hidden state signal ht-1 and a cell state signal ct-1, and generates an updated hidden state signal ht and an updated cell state signal ct. Equation (10)

[0045] The updated hidden state signal ht and the updated cell state signal ct are fed into the LNT function 458, and also into a feature dimension calibration slice of a subsequent slice (t+1). Further details regarding the second MGR unit 454 will be provided herein with reference to Figures 4C-4D.

[0046] The STD function 456 has the same form as the STD function 356 (Figure 3B, already discussed). The STD function 456 operates on the input feature slice xn,t by calculating a standardized feature as follows: Equation (11) where μ and σ are the mean and standard deviation calculated in the non-overlapping subset of the input feature map, and This is a small constant that maintains numerical stability. The output of the STD function 456... This is a standardized characteristic, which is expected to lie in a distribution with zero mean and unit variation. This standardized characteristic... Feed 458 into the LNT function.

[0047] The LNT function 458 has the same form as the LNT function 358 (Figure 3B, already discussed). The LNT function 458 has the same normalized characteristics. The operation is performed to calibrate and combine the feature representation capabilities of the feature slice. The LNT function 458 uses the hidden state signal ht and the cell state signal ct (which, as described herein, is generated by the second MGR unit 454) as scaling and shifting parameters to calculate an output yn,t, as shown below: Equation (12) where yn,t is the output used to calibrate the slice for the feature dimension of slice (t), ht and ct are the hidden state signal and cell state signal generated by the second MGR unit 454, respectively, and The standardized feature is generated by the STD function 456. In this way, the calibrated 3D feature yn,t receives the feature distribution dynamics of the previous time slice (e.g., timestamp) and relays its calibration statistics to the next time slice (e.g., timestamp) via the shared feature dimension relay structure.

[0048] Some or all of the components and features of the feature dimension calibration slice 450 may be implemented using one or more of a CPU, a GPU, an AI accelerator, an FPGA accelerator, an ASIC, and / or via a processor with software, or in a combination of a processor with software and an FPGA or ASIC. More specifically, the components and features of the feature dimension calibration slice 450 may be implemented in one or more modules as a set of logic instructions stored in a non-transitory machine or computer-readable storage medium such as RAM, read-only memory ROM, PROM, firmware, flash memory, etc.; stored in combinable logic such as, for example, PLA, FPGA, CPLD; or stored in fixed-function logic hardware that uses circuit technologies such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0049] Figure 4C provides a schematic diagram illustrating an example of an MGR unit 460 for a feature dimension calibration slice according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The MGR unit 460 may correspond to the second MGR unit 454 (Figure 4B, already discussed). The MGR unit 460 includes a modified LSTM cell 470. The modified LSTM cell 470 can be generated from an LSTM cell used in a neural network; an example of a modified LSTM cell is provided herein with reference to Figure 4D. The modified LSTM cell 470 receives the spatial aggregation x̅n,t (Equation 9) of the feature dimension calibration slice from the previous slice (t-1), the hidden state signal ht-1, and the cell state signal ct-1 as inputs to generate an updated hidden state signal ht and an updated cell state signal ct.

[0050] Figure 4D provides a schematic diagram illustrating an example of an MGR unit 480 for a feature dimension calibration slice according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The MGR unit 480 may correspond to the second MGR unit 454 (Figure 4B, already discussed) and / or the MGR unit 460 (Figure 4C, already discussed). In particular, the MGR unit 480 includes an example of a modified LSTM cell, such as the modified LSTM cell 470 (Figure 4C, already discussed). The MGR unit 480 provides a gate control mechanism, which can be represented as: Equation (13) where A bottleneck unit is used to process the spatial aggregation x̅n,t (Equation 9) from the previous feature dimension calibration slice (t-1) and the hidden state signal ht-1, where b is a bias. For example, this bottleneck unit It can be a shrink-expand bottleneck unit having a fully connected (FC) layer that maps the input to a low-dimensional space with the shrinkage ratio r, a ReLU activation layer, and another FC layer that maps the input back to the original dimensional space. In some embodiments, the bottleneck unit It can be implemented as any form of linear or nonlinear mapping. These dynamically generated parameters... A set of gates is formed to adjust the updates of the cell state signal ct and the hidden state signal ht for the MGR cell 480 of slice (t) as follows: Equation (14) and Equation (15) where ct represents the updated cell state signal, ht represents the updated hidden state signal, and ct-1 represents the cell state signal from the previous slice (t-1). This refers to the sigmoid function, and This is the Hadamard product operator.

[0051] Some or all of the components and features of the MGR unit 460 and / or the MGR unit 480 may be implemented using one or more of a CPU, a GPU, an AI accelerator, an FPGA accelerator, an ASIC, and / or via a processor with software, or in a combination of a processor with software and an FPGA or ASIC. More specifically, the components and features of the MGR unit 460 and / or the MGR unit 480 may be implemented in one or more modules as a set of logic instructions stored in a non-transitory machine or computer-readable storage medium such as RAM, read-only memory ROM, PROM, firmware, flash memory, etc.; stored in combinable logic such as, for example, PLA, FPGA, CPLD; or stored in fixed-function logic hardware using circuit technologies such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0052] The neural network structures and / or network depth calibration layers and feature dimension calibration layers described herein (e.g., Figures 2A-2D, 3A-3D, and 4A-4D) can be interleaved and applied to any existing 3D CNN (e.g., as shown in Figures 2A-2D) to amplify the capabilities of 3D CNN models.

[0053] Figure 5A is a flowchart illustrating a method 500 for constructing a neural network according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The method 500 can be used, for example, to construct the neural network 110 (Figures 1A-1B, already discussed), and / or the neural network structure 200 (Figures 2A-2D, already discussed), and can utilize the network depth calibration structure 300 (Figure 3A, already discussed), the feature dimension calibration structure 400 (Figure 4A, already discussed), and / or any of its components (Figures 3A-3D, already discussed, or Figures 4A-4D, already discussed). The method 500 can typically be implemented in the system 100 (Figures 1A-1B, already discussed), and / or using one or more of a CPU, a GPU, an AI accelerator, an FPGA accelerator, an ASIC, and / or via a processor with software, or in a combination of a processor with software and an FPGA or ASIC. More specifically, the method 500 can be implemented as a set of logic instructions in one or more modules, stored in a non-transitory machine or computer-readable storage medium such as RAM, read-only memory ROM, PROM, firmware, flash memory, etc.; stored in combinatorial logic such as, for example, PLA, FPGA, CPLD; or stored in fixed-function logic hardware that uses circuitry such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0054] Processing block 502 provided for generating a plurality of convolutional layers in a neural network. Processing block 504 provided for arranging a network depth relay structure comprising a plurality of network depth calibration layers in a neural network, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers. Processing block 506 provided for arranging a feature dimension relay structure comprising a plurality of feature dimension calibration slices in the neural network, wherein the feature dimension relay structure is coupled to an output of another layer in the plurality of convolutional layers.

[0055] Figure 5B is a flowchart illustrating a method 520 for constructing a neural network according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. This method 520 can be used, for example, to construct the neural network 110 (Figures 1A-1B, already discussed), and / or the neural network structure 200 (Figures 2A-2D, already discussed), and can utilize the network depth calibration structure 300 (Figure 3A, already discussed), the feature dimension calibration structure 400 (Figure 4A, already discussed), and / or any of its components (Figures 3A-3D, already discussed, or Figures 4A-4D, already discussed). This method 520 can typically be implemented in the system 100 (Figures 1A-1B, already discussed), and / or using one or more of a CPU, a GPU, an AI accelerator, an FPGA accelerator, an ASIC, and / or via a processor with software, or in a combination of a processor with software and an FPGA or ASIC. More specifically, this method 520 can be implemented as a set of logic instructions in one or more modules, stored in a non-transitory machine or computer-readable storage medium such as RAM, read-only memory ROM, PROM, firmware, flash memory, etc.; stored in combinatorial logic such as, for example, PLA, FPGA, CPLD; or stored in fixed-function logic hardware that uses circuitry such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0056] In the illustrated processing block 522, each network depth calibration layer includes a first meta-gateway relay (MGR) unit. In the illustrated processing block 524, each network depth calibration layer is coupled to the preceding network depth calibration layer via a first hidden state signal and a first cell state signal, each of which is generated by a separate first MGR unit of the preceding network depth calibration layer. The illustrated processing block 524 can typically replace at least a portion of the illustrated processing block 504.

[0057] In the illustrated processing block 526, each feature dimension calibration slice includes a second meta-gate relay (MGR) unit. In the illustrated processing block 528, each feature dimension calibration slice is coupled to the previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of which is generated by a separate second MGR unit of the previous feature dimension calibration unit. The illustrated processing block 528 can typically replace at least a portion of the illustrated processing block 506.

[0058] In the processing block 530 shown, each of the first MGR unit and the second MGR unit includes a modified long short-term memory (LSTM) cell. In some embodiments, the modified LSTM cell may include a gate mechanism employing a bottleneck unit.

[0059] In the processing block 532 shown, each network depth calibration layer calibration unit further includes a first global average pooling (GAP) function, a first standardization (STD) function, and a first linear transformation (LNT) function. The first GAP function operates on a feature map, the first STD function operates on the feature map, and the first LNT function operates on an output of the first STD function, wherein the first LNT function is based on the first hidden state signal generated by the first MGR unit and the first cell state signal generated by the first MGR unit.

[0060] In the processing block 534 shown, each feature dimension calibration unit further includes a second GAP function, a second STD function, and a second LNT function. The second GAP function operates on a feature slice, the second STD function operates on the feature slice, and the second LNT function operates on an output of the second STD function, wherein the second LNT function is based on the second hidden state signal generated by the second MGR unit and the second cell state signal generated by the second MGR unit.

[0061] Therefore, the disclosed technique provides a combination of the network depth relay structure and the feature dimension relay structure, which is used to combine the 3D feature distribution dependencies simultaneously along the time dimension and along the network depth (e.g., between adjacent layers or modules). By employing the neural network technique as described herein with reference to Figures 1A-1B, 2A-2D, 3A-3D, 4A-4D, and 5A-5B, the MGR structure is integrated with meta-learning, such that the hidden state hk and cell state ck are set to scaling and shift parameters for calibrating the visual feature tensor xk of the k-th module (along the network depth), and the hidden state ht and cell state ct are set to scaling and shift parameters for calibrating the feature slice xn,t of the t-th input slice (along the time dimension). By using the network's deep relay structure, the feature dimension relay structure, and the gate control mechanism of each individual MGR unit, the calibration parameters for the k-th layer feature map and the t-th frame feature slice can be based not only on the current input feature map xk and the current input feature slice xn,t, but also on the estimated calibration parameters ck-1 and hk-1 for the previous (k-1) layer and the estimated calibration parameters ct-1 and ht-1 for the previous (t-1) feature slice. Furthermore, the neural network technique described herein utilizes observed feature distributions to guide the learning dynamics of the current feature calibration layer. The intermediate feature distributions are implicitly interdependent as a whole system, and these latent conditions are extracted for learning the calibration parameters using the shared MGR units in the SA-FCAA technique disclosed herein. Moreover, the technique disclosed herein explicitly utilizes the feature correlations across layers and along the time dimension, generating calibration parameters jointly in an adaptive relay manner for each individual video sample in both training and inference. These parameters can be optimized simultaneously with those of the main network in a reverse propagation because their computational flow is fully differentiable.

[0062] Figures 6A-6F illustrate examples of input image sequences and corresponding activation maps in a system for image sequence / video analysis according to one or more embodiments, referencing the components and features described herein, including but not limited to the figures and associated descriptions. The input image sequences (shown as grayscale images in Figures 6A, 6C, and 6E) are taken from sample image sequences in the Kinetics-200 dataset. Although each input sequence is shown in eight frames in Figures 6A, 6C, and 6E, the input sequences used include video clips with thirty-two frames. The activation maps (shown as being stacked on top of the respective input images from Figures 6A, 6C, and 6E and converted to grayscale in Figures 6B, 6D, and 6F) are generated by processing the input image sequences using an example of the neural network technology described herein. Figure 6A provides an example of an input image sequence for a trumpet performance, as indicated by reference numeral 602. Figure 6B provides a set of activation maps as shown in reference numeral 604, each of which is stacked on top of and corresponds to one of the input images in Figure 6A. Figure 6C provides an example of an input image sequence for breakdancing, as shown in reference numeral 612. Figure 6D provides a set of activation maps as shown in reference numeral 614, each of which is stacked on top of and corresponds to one of the input images in Figure 6C. Figure 6E provides an example of an input image sequence for juggling balls, as shown in reference numeral 622. Figure 6F provides a set of activation maps as shown in reference numeral 624, each of which is stacked on top of and corresponds to one of the input images in Figure 6E.

[0063] As shown in Figures 6B, 6D, and 6F, the bright areas in each activation map indicate regions identified as motion regions by the neural network, with motion regions identified during the sequence highlighted. As demonstrated in each set of examples, the neural network technique described herein provides consistent emphasis on motion-related attention regions within an image sequence or video clip with high confidence and accuracy. Therefore, the techniques disclosed herein can be used to augment spatiotemporal feature learning for 3D CNNs and provide key improvements to image sequence / video representation learning for high-performance image sequence / video analytics tasks.

[0064] Figure 7 illustrates a block diagram of an example computing system 10 for image sequence / video analysis according to one or more embodiments. The diagram references the components and features described herein, including but not limited to the figures and associated descriptions. System 10 can typically be part of an electronic device / platform having computing and / or communication functions (e.g., servers, cloud infrastructure controllers, database controllers, laptops, desktop computers, personal digital assistants / PDAs, tablets, convertible tablets, smartphones, etc.), imaging functions (e.g., cameras, camcorders), media playback functions (e.g., smart TVs / TVs), wearable functions (e.g., watches, glasses, headwear, footwear, jewelry), vehicle functions (e.g., cars, trucks, motorcycles), robotic functions (e.g., autonomous robots), Internet of Things (IoT) functions, etc., or any combination thereof. In the illustrated example, system 10 may include a host processor 12 (e.g., a central processing unit / CPU) having an integrated memory controller (IMC) 14, which may be coupled to system memory 20. The host processor 12 may include any type of processing device, such as, for example, a microcontroller, microprocessor, RISC processor, ASIC, etc., accompanied by associated processing modules or circuitry. The system memory 20 may include any non-transitory machine or computer-readable storage medium such as RAM, ROM, PROM, EEPROM, firmware, flash memory, etc.; combinable logic such as, for example, PLA, FPGA, CPLD; fixed-function hardware logic using circuitry such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof suitable for storing instructions 28.

[0065] The system 10 may further include an input / output (I / O) subsystem 16. The I / O subsystem 16 may communicate with, for example, one or more input / output (I / O) devices 17, a network controller 24 (e.g., a wired and / or wireless NIC), and storage 22. The storage 22 may consist of any suitable non-transitory machine- or computer-readable memory type (e.g., flash memory, DRAM, SRAM (static random access memory), solid-state drive (SSD), hard disk drive (HDD), optical disk, etc.). The storage 22 may include mass storage. In some embodiments, the host processor 12 and / or the I / O subsystem 16 may communicate with the storage 22 (all or part thereof) via a network controller 24. In some embodiments, the system 10 may further include a graphics processor 26 (e.g., a graphics processing unit / GPU) and an AI accelerator 27. In one embodiment, the system 10 may further include a vision processing unit (VPU), but not shown in the figures.

[0066] The host processor 12 and the I / O subsystem 16 can be implemented together on a semiconductor die as a system-on-a-chip (SoC) 11, as shown in a solid box. The SoC 11 can therefore operate as a computing device for image sequence / video analysis. In some embodiments, the SoC 11 may also include one or more of the system memory 20, the network controller 24, and / or the graphics processor 26 (shown in dashed boxes). In some embodiments, the SoC 11 may also include other components of the system 10.

[0067] The host processor 12 and / or the I / O subsystem 16 can execute program instructions 28 retrieved from the system memory 20 and / or the storage 22 to perform one or more aspects of program 500 and / or program 520 as described herein with reference to Figures 5A-5B. The system 10 can implement one or more aspects of the system 100, the neural network 110, the neural network structure 200, the network depth calibration structure 300, the network depth relay structure 310, the network depth calibration layer 350, the MGR unit 360, the MGR unit 380, the feature dimension calibration structure 400, the feature dimension relay structure 410, the feature dimension calibration slice 450, the MGR unit 460, and / or the MGR unit 480 as described herein with reference to Figures 1A-1B, 2A-2D, 3A-3D, and 4A-4D. Therefore, the system 10 is considered to be performance-enhanced, at least to the extent that the technology provides a consistent ability to identify motion-related attention regions in a sequence of images / videos.

[0068] The computer program code that executes the above-described programs can be written in any combination of one or more programming languages, including an object-oriented programming language such as JAVA, JAVASCRIPT, PYTHON, SMALLTALK, C++, etc., and / or a traditional programming language such as the "C" programming language or a similar programming language, and is implemented as program instructions 28. In addition, program instructions 28 may include combination instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, status-setting data, assembly data for integrated circuits, status information of personalized electronic circuits, and / or other structural components inherent to the hardware (e.g., host processor, central processing unit / CPU, microcontroller, microprocessor, etc.).

[0069] I / O device 17 may include one or more input devices, such as a touch screen, keyboard, mouse, cursor control, microphone, digital camera, video recorder, camcorder, biometric scanner, and / or sensor; the input devices may be used to input information and interact with system 10 and / or other devices. The I / O device 17 may also include one or more output devices, such as a display (e.g., touch screen, liquid crystal display / LCD, light-emitting diode / LED display, plasma panel, etc.), speaker, and / or other visual or audio output devices. These input and / or output devices may be used, for example, to provide a user interface.

[0070] Figure 8 shows a block diagram illustrating an example semiconductor device 30 for image sequence / video analysis according to one or more embodiments. The diagram references the components and features described herein, including but not limited to the figures and associated descriptions. The semiconductor device 30 can be implemented, for example, as a wafer, die, or other semiconductor package. The semiconductor device 30 may include one or more substrates 32 made of, for example, silicon, sapphire, gallium arsenide, etc. The semiconductor device 30 may also include logic 34, which is composed of, for example, transistor arrays and other integrated circuit (IC) components coupled to the substrates 32. The logic 34 may be implemented at least partially as configurable logic or fixed-function logic hardware. The logic 34 can implement the system SoC 11 described above with reference to Figure 7. The logic 34 can implement one or more aspects of the above-described procedures, including procedure 500 and / or procedure 520. The logic 34 can implement one or more aspects of the system 100, the neural network 110, the neural network structure 200, the network depth calibration structure 300, the network depth relay structure 310, the network depth calibration layer 350, the MGR unit 360, the MGR unit 380, the feature dimension calibration structure 400, the feature dimension relay structure 410, the feature dimension calibration slice 450, the MGR unit 460, and / or the MGR unit 480, as described herein with reference to Figures 1A-1B, 2A-2D, 3A-3D, and 4A-4D. Therefore, the device 30 is considered to have enhanced performance, at least to the extent that the technology provides a consistent ability to identify motion-related attention regions in an image sequence / video.

[0071] The semiconductor device 30 can be constructed using any suitable semiconductor process or technology. For example, the logic 34 may include a transistor channel region located (e.g., embedded) within the substrate 32. Therefore, the interface between the logic 34 and the substrate 32 may not be a steep interface. The logic 34 may also be considered as an epitaxial layer grown on an initial wafer of the substrate 34.

[0072] Figure 9 is a block diagram illustrating a processor core 40 according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The processor core 40 can be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a graphics processing unit (GPU), or other device for executing code. Although only one processor core 40 is shown in Figure 9, a processing element may alternatively include more than one of the processor cores 40 shown in Figure 9. The processor core 40 can be a single-threaded core, or, for at least one embodiment, the processor core 40 can be multi-threaded, as each core may have more than one hardware thread environment (or "logic processor").

[0073] Figure 9 also illustrates a memory 41 coupled to the processor core 40. The memory 41 can be any of various types of memory known to those skilled in the art or otherwise available (including various layers of a memory hierarchy architecture). The memory 41 may include one or more code 42 instructions to be executed by the processor core 40. The code 42 may implement one or more aspects of the programs 500 and / or 520 described above. The processor core 40 can implement one or more aspects of the system 100, the neural network 110, the neural network structure 200, the network depth calibration structure 300, the network depth relay structure 310, the network depth calibration layer 350, the MGR unit 360, the MGR unit 380, the feature dimension calibration structure 400, the feature dimension relay structure 410, the feature dimension calibration slice 450, the MGR unit 460, and / or the MGR unit 480, as described herein with reference to Figures 1A-1B, 2A-2D, 3A-3D, and 4A-4D. The processor core 40 can follow a program sequence of instructions indicated by the code 42. Each instruction can enter a front-end section 43 and be processed by one or more decoders 44. The decoder 44 can produce a microoperation as its output, such as a microoperation of a fixed width with one of a predefined format, or can produce other instructions, microinstructions, or control signals that reflect the original code instructions. The front-end section 43 shown also includes register renaming logic 46 and scheduling logic 48, which typically allocate resources and queue the operation corresponding to the conversion instruction for execution.

[0074] The processor core 40 is shown to include execution logic 50 with a set of execution units 55-1 to 55-N. Some embodiments may include multiple execution units dedicated to a specific function or group of functions. Other embodiments may include a single execution unit or a single execution unit capable of performing a specific function. The execution logic 50 shown performs the operations specified by code instructions.

[0075] After the execution of the operations specified by the code instructions is completed, the back-end logic 58 retires the instructions of code 42. In one embodiment, the processor core 40 allows out-of-order execution but requires instructions to be retired sequentially. The retirement logic 59 may take various forms known to those skilled in the art (e.g., reorder buffers, etc.). In this way, the processor core 40 is transformed during the execution of the code 42, at least in terms of the output generated by the decoder, the hardware registers and tables used by the register renaming logic 46, and any registers (not shown) modified by the execution logic 50.

[0076] Although not shown in Figure 9, a processing element may include other elements on a chip having the processor core 40. For example, a processing element may include memory control logic integrated with the processor core 40. The processing element may include I / O control logic and / or may include I / O control logic integrated with the memory control logic. The processing element may also include one or more caches.

[0077] Figure 10 is a block diagram illustrating an example of a multiprocessor-based computing system 60 according to one or more embodiments. The figure references the components and features described herein, including but not limited to the figures and associated descriptions. The multiprocessor system 60 includes a first processing element 70 and a second processing element 80. Although two processing elements 70 and 80 are shown, it should be understood that an embodiment of the system 60 may also include only one such processing element.

[0078] The system 60 is illustrated as a point-to-point interconnect system, wherein the first processing element 70 and the second processing element 80 are coupled via a point-to-point interconnect 71. It should be understood that any or all of the interconnects shown in FIG10 may be implemented as a multipoint bus rather than a point-to-point interconnect.

[0079] As shown in Figure 10, each of these processing elements 70 and 80 can be a multi-core processor, including a first and a second processor core (i.e., processor cores 74a and 74b and processor cores 84a and 84b). Such cores 74a, 74b, 84a, and 84b can be assembled to execute instruction code in a manner similar to that discussed above in conjunction with Figure 9.

[0080] Each processing element 70, 80 may include at least one shared cache 99a, 99b. The shared cache 99a, 99b may store data (e.g., instructions) used by one or more components of the processor, such as the cores 74a, 74b and 84a, 84b. For example, the shared cache 99a, 99b may locally cache data stored in a memory 62, 63 so that the processor components can access it more quickly. In one or more embodiments, the shared cache 99a, 99b may include one or more intermediate caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of caches, a last-level cache (LLC), and / or combinations thereof.

[0081] Although only two processing elements 70 and 80 are shown, it should be understood that the scope of these embodiments is not limited thereto. In other embodiments, one or more additional processing elements may reside within a given processor. Alternatively, one or more of these processing elements 70 and 80 may be elements other than a processor, such as an accelerator or a field-programmable gate array. For example, additional processing elements may include an additional processor identical to a first processor 70, an additional processor heterogeneous or asymmetric to a first processor 70, an accelerator (such as, for example, a graphics accelerator or digital signal processing (DSP) unit), a field-programmable gate array, or any other processing element. A wide variety of differences may exist between these processing elements 70 and 80 in terms of a spectrum of quality indicators including architecture, microarchitecture, thermal, power consumption characteristics, etc. These differences can be effectively manifested as asymmetry and heterogeneity between these processing elements 70 and 80. For at least one embodiment, these various processing elements 70 and 80 may reside in the same die package.

[0082] The first processing element 70 may further include memory controller logic (MC) 72 and point-to-point (PP) interfaces 76 and 78. Similarly, the second processing element 80 may include an MC 82 and PP interfaces 86 and 88. As shown in FIG10, MCs 72 and 82 couple the processors to individual memories, namely a memory 62 and a memory 63, which may be part of the main memory locally connected to the individual processors. Although MCs 72 and 82 are illustrated as being integrated into the processing elements 70 and 80, in alternative embodiments, the MC logic may be discrete logic external to the processing elements 70 and 80, rather than being integrated therein.

[0083] The first processing element 70 and the second processing element 80 can be coupled to an I / O subsystem 90 via PP interconnects 76 and 86, respectively. As shown in FIG. 10, the I / O subsystem 90 includes PP interfaces 94 and 98. Furthermore, the I / O subsystem 90 includes an interface 92 for coupling the I / O subsystem 90 to a high-performance graphics engine 64. In one embodiment, a bus 73 may be used to couple the graphics engine 64 to the I / O subsystem 90. Alternatively, a point-to-point interconnect may be used to couple these components.

[0084] Therefore, the I / O subsystem 90 can be coupled to a first bus 65 via an interface 96. In one embodiment, the first bus 65 may be a peripheral component interconnect (PCI) bus, or a bus such as a fast PCI bus or another third-generation I / O interconnect bus, but the scope of the embodiment is not limited thereto.

[0085] As shown in Figure 10, various I / O devices 65a (e.g., biometric scanners, speakers, cameras, and / or sensors) can be coupled to the first bus 65, accompanied by a bus bridge 66 that can couple the first bus 65 to a second bus 67. In one embodiment, the second bus 67 may be a low pin count (LPC) bus. Various devices can be coupled to the second bus 67, and in one embodiment, these devices include, for example, a keyboard / mouse 67a, a communication device 67b, and a data storage unit 68 such as a magnetic disk or other mass storage device that may include code 69. The code 69 shown can implement one or more aspects of the above-described programs, including program 500 and / or program 520. The code 69 shown may be similar to the code 42 discussed previously (Figure 9). Furthermore, an audio I / O 67c can be coupled to the second bus 67, and a battery 61 can power the computing system 60. The system 60 can realize one or more aspects of the system 100, the neural network 110, the neural network structure 200, the network depth calibration structure 300, the network depth relay structure 310, the network depth calibration layer 350, the MGR unit 360, the MGR unit 380, the feature dimension calibration structure 400, the feature dimension relay structure 410, the feature dimension calibration slice 450, the MGR unit 460, and / or the MGR unit 480 as described herein with reference to Figures 1A-1B, 2A-2D, 3A-3D, and 4A-4D.

[0086] Note that other embodiments are conceivable. For example, instead of the point-to-point architecture of FIG10, a system could implement a multipoint bus or another such communication topology. Furthermore, the elements of FIG10 could alternatively be partitioned using more or fewer integrated chips than shown in FIG10.

[0087] Embodiments of each of the above-described systems, devices, components, and / or methods, including system 100, neural network 110, neural network structure 200, network depth calibration structure 300, network depth relay structure 310, network depth calibration layer 350, MGR unit 360, MGR unit 380, feature dimension calibration structure 400, feature dimension relay structure 410, feature dimension calibration slice 450, MGR unit 460, MGR unit 480, program 500 and / or program 520, and / or any other system component, can be implemented in hardware, software, or any suitable combination thereof. For example, hardware implementations may include combinatorial logic such as, for example, PLA, FPGA, CPLD; or fixed-function logic hardware using circuit technologies such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0088] Alternatively or additionally, all or part of the aforementioned systems and / or components and / or methods may be implemented as a set of logical instructions in one or more modules, stored in a machine or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., for execution by a processor or computing device. For example, the computer program code performing the operations of such components may be written in any combination of one or more operating system (OS) suitable / appropriate programming languages, including object-oriented programming languages ​​such as Python, Perl, Java, SmartTalk, C++, C#, etc., and traditional procedural programming languages ​​such as the "C" programming language or similar languages. Additional notes and examples:

[0089] Example 1 includes a computing system comprising a processor and a memory coupled to the processor, the memory storing a neural network comprising a plurality of convolutional layers, a network depth relay structure comprising a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers, and a feature dimension relay structure comprising a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

[0090] Example 2 includes the computing system of Example 1, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

[0091] Example 3 includes the computing system of Example 2, wherein each feature dimension calibration slice includes a second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a separate second MGR unit of one of the previous feature dimension calibration units.

[0092] Example 4 includes the computing system of Example 3, wherein each of the first MGR unit and the second MGR unit contains a modified long short-term memory (LSTM) cell.

[0093] Example 5 includes the computational system of Example 4, wherein each network depth calibration layer further includes a first global average pooling (GAP) function operating on a feature map, a first normalization (STD) function operating on the feature map, and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and the first cell state signal generated by the first MGR unit, and wherein each feature dimension calibration slice further includes a second GAP function operating on a feature slice, a second STD function operating on the feature slice, and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and the second cell state signal generated by the second MGR unit.

[0094] Example 6 includes any of Examples 1-5 of the computing system, wherein the feature dimension relay structure is coupled along a time dimension of calibrated features.

[0095] Example 7 includes a semiconductor device comprising one or more substrates and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more configurable logic or fixed-function hardware logic, the logic coupled to the one or more substrates comprising a neural network comprising a plurality of convolutional layers; a network depth relay structure comprising a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and a feature dimension relay structure comprising a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

[0096] Example 8 includes the device of Example 7, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

[0097] Example 9 includes the device of Example 8, wherein each feature dimension calibration slice includes the second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to the previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a separate second MGR unit of one of the previous feature dimension calibration units.

[0098] Example 10 includes the device of Example 9, wherein each of the first MGR unit and the second MGR unit comprises a modified long short-term memory (LSTM) cell.

[0099] Example 11 includes the device of Example 10, wherein each network depth calibration layer further includes a first global average pooling (GAP) function operating on a feature map, a first normalization (STD) function operating on the feature map, and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and the first cell state signal generated by the first MGR unit, and wherein each feature dimension calibration slice further includes a second GAP function operating on a feature slice, a second STD function operating on the feature slice, and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and the second cell state signal generated by the second MGR unit.

[0100] Example 12 includes the device of any of Examples 7-11, wherein the feature dimension relay structure is combined along a time dimension of calibrated features.

[0101] Example 13 includes the device of Example 7, wherein the logic coupled to one or more substrates includes a transistor channel region located within one or more substrates.

[0102] Example 14 includes at least one computer-readable storage medium containing a set of instructions that, when executed by a computing system, cause the computing system to generate a plurality of convolutional layers in a neural network, arrange a network depth relay structure in the neural network including a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers, and a feature dimension relay structure including a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

[0103] Example 15 includes the at least one computer-readable storage medium of Example 14, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

[0104] Example 16 includes the at least one computer-readable storage medium of Example 15, wherein each feature dimension calibration slice includes a second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a respective second MGR unit of one of the previous feature dimension calibration units.

[0105] Example 17 includes the at least one computer-readable storage medium of Example 16, wherein each of the first MGR unit and the second MGR unit contains a modified long short-term memory (LSTM) cell.

[0106] Example 18 includes the at least one computer-readable storage medium of Example 17, wherein each network depth calibration layer further includes a first global average pooling (GAP) function operating on a feature map, a first normalization (STD) function operating on the feature map, and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and the first cell state signal generated by the first MGR unit, and wherein each feature dimension calibration slice further includes a second GAP function operating on a feature slice, a second STD function operating on the feature slice, and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and the second cell state signal generated by the second MGR unit.

[0107] Example 19 includes any of Examples 14-18 of the at least one computer-readable storage medium, wherein the feature dimension relay structure is coupled along a time dimension of calibrated features.

[0108] Example 20 includes a method comprising generating a plurality of convolutional layers in a neural network, arranging a network depth relay structure in the neural network including a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers, and arranging a feature dimension relay structure in the neural network including a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

[0109] Example 21 includes the method of Example 20, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

[0110] Example 22 includes the method of Example 21, wherein each feature dimension calibration slice includes a second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a respective second MGR unit of one of the previous feature dimension calibration units.

[0111] Example 23 includes the method of Example 22, wherein each of the first MGR unit and the second MGR unit comprises a modified long short-term memory (LSTM) cell.

[0112] Example 24 includes the method of Example 23, wherein each network depth calibration layer further includes a first global average pooling (GAP) function operating on a feature map, a first normalization (STD) function operating on the feature map, and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and the first cell state signal generated by the first MGR unit, and wherein each feature dimension calibration slice further includes a second GAP function operating on a feature slice, a second STD function operating on the feature slice, and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and the second cell state signal generated by the second MGR unit.

[0113] Example 25 includes the method of any of Examples 20-24, wherein the feature dimension relay structure is combined along a time dimension of calibrated features.

[0114] Example 26 includes a device having components for performing the method as described in any of requests 20-24.

[0115] Therefore, the technique described in this paper improves the performance of computing systems used in image sequence / video analysis tasks, significantly accelerating training speed and improving accuracy. This technique is applicable to any number of computing scenarios, including, for example, deploying deep video models on edge / cloud devices and in high-performance distributed / parallel computing systems.

[0116] The embodiments are applicable to all types of semiconductor integrated circuit (“IC”) chips. Examples of these IC chips include, but are not limited to, processors, controllers, chipset assemblies, PLAs, memory chips, network chips, system-on-a-chip (SoC), SSD / NAND controller ASICs, and so on. Furthermore, in some of these figures, signal lines are represented by lines. Some may differ to indicate more constituent signal paths; have a numerical label to indicate multiple constituent signal paths; and / or have arrows at one or more endpoints to indicate the primary direction of information flow. However, this should not be construed as a limiting approach. Rather, such added details may be used in conjunction with one or more exemplary embodiments to facilitate a clearer understanding of a circuit. Any represented signal line, whether or not it carries additional information, may actually contain one or more signals that can propagate in multiple directions and can be implemented using any suitable type of signaling scheme, such as digital or analog lines implemented using differential pairs, fiber optic cables, and / or single-ended lines.

[0117] Instance dimensions / models / values / ranges may have been given, but the embodiments are not limited thereto. As manufacturing technologies (e.g., photolithography) mature over time, it is expected that smaller devices can be manufactured. Furthermore, for the sake of simplicity of description and discussion, and to avoid obscuring certain aspects of these embodiments, well-known power / ground connections to the IC chip and other components may be shown or not shown in these figures. Additionally, to avoid obscuring the embodiments, arrangements may be shown in block diagram form, and it is also taken into consideration that details regarding the implementation of such block diagram arrangements are highly dependent on the platform in which the embodiment will be implemented; that is, such details should be within the capabilities of those skilled in the art. In the case of specific details (e.g., circuitry) set forth for the purpose of describing exemplary embodiments, those skilled in the art should understand that embodiments can be implemented without these specific details or with modifications to them. This description is therefore considered illustrative rather than restrictive.

[0118] As used herein, the term "coupled" can refer to any type of direct or indirect relationship between the components under discussion and can be applied to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections, including logical connections via intermediate components (e.g., device A can be coupled to device C via device B). Furthermore, unless otherwise stated, the terms "first," "second," etc., used herein may be for the purpose of discussion only and do not have a specific temporal or chronological meaning.

[0119] As used in this application and in those claims, the list of items connected by the term "one or more" can represent any combination of the listed items. For example, the term "one or more of A, B or C" can mean A, B, C; A and B; A and C; B and C; or A, B and C.

[0120] Those skilled in the art will understand from the foregoing description that these broad techniques of the embodiments can be implemented in a variety of forms. Therefore, although these embodiments have been described in conjunction with specific examples, the true scope of these embodiments should not be so limited, as other modifications will become apparent to those skilled in the art upon reading the accompanying drawings, specification, and appended claims.

[0121] 10: System 11:SoC 12: Host Processor 14:IMC 16: I / O Subsystem 17:I / O device 20: System Memory 22: Storage 24: Network Controller 26: Graphics Processor 27: AI Accelerator 28: Instructions 30: Semiconductor equipment 32:Substrate 34: Logic 40: Processor core 41, 62, 63: Memory 42,69: Code 43: Frontend 44: Decoder 46: Rename the temporary register 48: Scheduling 50: Execution Logic 55-1:EU-1 55-2:EU-2 58: Backend 59: The Logic of Retirement 60: Multiprocessor System 61: Battery 64: High-performance graphics engine 65: First busbar 65a: I / O device 66: Busbar Bridge 67: Second busbar 67a: Keyboard / Mouse 67b: Communication device 67c: Audio I / O 68: Data storage 70: First processing element 71: Point-to-point interconnection 72,82:MC 73: Busbar 74a, 74b, 84a, 84b: Processor cores 76,78,86,88,94,98:PP 80: Second processing element 90: I / O Subsystem 92,96:I / F 99a, 99b: Shared cache 100: System 110: Neural Networks 120: Convolutional layer 130: MGR structure 132: Network Deep Relay Structure 134: Feature Dimension Relay Structure 140: Input image sequence 150: Output feature map 200: Neural Network Structure 210: Module (k-1) 220: Module (k) 221,224,271,291,302,304,306,402: Convolution 222,272:FCAA-D 223,226,273,293,303,305,307,408: Activation 225,292:FCAA-T 227,274,294: Convolution / Regularization / Activation 230: Module (k+1) 240, 260, 280: Schematic diagram 300: Network Depth Calibration Structure 310: Network Deep Relay Structure 312:FCAA-D (Layer in module k-1) 314:FCAA-D (Layer in module k) 316:FCAA-D (Layer in module k+1) 350: Network Depth Calibration Layer 352,452:GAP 354,454:MGR(k) 356,456:STD 358,458:LNT 360, 380, 460, 480: Yuan gate control relay unit 370, 470: (Modified) LSTM cells 400: Feature Dimension Calibration Structure 404, 406: Slices 410: Feature Dimension Relay Structure 412:FCAA-T (slice t-1) 414:FCAA-T (slice t) 416:FCAA-T(slice t+1) 450: Feature Dimension Calibration Slice 500, 520: Method 502-506, 522-534: Squares 602, 604, 612, 614, 622, 624: Labels

Claims

1. A computing system for image sequence or video analysis, comprising: a processor; and a memory coupled to the processor, the memory storing a neural network comprising: a plurality of convolutional layers; a network depth relay structure comprising a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and a feature dimension relay structure comprising a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

2. The computing system of claim 1, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

3. The computing system of claim 2, wherein each feature dimension calibration slice includes a second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a separate second MGR unit of one of the previous feature dimension calibration units.

4. The computing system of claim 3, wherein each of the first MGR unit and the second MGR unit comprises a modified long short-term memory (LSTM) cell.

5. The computing system of claim 4, wherein each network depth calibration layer further comprises: a first global average pooling (GAP) function operating on a feature map; a first normalization (STD) function operating on the feature map; and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and based on the first cell state signal generated by the first MGR unit; and wherein each feature dimension calibration slice further comprises: a second GAP function operating on a feature slice; a second STD function operating on the feature slice; and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and based on the second cell state signal generated by the second MGR unit.

6. A computational system as requested in any of items 1-5, wherein the relay structure in the feature dimension is coupled along a time dimension of calibrated features.

7. A semiconductor device for image sequence or video analysis, comprising: one or more substrates; and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic or fixed-function hardware logic, the logic coupled to the one or more substrates comprising a neural network comprising: a plurality of convolutional layers; a network depth relay structure comprising a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and a feature dimension relay structure comprising a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

8. The device of claim 7, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

9. The device of claim 8, wherein each feature dimension calibration slice includes a second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a respective second MGR unit of one of the previous feature dimension calibration units.

10. The device of claim 9, wherein each of the first MGR unit and the second MGR unit comprises a modified long short-term memory (LSTM) cell.

11. The device of claim 10, wherein each network depth calibration layer further comprises: a first global average pooling (GAP) function operating on a feature map; a first normalization (STD) function operating on the feature map; and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and based on the first cell state signal generated by the first MGR unit; and wherein each feature dimension calibration slice further comprises: a second GAP function operating on a feature slice; a second STD function operating on the feature slice; and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and based on the second cell state signal generated by the second MGR unit.

12. The device of any of the requests 7-11, wherein the relay structure of the feature dimension is combined along a time dimension of calibrated features.

13. The device of claim 7, wherein the logic coupled to the one or more substrates includes a transistor channel region located within the one or more substrates.

14. A computer-readable storage medium for image sequence or video analysis containing at least one set of instructions, which, when executed by a computing system, causes the computing system to: generate a plurality of convolutional layers in a neural network; arrange a network depth relay structure in the neural network including a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and arrange a feature dimension relay structure in the neural network including a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

15. At least one computer-readable storage medium as claimed in claim 14, wherein each network depth calibration layer includes a first meta gate relay (MGR) unit, and wherein each network depth calibration layer is coupled to a preceding network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the preceding network depth calibration layer.

16. At least one computer-readable storage medium as claimed in claim 15, wherein each feature dimension calibration slice includes a second meta gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a respective second MGR unit of one of the previous feature dimension calibration slices.

17. At least one computer-readable storage medium as claimed in claim 16, wherein each of the first MGR unit and the second MGR unit comprises a modified long short-term memory (LSTM) cell.

18. At least one computer-readable storage medium as claimed in claim 17, wherein each network depth calibration layer further comprises: a first global average pooling (GAP) function operating on a feature map; a first normalization (STD) function operating on the feature map; and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on a first hidden state signal generated by the first MGR unit and a first cell state signal generated by the first MGR unit; and wherein each feature dimension calibration slice further comprises: a second GAP function operating on a feature slice; a second STD function operating on the feature slice; and a second LNT function operating on an output of the second STD function, the second LNT function being based on a second hidden state signal generated by the second MGR unit and a second cell state signal generated by the second MGR unit.

19. At least one of the requests 14-18, a computer-readable storage medium, wherein the relay structure in the feature dimension is coupled along a time dimension of calibrated features.

20. A method for analyzing image sequences or videos, comprising: generating a plurality of convolutional layers in a neural network; arranging a network depth relay structure in the neural network including a plurality of network depth calibration layers, wherein each network depth calibration layer is coupled to an output of one of the plurality of convolutional layers; and arranging a feature dimension relay structure in the neural network including a plurality of feature dimension calibration slices, wherein the feature dimension relay structure is coupled to an output of another layer of the plurality of convolutional layers.

21. The method of claim 20, wherein each network depth calibration layer includes a first meta-gateway relay (MGR) unit, and wherein each network depth calibration layer is coupled to a previous network depth calibration layer via a first hidden state signal and a first cell state signal, each of the first hidden state signal and the first cell state signal being generated by a respective first MGR unit of the previous network depth calibration layer.

22. The method of claim 21, wherein each feature dimension calibration slice includes a second meta-gate relay (MGR) unit, and wherein each feature dimension calibration slice is coupled to a previous feature dimension calibration slice via a second hidden state signal and a second cell state signal, each of the second hidden state signal and the second cell state signal being generated by a respective second MGR unit of one of the previous feature dimension calibration units.

23. The method of claim 22, wherein each of the first MGR unit and the second MGR unit comprises a modified long short-term memory (LSTM) cell.

24. The method of claim 23, wherein each network depth calibration layer further comprises: a first global average pooling (GAP) function operating on a feature map; a first normalization (STD) function operating on the feature map; and a first linear transformation (LNT) function operating on an output of the first STD function, the first LNT function being based on the first hidden state signal generated by the first MGR unit and based on the first cell state signal generated by the first MGR unit; and wherein each feature dimension calibration slice further comprises: a second GAP function operating on a feature slice; a second STD function operating on the feature slice; and a second LNT function operating on an output of the second STD function, the second LNT function being based on the second hidden state signal generated by the second MGR unit and based on the second cell state signal generated by the second MGR unit.

25. The method of any of requests 20-24, wherein the relay structure in the feature dimension is combined along a time dimension of calibrated features.

Images