Real-time salient target detection in images and videos

Abstract

The invention relates to real-time salient target detection in images and videos. The machine learning model may generate an output such as significant object detection. However, a high performance large scale model has a high computational cost, making it difficult for the model to deploy on resource-constrained devices and make predictions in real-time. Differential convolutional reparameterization techniques may be used to fuse parameter sets learned during training time into a single parameter set to be used at inference time or at deployment. The technique can improve performance without adding a large number of parameters. In addition, prediction can be made using temporal information using an efficient and efficient spatio-temporal differential convolution block. The spatio-temporal differential convolution block may also apply differential convolution re-parameterization techniques to reduce the need to increase parameters to achieve desired performance.

Description

Background Art

[0001] Machine learning models (e.g., deep learning models, deep neural networks, convolutional neural networks, transformer-based models, etc.) are used in various artificial intelligence and machine learning applications, such as computer vision, speech recognition, and natural language processing. Machine learning models can receive and process images, videos, audio, speech, text, etc. Machine learning models can achieve high accuracy, but at the cost of high computational cost. Some machine learning models may have millions of parameters. Running such a machine learning model to make predictions will involve performing many multiply-accumulate operations and many read and write data access operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, the same reference numerals designate the same structural elements. In the figures of the accompanying drawings, the embodiments are illustrated by way of example and not by way of limitation.

[0003] Figure 1 An exemplary machine learning model for processing one or more inputs (e.g., videos) and generating one or more outputs according to some embodiments of the present disclosure is shown.

[0004] Figure 2 An exemplary machine learning model for processing one or more inputs (e.g., images) and generating one or more outputs according to some embodiments of the present disclosure is shown.

[0005] Figure 3A-3D Different pixel pair selection strategies according to some embodiments of the present disclosure are shown.

[0006] Figure 4 Difference convolution re-parameterization according to some embodiments of the present disclosure is shown.

[0007] Figure 5A-Figure 5B An exemplary implementation of a spatiotemporal difference module according to some embodiments of the present disclosure is shown.

[0008] Figure 6 The operation of a height-time spatiotemporal convolution block, for example during training time, is shown in accordance with some embodiments of the present disclosure.

[0009] Figure 7 The operation of a width-time spatiotemporal convolution block, for example during training time, is shown in accordance with some embodiments of the present disclosure.

[0010] Figure 8 Depicted are spatiotemporal feature maps and processing by operators using different spatiotemporal convolution types according to some embodiments of the present disclosure.

[0011] Fig. 9 Depicted are exemplary implementations of layers in a backbone level according to some embodiments of the present disclosure.

[0012] Fig.10 Depicted is an exemplary implementation of a dilated convolution module in accordance with some embodiments of the present disclosure.

[0013] Fig.11 Depicted are exemplary implementations of a spatial attention module in accordance with some embodiments of the present disclosure.

[0014] Fig.12 is a flow chart illustrating a method for generating one or more outputs according to some embodiments of the present disclosure.

[0015] Fig.13 is a flowchart illustrating a method for determining parameters of a machine learning model according to some embodiments of the present disclosure.

[0016] Fig.14 is a flow chart illustrating a method for extracting spatiotemporal information according to some embodiments of the present disclosure.

[0017] Fig.15 is a flowchart illustrating a method for determining parameters of a machine learning model according to some embodiments of the present disclosure.

[0018] Fig.16 is a block diagram of an exemplary computing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION Overview

[0019] A machine learning model can receive and / or process one or more inputs and generate one or more outputs. The output can include predictions and / or inferences. Examples of outputs in computer vision can include edge detection, object detection, object classification, image segmentation, significant object detection, foreground detection, background removal, etc. Some machine learning models can process images and make predictions and / or inferences. Some machine learning models can process images of videos and make predictions and / or inferences. For significant object detection, machine learning models can make pixel-by-pixel predictions, and many machine learning models are computationally expensive. The highest-performance large-scale models have high computational costs, making it difficult for the model to be deployed on resource-constrained devices (e.g., edge devices with limited computing power) and difficult to make predictions in real time. It is a challenging technical task to implement a lightweight, efficient, but effective model for significant object detection in images and videos without significantly sacrificing performance.

[0020] Some machine learning models, if implemented efficiently, may be suitable for making inferences and / or generating predictions in real time with high accuracy. Real-time salient object detection in images and videos as a computer vision task can benefit applications such as object recognition and detection, visual tracking, retrieval, image and video compression, and semantic segmentation. These applications can be found in contexts such as self-navigating vehicles, self-driving vehicles, autonomous driving, industrial automation, industrial monitoring, home automation, and home monitoring.

[0021] In some embodiments, the machine learning model may include a backbone having one or more levels and one or more branches, each branch receiving and / or processing the corresponding output of the level. One or more levels may include one or more spatial difference layers that utilize different convolution types without significantly increasing the number of parameters. As used herein, parameters may include kernel weights used in a part of the machine learning model, tuning parameters that can affect the behavior of a part of the machine learning model, internal parameters or model parameters of the model that can be learned or estimated from training data. Different convolution types can extract diverse spatial features by checking pixel intensities in a standard manner and checking the pixel intensity differences of pixel pairs selected using different pixel pair selection strategies. The pixel intensity differences of pixel pairs can enrich feature representations with image gradient information. Diverse spatial features may be beneficial for tasks such as significant object detection involving predicted edges, object boundaries, and internal areas of objects.

[0022] If the machine learning model is to make predictions and / or inferences about the video, one or more of the branches can contain a spatiotemporal differential convolution module, which can efficiently process temporal information and extract spatiotemporal features. The spatiotemporal differential convolution module can utilize different convolution types in a similar manner to the spatial differential layer without significantly increasing the number of parameters.

[0023] The resulting implementation of the machine learning model can be lightweight enough and can achieve good performance with real-time inference speed on embedded devices.

[0024] Different convolution types can be exploited using differential convolution reparameterization techniques without significantly increasing the number of parameters in a machine learning model. The technique can capture the effect of using multiple operators with different convolution types at the computational cost of a single convolution operator with no additional parameters. The technique can add pixel differential convolution types to standard convolution operators without adding computational cost and parameters. During training time, parallel operators that implement different convolution types can be trained using training data. The differential convolution reparameterization technique can fuse sets of parameters learned during training time (e.g., parameters in parallel operators) into a single fused set of parameters (e.g., in a single operator) to be used at inference time or at deployment time. The technique can improve performance by increasing the model capacity for capturing intensity and gradient cues without adding parameters and computational complexity.

[0025] When making predictions and / or inferences based on videos (such as video salient object detection), machine learning models can benefit from extracting information such as temporal consistency information and temporal cues. Optical flow, recurrent neural networks, convolutional long short-term memory networks, and 3D convolution can be used to extract temporal information. However, optical flow, recurrent neural networks, and convolutional long short-term memory networks may not be suitable for real-time processing. Although 3D convolution can be faster, the representation capacity may be limited, and therefore may affect the performance of video salient object detection.

[0026] To address some of these issues, an efficient and effective spatiotemporal differential convolution block can be used to make predictions using temporal information. The spatiotemporal differential convolution block can extract spatiotemporal information using information on one or more two-dimensional spatiotemporal planes. The spatiotemporal differential convolution block can include a height-time spatiotemporal convolution block that processes one or more local regions across the time dimension and the height dimension, and uses pixel differences of selected pixel pairs to detect local spatiotemporal changes. The spatiotemporal differential convolution block can include a width-time spatiotemporal convolution block that processes one or more local regions across the time dimension and the width dimension, and uses pixel differences of selected pixel pairs to detect local spatiotemporal changes. The spatiotemporal differential convolution block can complement the three-dimensional convolution, enrich the overall feature representation, and increase the model representation capacity.

[0027] The spatiotemporal differential convolution block can also apply pixel differential convolution techniques and differential convolution reparameterization techniques to one or more of the following convolution blocks: height-time spatiotemporal convolution blocks and / or width-time spatiotemporal convolution blocks. For example, a height-time spatiotemporal convolution block can be trained using multiple parallel operators that implement different convolution types, and the parameter sets corresponding to the parallel operators can be fused into a fused parameter set for use in a single operator during inference time or at deployment. This technique can improve performance by increasing the model capacity for capturing additional temporal consistency information and temporal saliency cues without adding parameters and computational complexity. Example machine learning model and its parts

[0028] A machine learning model is a computer-implemented system that can perform one or more tasks. A machine learning model can take input and generate output for the task at hand. Using and implementing machine learning models may involve supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Machine learning models can be implemented in different ways. Machine learning models can contain one or more of the following algorithms: artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks, Gaussian processes, genetic algorithms, etc.

[0029] Artificial neural networks may include one or more layers, modules, networks, blocks and / or operators that transform inputs into outputs. In some embodiments, layers, modules, networks, blocks or operators may include one or more processing units and / or one or more processing nodes. A processing unit may receive one or more inputs, perform a processing function or operation, and generate one or more outputs. Processing units may be interconnected to form a network. In some cases, a processing unit or node may be referred to as a neuron. Different types of processing units or nodes may be distinguished by the processing function / operation being performed by the processing unit or node. A processing unit may include one or more parameters. The parameters may be trained or learned. A processing unit may include one or more hyperparameters. Hyperparameters may be set or tuned by one or more users of a machine learning model.

[0030] One type of processing unit is a convolution operator. A processing unit applies a convolution operation to an input and generates an output. A convolution operation can extract features from an input and output these features as output. A convolution operation can transform an input and generate an output. A processing unit can convolve an input with a kernel to generate an output. A kernel can contain a matrix. A kernel can encode a function or operation that can transform an input. A kernel can contain parameters or values ​​that can be trained or learned. A processing unit can calculate an inner product (e.g., a dot product) by a sliding / moving window that captures a local area or patch of the input, and sums and / or accumulates the inner products to generate an output. The inner product can be continuously calculated across the input matrix as the sliding / moving window moves across the input matrix. The convolution operator may be defined by the size of the kernel, e.g., a 1×1 convolution operator has a kernel size of 1×1, a 2×2 convolution operator has a kernel size of 2×2, a 3×3 convolution operator has a kernel size of 3×3, a 4×4 convolution operator has a kernel size of 4×4, a 5×5 convolution operator has a kernel size of 5×5, and so on. The distance that the window slides / moves may be set or defined by the stride of the convolution operator. In some cases, the convolution operator may not apply padding and use the input matrix as is. In some cases, the convolution operator may apply half padding and pad around a portion of the input matrix. In some cases, the convolution operator may apply full padding and pad around the input matrix.

[0031] An exemplary layer, module, block, or operator may include a dilated convolution block, which may increase the ability to extract features at various scales. A dilated convolution block may expand a kernel by inserting gaps between weights in the kernel. A dilated convolution module may have a dilation rate that indicates how much the kernel is widened. Parameters in the kernel may be trained or learned.

[0032] Another type of processing unit is a transformer cell or block. A transformer cell can be used in a transformer block. A transformer cell can implement an attention mechanism to extract dependencies between different parts of the input of the transformer cell. A transformer cell can receive an input and generate an output representing the saliency or attention of various parts of the input. A transformer cell can include query weights, key weights, and value weights as parameters that can be trained or learned. The transformer cell can apply these parameters to extract relationship information between different parts of the input of the transformer cell. A transformer block can include multiple transformer cells in parallel. A transformer block can include multiple transformer cells in series.

[0033] Another type of processing unit is an activation unit or block. An activation block can implement or apply an activation function (e.g., a sigmoid function, a nonlinear function, a hyperbolic tangent function, a rectified linear unit, a leaky rectified linear unit, a parameterized rectified linear unit, a sigmoid linear unit, an exponential linear unit, a proportional exponential linear function, a logical activation function, a Heaviside activation function, an identity function, a binary step function, a soft step function, a Gaussian error linear unit, a Gaussian function, a soft addition function, etc.) to the input of the activation block and generate an output. An activation block can be used to map the input of a block to a value between 0 and 1. An activation block can be used to map the input of a block to 0 or 1. An activation block may introduce nonlinearity. An activation block can learn complex decision boundaries. One or more parameters of an activation function can be trained or learned.

[0034] An exemplary layer, module, block, or operator may include an upsampling block. An upsampling block may increase the size of an input feature or feature map. An upsampling block may synthesize values ​​that may be added to an input feature or feature map to increase the size of an upsampled output feature or feature map.

[0035] An exemplary layer, module, block or operator may include a downsampling block. The downsampling block may perform downsampling on a feature map or feature generated by a stage, which may improve the operating efficiency of a machine learning model. The downsampling block may include a pooling layer, which may receive a feature map at its input and apply a pooling operation to the feature map. The output of the pooling layer may be provided or input to a subsequent stage for further processing. The pooling operation may reduce the size of the feature map while retaining their (important) characteristics. Therefore, the pooling operation may improve efficiency and may avoid overlearning. The pooling layer may perform a pooling operation by average pooling (calculating the average value of each patch on the feature map), maximum pooling (calculating the maximum value of each patch of the feature map), or a combination of both. The size of the output of the pooling layer is smaller than the size of the feature map provided as the input of the pooling layer. In some embodiments, the pooling operation is 2×2 pixels applied with a stride of 2 pixels, so that the pooling operation reduces the size of the feature map by 1 / 2, for example, the number of values ​​or pixels in the feature map is reduced to one quarter of the size. In some embodiments, a pooling layer applied to a 6×6 feature map results in a 3×3 output pooled feature map.

[0036] Exemplary layers, modules, blocks, or operators may include a projection layer. The projection layer may transform the input features into a new space, such as a space that is suitable, information-rich, and / or useful for the task being performed by the downstream module (for processing by the downstream module). The projection layer may include a dense layer or a fully connected layer, in which each neuron (e.g., a node in a neural network) is connected to each neuron in the previous layer. The projection layer may generate and / or output a set of new features that are more abstract or advanced than the features in the input. The projection layer may implement one or more 1×1 convolution operations, in which the projection layer may convolve the input features with a filter of size 1×1 (e.g., a filter with zero padding and a stride of 1). The projection layer may implement channel-by-channel pooling or feature map pooling. The projection layer may reduce the dimensionality of the input features by pooling features across channels. The projection layer may implement a 1×1 filter to create a stacked linear projection of feature maps. The projection layer may implement a 1x1 filter to increase the number of feature maps. The projection layer may implement a 1×1 filter to reduce the number of channels. The projection layer may make the feature map compatible with subsequent processing layers, modules, blocks, or operators. The projection layer may ensure that an element-wise addition operation may be performed to add the output of the projection layer and another feature map. The projection layer may ensure that the dimensions of the output of the projection layer match the dimensions of the feature maps that are added together element-wise. The parameters of the projection layer may be trained or learned.

[0037] An exemplary block may include an adder block. An adder block may perform element-by-element addition of two or more inputs to generate an output. An adder block may be an exemplary block that may merge and / or combine two or more inputs together. Addition and summation may be synonymous. An adder block may be replaced by a cascade block.

[0038] An exemplary block may include a multiplier block. A multiplier block may perform an element-wise multiplication of two or more inputs to generate an output. A multiplier block may determine a Hadamard product.

[0039] Exemplary blocks may include cascade blocks. A cascade block may perform the cascade of two or more inputs to generate an output. A cascade block may append vectors and / or matrices to the input to form new vectors and / or matrices. Vector cascades may be appended to form larger vectors. Matrix cascades may be performed horizontally, vertically, or in a merged manner. Horizontal matrix cascades may be performed by cascading matrices (which have the same height) in the input width direction. Vertical matrix cascades may be performed by cascading matrices (which have the same width) in the input height direction. A cascade block may be an exemplary block that may merge and / or combine two or more inputs. When the two or more inputs do not have the same dimension, a cascade block may be suitable. When it is desired to keep the two or more inputs unchanged or complete (e.g., without losing information), a cascade block may be suitable. A cascade block may be replaced by an adder block.

[0040] Data in a machine learning model (such as inputs, intermediate outputs, and outputs) may be stored in a non-transitory computer-readable storage medium. Parameters in a machine learning model may be stored in a non-transitory computer-readable storage medium. Hyperparameters in a machine learning model may be stored in a non-transitory computer-readable storage medium. Instructions encoding the operation of a machine learning model and executable by one or more processors may be stored in a non-transitory computer-readable storage medium. Exemplary machine learning models for video processing and / or image processing

[0041] Figure 1 An exemplary machine learning model 100 for processing one or more inputs (e.g., videos) and generating outputs according to some embodiments of the present disclosure is shown. The machine learning model 100 can be implemented on a computing device, such as Fig.16 The computing device 1600 of the machine learning model 100 can be encoded and / or stored in one or more storage devices. Data of the machine learning model 100 (such as input data, output data, intermediate output data, intermediate input data, etc.) can be stored in the one or more storage devices. Instructions corresponding to the processing operations of the machine learning model 100 can be stored in the one or more storage devices. The processing operations can be executed (execute or perform) by the one or more processors.

[0042] The machine learning model 100 may receive and / or process one or more images 102, such as images of a video as input. The images 102 may include pixel intensity values ​​of one or more channels for each pixel in each image. The images 102 may include a sequence of images with timestamps. The images 102 may include a sequence of images across time. The images 102 may include multiple image frames per second. The images 102 may include images captured by an imaging device. The images 102 may include color camera images, grayscale camera images, black and white camera images, infrared color images, depth camera images or depth maps, bird's-eye images, geographic map images, topographic maps, etc. The images 102 may include other spatial input data in the form of pixel values ​​across two dimensions.

[0043] The machine learning model 100 may generate and / or output one or more outputs 104. The outputs 104 may include predictions. The outputs 104 may include inferences. The outputs 104 may include pixel-by-pixel determinations. The outputs 104 may include determinations about individual pixels in the image 102. These determinations may include, for example, edge detection, object classification, foreground detection, background detection, salient object detection, image segmentation, etc. The outputs 104 may include a sequence of output predictions, wherein each output prediction in the sequence includes a prediction value across two dimensions. The prediction value may include a binary value indicating whether the pixel is part of a salient object in the image in the image 102. The prediction value may include a probability value indicating how likely the pixel is to be part of a salient object in the image in the image 102. The outputs 104 may directly correspond to the corresponding input image in the image 102.

[0044] The machine learning model 100 may include layers and modules coupled together to form a network. The machine learning model 100 may include a backbone 106. The backbone 106 may include one or more stages. The backbone 106 may include two or more stages. Figure 1 , backbone 106 may include four stages. Exemplary stages of backbone 106 are shown, such as stage 1101, stage 1102, stage 1103, and stage 1104. As shown, an exemplary implementation may include four stages. It is contemplated by the present disclosure that a suitable number of stages may be included as part of backbone 106. The stages of backbone 106 may generate outputs, including features and / or feature maps.

[0045] The machine learning model 100 may include one or more side structures 108. The side structure 108 may include one or more branches. The side structure 108 may include two or more branches. The branches may generate outputs, including features and / or feature maps. Figure 1 As depicted in , side structure 108 may include four branches. Exemplary branches in side structure 108 are shown, such as branch 1121, branch 1122, branch 1123, and branch 1124. Branches in side structure 108 may correspond to corresponding levels in backbone 106. Branches in side structure 108 may receive and / or process corresponding outputs (e.g., features or feature maps) of corresponding levels in backbone 106. The output of a branch that receives and / or processes the output of a level after a given level in backbone 106 may be combined and / or combined with a branch that processes the output of a given level. A branch may receive one or more feature maps generated by a level in backbone 106, and one or more feature maps generated by another branch. A branch may combine one or more feature maps generated by a level in backbone 106 with one or more feature maps generated by another branch. The output of a branch may include features and / or one or more feature maps. Combining or combining may include cascading. Combining or combining may include adding or summing. Combining or combining may include merging.

[0046] like Figure 1 As depicted in , backbone 106 may include stage 1101, stage 1102, stage 1103, and stage 1104. Each stage may have one or more layers or blocks to process the input of the stage and generate features or one or more feature maps as the output of the stage. The layers or blocks in the stage may extract spatial information. Figure 3A-3D , Figure 4 and Fig. 9 Details related to the training and implementation of the stages in the backbone 106 are described. In some embodiments, one or more stages in the backbone 106 may include one or more spatial difference layers, which may form a spatial difference network. Multiple spatial difference layers may be connected in series or stacked to form a spatial difference network. The spatial difference layers may perform convolution. The spatial difference layers may use one or more parameters (e.g., kernel values) to perform convolution. The one or more parameters may be trained based on different spatial convolution types. The spatial difference layers may have one or more parameters that are fused or based on a parameter set that is trained based on different convolution types or different spatial convolution types. The spatial difference layers may have one or more parameters that are trained based on different convolution types or different spatial convolution types. By Figure 3A-3D , Figure 4 , Fig. 9 and Fig.13 Additional details regarding spatially differential stratification are described.

[0047] like Figure 1 , side structure 108 may include branch 1121, branch 1122, branch 1123, and branch 1124. Stage 1101 may receive and / or process one or more images 102. Stage 1102 may receive and / or process the output of stage 1101. Stage 1103 may receive and / or process the output of stage 1102. Stage 1104 may receive and / or process the output of stage 1103. The outputs of the stages in backbone 106 may include features, such as feature maps. Details regarding the stages in backbone 106 are provided by Figure 3A-3B , Figure 4 and Fig.13 Description. The corresponding outputs of the stages in backbone 106 can be input into corresponding branches in side structure 108. Branch 1121 can receive and / or process the output of stage 1101. Branch 1122 can receive and / or process the output of stage 1102. Branch 1123 can receive and / or process the output of stage 1103. Branch 1124 can receive and / or process the output of stage 1104. As data is being processed through machine learning model 100, the network of stages in backbone 106 and corresponding branches in side structure 108 can further refine the features or feature maps being extracted.

[0048] Branch 1121 may generate and / or output one or more outputs 104. Branch 1122 may generate and / or output features or feature maps that may be combined or combined with branch 1121. Branch 1123 may generate and / or output features or feature maps that may be combined or combined with branch 1122. Branch 1124 may generate and / or output features or feature maps that may be combined or combined with branch 1123. The combining or combining may occur by cascading, such as using a cascade block 1141 in branch 1121. The features or feature maps at the output of branch 1122 may be combined or combined with branch 1121 by cascading the features or feature maps at the output of branch 1122 with the features or feature maps in branch 1121 (e.g., the output of stage 1101 or a transformed version of the output of stage 1101) at a cascade block 1141 in branch 1121. The features or feature maps at the output of branch 1123 may be combined or combined with branch 1122 by cascading the features or feature maps at the output of branch 1123 with the features or feature maps in branch 1122 (e.g., the output of stage 1102 or a transformed version of the output of stage 1102) at cascade block 1142 in branch 1122. The features or feature maps at the output of branch 1124 may be combined or combined with branch 1123 by cascading the features or feature maps at the output of branch 1124 with the features or feature maps in branch 1123 (e.g., the output of stage 1103 or a transformed version of the output of stage 1103) at cascade block 1143 in branch 1123. Figure 1 Combining or combining the outputs from the stages and branches in the manner shown in can be used to aggregate features generated by multiple stages and branches. The branches can include cascade blocks or suitable blocks to perform the combination or combination of feature maps to generate a combined feature map. In some cases, the branches can include adder blocks to aggregate or sum feature maps to generate a combined feature map.

[0049] The branches in the side structure 108 can further extract information and / or features from the outputs of the levels in the backbone 106. The processing performed in the branches can implement spatial feature refinement or extract spatial features. The processing can implement temporal feature refinement or extract temporal features. The processing can implement spatiotemporal feature refinement or extract spatiotemporal features. The processing can implement attention feature refinement. The processing can include one or more modules, blocks and / or layers that can extract features. The modules, blocks and / or layers can be arranged in a suitable manner. The branches can implement the same or similar collection of one or more modules, blocks and / or layers. For simplicity, the processing that occurs in one of the branches (e.g., branch 1121) is described below. It should be understood that other branches can implement the same / similar processing.

[0050] In some embodiments, branch 1121 may receive and / or process output (e.g., features or feature maps) from stage 1101. Optionally, the output from stage 1101 may be provided as input to projection layer 120. A projection layer, such as projection layer 120, may transform the input features into a new space, such as a space that is suitable, informative, and / or useful for the task being performed by a downstream module (for processing by the downstream module).

[0051] Prior to additional processing, concatenation block 1141 may concatenate the output of stage 1101 or the output of projection layer 120 with the output of branch 1122. The features or feature maps at the output of branch 1122 may be concatenated with the features or feature maps (or transformed / derived versions thereof) in the output of stage 1101. The output of concatenation block 1141 (e.g., concatenated features or feature maps) may be processed by one or more modules.

[0052] Branch 1121 may include one or more spatiotemporal convolution modules, spatiotemporal difference modules (STDMs), or spatiotemporal difference convolution modules, such as STDM 122. A spatiotemporal difference module (e.g., STDM 122) may receive and / or process the cascaded features from the output of cascade block 1141. A spatiotemporal difference module (e.g., STDM 122) may extract spatiotemporal features. In some cases, a spatiotemporal difference module may include parameters trained based on different spatiotemporal convolution types. These parameters may be a fusion of parameter sets trained based on different spatiotemporal convolution types, or may be the result of a fusion of parameter sets trained based on different spatiotemporal convolution types. By Figure 5A-Figure 5B , Figure 6-Figure 8 , Fig.14 and Fig.15 Details related to the design, training, and implementation of the spatiotemporal difference module are described.

[0053] Branch 1121 may include one or more dilated convolution modules (DCMs) or compact dilated convolution modules, such as DCM 124. A dilated convolution module (e.g., DCM 124) may perform one or more dilated convolutions. A dilated convolution module (e.g., DCM 124) may increase the receptive field of a kernel without increasing the number of parameters or weights. A dilated convolution module (e.g., DCM 124) may receive and / or process the concatenated features from the output of concatenation block 1141. By Fig.10 Details related to the design and implementation of the dilated convolution module are described.

[0054] Branch 1121 may include a concatenation block 156 to combine and / or concatenate the output (e.g., features or feature maps) of DCM 112 and the output (e.g., features or feature maps) of STDM 122. The concatenated features from concatenation block 156 may be provided to projection layer 126. Projection layer 126 may transform input features or feature maps (e.g., concatenated features or concatenated feature maps from concatenation block 156) to a new space, such as a space that is suitable, informative, and / or useful for the task being performed by a downstream module (e.g., one or more spatial attention modules). In some cases, concatenation block 156 may be replaced by an adder block.

[0055] Branch 1121 may include one or more spatial attention modules (SAMs) or compact spatial attention modules, such as SAM 128. A spatial attention module (e.g., SAM 128) may remove background noise or perform background suppression. A spatial attention module (e.g., SAM 128) may receive and / or process concatenated features from the output of concatenation block 156 or features at the output of projection layer 126. A spatial attention module (e.g., SAM 128) may determine one or more convolutions to extract spatial correlation information at the input of the spatial attention module. By Fig.11 Details related to the design and implementation of the spatial attention module are described.

[0056] Modules in branches (such as DCM 124, STDM 122, and SAM 128 in branch 1121) can be arranged and / or connected differently in machine learning model 100. In some embodiments, SAM 128 can be arranged before DCM 124 and STDM 122. In some embodiments, SAM 128 can be arranged in parallel with DCM 124. In some embodiments, SAM 128 can be arranged in parallel with STDM 122. In some embodiments, SAM 128, DCM 124, and STDM 122 can be arranged in parallel with each other. The present disclosure contemplates other arrangements.

[0057] In some embodiments, features or feature maps generated by a branch in backbone 106 processing the output of a given stage may be upsampled by an upsampling block before the features or feature maps are combined with a branch in backbone 106 processing the output of a previous stage. For example, the output of SAM 130 in branch 1122 may be upsampled by an upsampling block (e.g., upsampling block 132). The upsampling block (e.g., upsampling block 132) may increase the size of the features or feature maps generated by SAM 130. The upsampling block (e.g., upsampling block 132) may synthesize values ​​that may be added to the features or feature maps generated by SAM 130 to increase the size.

[0058] The branches in the side structure 108 can propose the refinement of features at different levels of the machine learning model 100. In branch 1121, modules such as DCM 124 and STDM 122 implement spatial and spatiotemporal feature refinement respectively. The output features from DCM 124 and STDM 122 can be aggregated via cascade, for example at cascade block 156. The cascaded features can be further processed by SAM128 for background suppression. In order to encourage multi-level feature aggregation, the machine learning model 100 can implement a top-down approach for refining feature maps. Specifically, before being fed by a refinement module (e.g., DCM 124 and STDM 122), the output of the SAM at a given level is cascaded with the features extracted from the backbone of the previous level. For example, the output of SAM 130 in branch 1122 may be cascaded with features or feature maps from the output of stage 1101 (or a transformed version thereof at the output of projection layer 120) at cascade block 1141 before being processed by downstream modules (e.g., DCM 124, STDM 122, and SAM 128 in branch 1121).

[0059] In some embodiments, the output of SAM 128 in branch 1121 (e.g., features or feature maps generated by SAM 128) can be processed to generate one or more outputs 104. The output of SAM 128 can be provided to activation block 134. Activation block 134 can receive and / or process the output of SAM 128. The activation block can be used to map the output of SAM 128 to a value between 0 and 1. The activation block may introduce nonlinearity. The activation block can learn complex decision boundaries (e.g., whether a particular feature or pixel is a salient object or not a salient object).

[0060] In some embodiments, the output of activation block 134 in branch 1121 may be upsampled by upsampling block 136. Upsampling block 136 may generate one or more outputs 104.

[0061] In some cases, downsampling layers may be included between stages in backbone 106 (e.g., between stages 1101 and 1102, between stages 1102 and 1103, and / or between stages 1102 and 1103). The downsampling layers may perform downsampling on features or feature maps generated by these stages, which may improve the operating efficiency of backbone 106.

[0062] Figure 2 An exemplary machine learning model 200 for processing images and generating outputs according to some embodiments of the present disclosure is shown. The machine learning model 200 is similar to Figure 1The machine learning model 100 may be encoded and / or stored in one or more storage devices. The machine learning model 200 may include a backbone 106 and a side structure 208. The backbone 106 of the machine learning model 200 may receive and / or process one or more images 202, such as images that are not video or continuous image frames, images that are not temporally related to each other, images that have no temporal relationship to each other, etc. The side structure 208 may generate and / or output one or more outputs 204 that make predictions, inferences, and / or determinations about the one or more images 202. The branches in the side structure 208 may be different from Figure 1 The branches in the side structure 108 of the machine learning model 200 are implemented. The side structure 208 of the machine learning model 200 may include one or more branches, such as branch 2121, branch 2122, branch 2123, and branch 2124. Because one or more images 202 do not have time information, modules related to spatiotemporal processing, such as a spatiotemporal convolution module, a spatiotemporal difference module, or a spatiotemporal difference convolution module, and any related cascade blocks may be omitted. For example, branch 2121 may include DCM 124 and SAM 128.

[0063] In some embodiments, Figure 1 The machine learning model 100 may include a toggle switch that can turn off or omit spatiotemporal processing in a branch of the side structure 208 (e.g., a spatiotemporal convolution module, a spatiotemporal difference module, or a spatiotemporal difference convolution module). Visual transformer in the backbone

[0064] In some embodiments, one or more backbone levels in the machine learning model 100 and / or the machine learning model 200 may include one or more visual transformer blocks. Including visual transformer blocks in the backbone level may enhance spatial information and / or extract spatial attention features. Level 1101 may include one or more visual transformer blocks. Level 1102 may include one or more visual transformer blocks. Level 1103 may include one or more visual transformer blocks. Level 1103 may include one or more visual transformer blocks. The visual transformer block may extract distance relationships between image regions or regions in feature maps. The visual transformer block may include a global attention mechanism. The global attention mechanism may be trained on large-scale data. The visual transformer block may measure the relationship between different image regions or regions in feature maps. Image regions or regions in feature maps may be placed in sequence or linearly embedded. Position embedding may be added. The resulting vector sequence may be fed to a transformer encoder. The transformer encoder may implement a global attention mechanism to draw global dependencies between inputs and outputs. Different example convolution types

[0065] In computer vision, convolutions can be performed according to different convolution types. Different convolution types can extract different kinds of spatial information from local regions of the input feature map.

[0066] In a convolution module that implements and / or uses a standard convolution type, an output value may be calculated from an inner product between kernel weights (eg, parameters) and pixel intensities in a local region of an input feature map.

[0067] In some convolution modules that implement and / or use other different convolution types (e.g., different pixel difference convolution types), the output value can be calculated from the inner product between the pixel intensity difference and the kernel weights between selected pixel pairs of the local region of the input feature map. According to the selection strategy, the pixel pairs can be selected within the local region, and the pixel difference convolution can be performed based on the inner product between the pixel difference between the pair and the kernel weights.

[0068] Different pixel difference convolution types may use different selection strategies to select pixel pairs, which selection strategies may emphasize certain types or kinds of information. A first pixel difference convolution type may include computing a first inner product between a first pixel intensity difference between a first pixel pair selected using a first selection strategy and a first kernel weight. A second pixel difference convolution type may include computing a second inner product between a second pixel intensity difference between a second pixel pair selected using a second selection strategy different from the first selection strategy and a second kernel weight.

[0069] Assume that the current local region R contains pixels: A convolution module that implements and / or uses the standard convolution type may perform the following equation:

[0070] A convolution module implementing and / or using a pixel difference convolution type may perform the following equation:

[0071] For Equation 1 and Equation 2, y R can be the output value at the center of region R, θ = {w1, w2, ..., w i ,…} can contain kernel weights or parameters, Can contain from x R (For example ), and m can be the number of pixel pairs selected in the local area.

[0072] Figure 3A-3D Different pixel pair selection strategies according to some embodiments of the present disclosure are shown. Figure 3AAn exemplary selection of pixel pairs in a local region is depicted. As depicted, a first pixel pair {x1, x′1} in the local region, a second pixel pair {x2, x′2} in the local region, ... an i-th pixel pair {x1, x′1} in the local region, can be selected according to a particular selection strategy. i , x′ i}.

[0073] Examining pixel differences can extract image gradient information and / or pixel relationships, which can be helpful in capturing edge cues. Different pixel difference convolution types can use different selection strategies to select pixel pairs, which can emphasize different types of rich gradient information (e.g., gradient information in various directions).

[0074] Figure 3B A center pixel pair selection strategy is shown. A pixel pair can be selected in such a way that the pixel intensity difference of the pixel relative to the center pixel can be convolved with the kernel for the pixel difference convolution operation.

[0075] Figure 3C The corner pixel pair selection strategy is shown. The pixel pairs can be selected in the corner direction so that the corner pixel intensity differences can be convolved with the kernel for the pixel difference convolution operation.

[0076] Figure 3D A radial pixel pair selection strategy is shown. Pixel pairs can be selected radially so that radial pixel intensity differences can be convolved with the kernel for the pixel difference convolution operation. Different pixel pair selection strategies can allow the detection of microstructure patterns in a complementary manner.

[0077] However, using a variety of different convolution types at inference time or at deployment time may increase the number of parameters in the machine learning model. Differential convolution reparameterization can alleviate this problem by combining different convolution types (trained at training time) (e.g., including standard convolution and other pixel differential convolution types) into a single convolution operator (for use at inference time or at deployment time) without the need for additional parameters and computational overhead at inference time or at deployment time.

[0078] As discussed below, the differential convolution reparameterization can be applied to Figure 1-Figure 2 The convolution operation occurs in the backbone 106 to extract rich spatial information. Fig.13 An exemplary method for differential convolution reparameterization is shown in .

[0079] In some embodiments, the differential convolution reparameterization can be applied to a spatiotemporal convolution module, a spatiotemporal difference module, or a spatiotemporal differential convolution module (e.g., Figure 1 The different convolution operations that occur in STDM 122) are used to extract rich spatiotemporal information. Fig.15 Another exemplary method for differential convolution reparameterization is shown in .

[0080] Figure 4 FIG. 4 shows a differential convolution reparameterization according to some embodiments of the present disclosure. FIG. Figure 1-Figure 2 The backbone level of one of the backbone levels 106 is Figure 4 404, which is depicted as level 402 at training time, and level 404 at inference time or deployment time. At training time, level 402 may include one or more layers, such as layer 4041, layer 4042, layer 4043, and layer 4044. In the layer, one or more (parallel) operators implementing different convolution types (e.g., standard convolution type, pixel difference convolution type) may be included in the layer. As shown, layer 4041 may include a standard convolution operator, a first pixel difference convolution operator, a second pixel difference convolution operator, and a third pixel difference convolution operator. Layers (such as layer 4041) may extract pixel intensity information in parallel through standard convolution operators and various gradient statistics using various different pixel difference convolution operators. Other layers may include one or more operators, sets of operators, or appropriate mixtures of operators. As shown, layer 4041, layer 4042, layer 4043, and layer 4044 can each include a standard convolution operator, a first pixel difference convolution operator, a second pixel difference convolution operator, and a third pixel difference convolution operator. Features extracted by the corresponding convolution operators in the layer can be added or summed together and can be processed by the next layer. The convolution operators each include parameters or kernel weights learned or trained from training data during training time.

[0081] In a layer, a convolution operator that implements a certain convolution type may contain a weighting parameter (or weighting coefficient) α i . Weighting parameters may be learned or trained from training data during training time. The weighting parameters may learn which features generated by using which convolution type may be more predictive or more informative. At inference time or at deployment time, stage 404 may include one or more layers, such as layer 4061, layer 4062, layer 4063, and layer 4064. In the layer, a single merged convolution operator that merges different convolution types (e.g., standard convolution type, pixel difference convolution type) may be included in the layer. In some embodiments, a layer may include one or more merged convolution operators. In some embodiments, a layer may include a standard convolution operator and one or more merged convolution operators. A merged convolution operator may be created by fusing parameter sets obtained in various convolution operators corresponding to different convolution types at training time into a parameter set to be used in the merged convolution operator. Fig. 9 An exemplary merged convolution operator in a backbone-level layer is shown in FIG.

[0082] In some embodiments, a level in the backbone (e.g., level 404) may include one or more spatial difference layers (e.g., layer 4061, layer 4062, layer 4063, and layer 4064). The level may include a first spatial difference layer (e.g., layer 4061) having a first parameter having a first parameter based on training of different convolution types. The first parameter may fuse or combine a first parameter set, wherein each first parameter set is obtained by training using a specific convolution type. The level may include a second spatial difference layer (e.g., layer 4062) having a second parameter based on training of different convolution types. The second parameter may fuse or combine a second parameter set, wherein each second parameter set is obtained by training using a specific convolution type. The level may include a third spatial difference layer (e.g., layer 4063) having a third parameter based on training of different convolution types. The third parameter may fuse or combine a third parameter set, wherein each third parameter set is obtained by training using a specific convolution type. The level may include a fourth spatial difference layer (e.g., layer 4064) having a fourth parameter based on training of different convolution types. The fourth parameters may be fused or combined with fourth parameter sets, where each fourth parameter set is obtained by training using a specific convolution type. The spatial difference layers may be arranged sequentially or in a cascaded manner, where the output of one layer is fed as input to the next layer. Fig. 9 An exemplary spatially differential stratification is shown in .

[0083] During training, parallel operators implementing different convolution types are trained to obtain different parameter sets corresponding to the parallel operators. The differential convolution reparameterization technique effectively flattens the parallel operators into a single operator by fusing or combining the different parameter sets into a single parameter set for a single operator to be used at inference time or at deployment.

[0084] Reparameterization (e.g., fusing or combining parameter sets) is not a trivial task. First, the operations performed by a convolution module that implements and / or uses a pixel difference convolution type can be rewritten or reformulated as follows:

[0085] Q j The "-x" in equation 3 can be collected j " term. For simplicity, R has been removed. This reformulation allows the form of the pixel difference convolution operation to be transformed into the form of the standard convolution operation. This reformulation can allow the kernel weights or parameters in the pixel difference convolution operator to be transformed into an equivalent set of kernel weights or parameters in the standard convolution operator. Transforming the kernel weights or parameters into an equivalent set of kernel weights or parameters can enable direct summation of parameters from each convolution type to form a single set of parameters that can be used in the combined convolution operator.

[0086] The parameter sets trained using different convolution operators in the layer (e.g., {α i ,θ′ i}, corresponding to the corresponding convolution operators (e.g., including the parameters of the transformation)) can be fused or combined together to form one or a single parameter set. The fusion or combination can include summing the parameters. The fusion or combination can include summing the parameters using corresponding weighted parameters (e.g., as a weighted sum). For example, the parameters from each convolution type can be summed using the weighted parameters to form the fused parameters. The weighted parameters can be used to perform a parameter-by-parameter summation of parameters from different convolution types. An exemplary formula can be as follows: y=∑ i α i f(Δx,θ i )=∑ i α i f(x,θ′ i )=∑ i α i ·∑ j x j ·w′ i，j =∑ j x j ·∑ i α i ·w′ i，j =f(x,∑ i α i ·θ′ i ) = f(x, θ′) (Equation 4) Exemplary Spatiotemporal Difference Network

[0087] Video can be processed using standard 3D convolutions, which use pixel intensities in local 3D regions (e.g., cubes) to detect spatiotemporal patterns. Pixel intensities by themselves may not necessarily encode local temporal variations in consecutive frames, which can be useful for determining temporal consistency. To address this concern, pixel intensity differences can be used to detect spatiotemporal patterns that may not be captured by standard 3D convolutions. Pixel intensity differences can produce higher-order spatiotemporal variations in feature maps. As discussed herein, two or more exemplary pixel pair selection strategies can be used to implement different spatiotemporal convolution types. The selection strategies are described in

[14] ,

[15] ,

[16] ,

[17] ,

[18] ,

[19] ,

[20] ,

[21] ,

[22] ,

[23] ,

[24] ,

[25] ,

[26] ,

[27] ,

[28] ,

[29] ,

[30] ,

[31] ,

[32] ,

[33] ,

[34] ,

[35] ,

[36] ,

[37] ,

[38] ,

[39] ,

[40] ,

[41] ,

[42] ,

[43] ,

[44] ,

[45] ,

[46] ,

[47] ,

[48] ,

[49] ,

[50] ,

[51] ,

[52] ,

[53] ,

[54] ,

[55] ,

[56] ,

[57] ,

[58] ,

[59] ,

[60] ,

[61] ,

[62] ,

[63] ,

[64] ,

[65] ,

[66] ,

[67] ,

[68] ,

[69] ,

[70] ,

[71] ,

[72] ,

[73] Figure 6-Figure 8 Shown in.

[0088] By slicing the three-dimensional input feature space and operating on one or more two-dimensional space-time planes in the three-dimensional input feature space, the implementation of pixel intensity differential convolution for extracting space-time information can be simplified. In some embodiments, the three-dimensional input feature space may include a height dimension, a width dimension, and a time dimension. An exemplary two-dimensional space-time plane can be sliced ​​in the width dimension, and a height-time plane is obtained. Another exemplary two-dimensional space-time plane can be sliced ​​in the height dimension, and a width-time plane is obtained. The two-dimensional space-time plane can be provided as input and can be processed by one or more (parallel) operators to extract space-time information. The operator can use a two-dimensional local area on a two-dimensional space-time plane (e.g., a height-time plane or a width-time plane) to perform a two-dimensional convolution, such as a standard convolution or a pixel differential convolution. Equation 1 can be used to perform a standard convolution, where a local area in the height-time plane or a local area in the width-time plane is used as the local area R in Equation 1. Equation 2 can be used to perform a pixel differential convolution, where a local area in the height-time plane or a local area in the width-time plane is used as the local area R in Equation 2. If desired, Figure 3A-3D The pixel selection strategy shown in can be applied to local regions in the height-time plane or to local regions in the width-time plane.

[0089] like Figure 1 As shown in , a spatiotemporal convolution module, a spatiotemporal difference module, or a spatiotemporal difference convolution module can be implemented to extract spatiotemporal information. This module can be called a spatiotemporal difference network. Figure 5A-Figure 5BAn exemplary implementation of a spatiotemporal difference module 502 according to some embodiments of the present disclosure is shown. The spatiotemporal difference module 502 may include one or more spatiotemporal difference convolution blocks, for example, a first spatiotemporal difference convolution block 512 for processing a local area in a height-time plane, and a second spatiotemporal difference convolution block 522 for processing a local area in a width-time plane. The input feature space may be three-dimensional. The input feature space may include a height dimension, a width dimension, and a time dimension. The input feature space may include a time series or sequence of two-dimensional feature maps having a height dimension and a width dimension. When together and / or stacked, the time series or sequence of two-dimensional feature maps may form a three-dimensional input feature space having a height dimension, a width dimension, and a time dimension. The first spatiotemporal difference convolution block 512 may convolve a height-time slice of the input feature space using a first kernel (e.g., a matrix with parameters). Specifically, the first spatiotemporal difference convolution block 512 may calculate the inner product between the pixel intensity in the first local area of ​​the height-time plane and the first kernel. The second spatiotemporal differential convolution block 522 can use a second kernel (e.g., a matrix with parameters) to convolve the width-time slice of the input feature space. Specifically, the second spatiotemporal differential convolution block 522 can calculate the inner product between the pixel intensity in the second local area of ​​the width-time plane and the second kernel.

[0090] The first spatiotemporal differential convolution block 512 can extract a spatiotemporal pattern in a first two-dimensional spatiotemporal plane (e.g., a height-time plane). The second spatiotemporal differential convolution block 522 can extract a spatiotemporal pattern in a second two-dimensional spatiotemporal plane (e.g., a height-time plane). The features extracted by the first spatiotemporal differential convolution block 512 and the features extracted by the second spatiotemporal differential convolution block 522 can be added element-by-element to the original input features provided to the spatiotemporal differential module 502.

[0091] exist Figure 5A, the first spatiotemporal differential convolution block 512 is arranged before the second spatiotemporal differential convolution block 522. The first spatiotemporal differential convolution block 512 can receive and / or process a first input (e.g., an input feature space having a height dimension, a width dimension, and a time dimension) and generate a first output. The first input can be added element by element by element adder block 520 to the first output. The result of the addition (e.g., the output of the element by element adder block 520) can be provided as a second input to the second spatiotemporal differential convolution block 522. The second spatiotemporal differential convolution block 522 can generate a second output. The second input to the second spatiotemporal differential convolution block 522 can be added element by element by element adder block 530 to the second output. Optionally, a projection layer 540 can be included in the spatiotemporal differential module 502 to process the output of the element by element adder block 520, thereby transforming the output of the element by element adder block 520 to a different feature space for additional processing. The projection layer 540 may adjust and / or reduce the number of channels in the result of the adder block 530. The projection layer 540 may transform the result of the adder block 530 to make the result more suitable for further processing.

[0092] exist Figure 5B , the second spatiotemporal differential convolution block 522 is arranged before the first spatiotemporal differential convolution block 512. The second spatiotemporal differential convolution block 522 can receive and / or process a first input (e.g., an input feature space having a height dimension, a width dimension, and a time dimension) and generate a first output. The first input can be added element by element by element adder block 520 to the first output. The result of the addition (e.g., the output of the element by element adder block 520) can be provided as a second input to the first spatiotemporal differential convolution block 512. The first spatiotemporal differential convolution block 512 can generate a second output. The second input to the first spatiotemporal differential convolution block 512 can be added element by element by element adder block 530 to the second output. Optionally, a projection layer 540 can be included in the spatiotemporal differential module 502 to process the output of the element by element adder block 520, thereby transforming the output of the element by element adder block 520 to a different feature space for additional processing.

[0093] The spatiotemporal differential convolution blocks (e.g., the first spatiotemporal differential convolution block 512 and the second spatiotemporal differential convolution block 522) can be implemented by using one or more (parallel) operators of one or more specific spatiotemporal convolution types. The operator can perform a standard convolution operation, as shown in Equation 1. The operator can perform a pixel differential convolution operation, as shown in Equation 2. The pixel differential convolution operation can implement a certain pixel selection strategy. The operator can implement a pixel differential convolution operation that can select pixel pairs according to the selection strategy, determine pixel intensity differences, and extract micro-patterns in different directions from a first local area on the height-time plane or a second local area on the width-time plane. As shown above, for example, in Figure 3A-3DIn the diagram depicted in , exemplary selection strategies may include a center pixel pair selection strategy, a corner pixel pair selection strategy, and a radial pixel pair selection strategy.

[0094] Although some examples describe the first spatiotemporal differential convolution block 512 and / or the second spatiotemporal differential convolution block 522 as implementing pixel differential convolution (e.g., implementing a pixel selection strategy to select pixel pairs to determine pixel differences in a local patch), the present disclosure contemplates that the first spatiotemporal differential convolution block 512 and / or the second spatiotemporal differential convolution block 522 can implement standard convolution.

[0095] To illustrate, Figure 6-Figure 8 Depicted and / or illustrated are inner product operations performed on pixel differences of pixel pairs selected using a center pixel pair selection strategy and a corner pixel pair selection strategy to extract spatiotemporal information from a local region in a height-time plane or a local region in a width-time plane. Although a radial pixel pair selection strategy can be used, some implementations may not utilize the radial pixel pair selection strategy due to the size and complexity of the kernel. Standard convolution does not have Figure 6-Figure 7 , but can be used as a convolution type when processing local areas in the height-time plane and local areas in the width-time plane.

[0096] Figure 6 The operation of a height-time spatiotemporal convolution block, for example during training time, according to some embodiments of the present disclosure is shown. An input 602 may have a height dimension (shown as H), a width dimension (shown as W), and a time dimension (shown as T). A first local region 610 may be obtained in or within a height-time plane of the input 602. The first local region 610 may be processed by parallel operators, such as a center pixel differential spatiotemporal convolution operator 650 and a corner pixel differential spatiotemporal convolution operator 660. The center pixel differential spatiotemporal convolution operator 650 and the corner pixel differential spatiotemporal convolution operator 660 may generate outputs 604 and 606, respectively.

[0097] It can be imagined that the height-time spatiotemporal convolution block can contain one or more (parallel) operators that implement one or more different (spatiotemporal) convolution types, such as standard convolution, pixel difference convolution type, center pixel difference convolution type, corner pixel difference convolution type, radial pixel difference convolution type, etc.

[0098] Figure 6The upper half of 600 shows a center pixel difference spatiotemporal convolution operator 650 that calculates the inner product between the pixel difference of the first local area 610 in the height-time plane and the kernel 620 (or kernel weights or parameters), where the pixel pair is selected based on the center pixel pair selection strategy. The inner products calculated in this way for the sliding / moving local area in the height-time plane can be summed and / or accumulated to form an output 604. The output 604 can represent the result of applying the pixel difference convolution operation to the features in the height-time plane according to the center pixel pair selection strategy.

[0099] Figure 6 The lower half of 600 shows an angular pixel difference spatiotemporal convolution operator 660 that computes inner products between pixel differences and kernel 630 (or kernel weights or parameters) for pixel pairs in a first local region 610 in the height-time plane, where the pixel pairs are selected based on an angular pixel pair selection strategy. The inner products computed in this manner for sliding / moving local regions in the height-time plane may be summed and / or accumulated to form output 606. Output 606 may represent the result of applying a pixel difference convolution operation to features in the height-time plane according to the angular pixel pair selection strategy.

[0100] Both the center pixel differential spatiotemporal convolution operator 650 and the corner pixel differential spatiotemporal convolution operator 660 can implement spatiotemporal differentiation to extract higher order spatiotemporal features than zero-order pixel intensity information. Output 604 and output 606 can be added together and provided to one or more additional layers / modules / blocks / operators for further processing. At training time, the parameters in kernel 620 and kernel 630 can be learned or trained using training data. Weighting parameter (or weighting coefficient) α i Each parallel operator (eg, center pixel difference convolution operator 650 and corner pixel difference spatiotemporal convolution operator 660) may be included in the parallel operators to learn or train which type of information may carry more weight.

[0101] Figure 7 The operation of the width-time spatiotemporal convolution block, for example during training time, according to some embodiments of the present disclosure is shown. The input 702 may have a height dimension (shown as H), a width dimension (shown as W), and a time dimension (shown as T). A second local region 710 may be obtained in or within the width-time plane of the input 702. The second local region 710 may be processed by parallel operators (e.g., a center pixel differential spatiotemporal convolution operator 750 and an angle pixel differential spatiotemporal convolution operator 760). The center pixel differential spatiotemporal convolution operator 750 and the angle pixel differential spatiotemporal convolution operator 760 may generate outputs 704 and 706, respectively.

[0102] It can be imagined that the width-time spatiotemporal convolution block can contain one or more (parallel) operators that implement one or more different (spatiotemporal) convolution types, such as standard convolution, pixel difference convolution type, center pixel difference convolution type, corner pixel difference convolution type, radial pixel difference convolution type, etc.

[0103] Figure 7 The upper half of the diagram shows a center pixel difference spatiotemporal convolution operator 750 that calculates the inner product between the pixel difference of the pixel pair in the second local area 710 in the width-time plane and the kernel 720 (or kernel weights or parameters), where the pixel pair is selected based on the center pixel pair selection strategy. The inner products calculated in this way for the sliding / moving local area in the width-time plane can be summed and / or accumulated to form an output 704. The output 704 can represent the result of applying the pixel difference convolution operation to the features in the width-time plane according to the center pixel pair selection strategy.

[0104] Figure 7 The lower half of 706 shows an angular pixel difference spatiotemporal convolution operator 760 that computes inner products between pixel differences and kernel 730 (or kernel weights or parameters) for pixel pairs in a second local region 710 in the width-time plane, where the pixel pairs are selected based on an angular pixel pair selection strategy. Inner products computed in this manner for sliding / moving local regions in the width-time plane may be summed and / or accumulated to form output 706. Output 706 may represent the result of applying a pixel difference convolution operation to features in the width-time plane according to the angular pixel pair selection strategy.

[0105] Both the center pixel differential spatiotemporal convolution operator 750 and the corner pixel differential spatiotemporal convolution operator 760 can implement spatiotemporal differentiation to extract higher order spatiotemporal features than zero-order pixel intensity information. Output 704 and output 706 can be added together and provided to one or more additional layers / modules / blocks / operators for further processing. At training time, the parameters in kernel 720 and kernel 730 can be learned or trained using training data. Weighting parameter (or weighting coefficient) α i Each parallel operator (eg, center pixel difference convolution operator 750 and corner pixel difference spatiotemporal convolution operator 760) may be included in the parallel operators to learn or train which type of information may carry more weight.

[0106] Figure 8Depicted are spatiotemporal feature maps and processing by operators using different spatiotemporal convolution types according to some embodiments of the present disclosure. It can be intuitively appreciated that spatiotemporal convolution operators using different convolution types (e.g., standard convolution, pixel difference convolution type, center pixel difference convolution, corner pixel difference convolution) can capture spatiotemporal features or information in different and / or complementary ways. With standard convolution, feature maps may contain primarily zero-order intensities. In contrast, feature maps generated using pixel difference convolution types may capture higher-order spatiotemporal variations and / or higher-order spatiotemporal gradients. In summary, different convolution types may enrich feature representations.

[0107] Although Figure 6-Figure 7 The example in shows a height-time spatiotemporal convolution block and a width-time spatiotemporal convolution block containing two convolution operators, but it is conceivable that the height-time spatiotemporal convolution block and / or the width-time spatiotemporal convolution block can contain one or more (parallel) convolution operators. The parallel convolution operators can implement different convolution types, such as standard convolution, pixel difference convolution type, center pixel difference convolution type, corner pixel difference convolution type, radial pixel difference convolution type, etc.

[0108] In some embodiments, reparameterization can be used to fuse or combine different spatiotemporal convolution types (including standard convolution types and pixel difference convolution types (e.g., Figure 6-Figure 8 The first spatiotemporal differential convolution block 512 may include a first kernel having, for example, first parameters trained based on different spatiotemporal convolution types. The first parameters may fuse different parameter sets obtained by training parallel operators implementing different spatiotemporal convolution types. The second spatiotemporal differential convolution block 522 may include a second kernel having second parameters trained based on different spatiotemporal convolution types. The second parameters may fuse different parameter sets obtained by training parallel operators implementing different spatiotemporal convolution types.

[0109] Different spatiotemporal convolution types may include standard convolution, pixel difference convolution types, center pixel difference convolution, corner pixel difference convolution, and radial pixel difference convolution. During training time, parallel operators using different convolution types may be implemented in a spatiotemporal convolution block (e.g., a height-time spatiotemporal convolution block or a width-time spatiotemporal convolution block). Parameters such as kernel weights and weighting parameters of the parallel operators may be trained and / or learned. Figure 4In a manner similar to the reparameterization technique shown in , different parameter sets corresponding to different parallel operators can be fused or combined together to form one parameter set that can be used in a merged height-time spatiotemporal convolution block or a merged width-time spatiotemporal convolution block during inference time or at deployment. Employing the reparameterization technique can mean that different convolution types can be used or utilized at the actual cost of only one convolution type. The additional convolution types can be considered to be computationally free and / or parameter-free. Example implementation of spatially differentiated slicing in the backbone level

[0110] Fig. 9 An exemplary implementation of a layer 900 in a backbone level according to some embodiments of the present disclosure is depicted. Figure 1-Figure 2 The backbone level may include one or more layers. In some cases, the backbone level may include four layers. Preferably, a reparameterization technique (such as Figure 4 ) to fuse different sets of parameters learned during training time into one set of parameters that can be used by the merged convolution operator at inference time (or at deployment time). Layer 900 may include a (depthwise) merged convolution operator 902 that uses the fused set of parameters. Layer 900 may include an activation block 904. Activation block 904 may implement a suitable activation function, such as a rectified linear function. Layer 900 may include a 1×1 convolution operator 906. Layer 900 may include a residual path and a forward path, the residual path may include the merged convolution operator 902, the activation block 904, and the 1×1 convolution operator 906, and the forward path passes the input of layer 900 through an adder 910 to form a residual output. Example implementation of a dilated convolution module

[0111] Fig.10 An exemplary implementation of the dilated convolution module 1000 according to some embodiments of the present disclosure is depicted. The dilated convolution module 1000 (e.g., Figure 1The DCM 124 of the embodiment of the present invention may perform dilation and extract features at different scales. The dilated convolution module 1000 may perform one or more dilated convolutions. The dilated convolution module 1000 may generate features or one or more feature maps. The dilated convolution module 1000 may include an activation block 1002. The activation block 1002 may implement a suitable activation function, for example, a rectified linear function. The dilated convolution module 1000 may include a 1×1 convolution operator 1004. The dilated convolution module 1000 may include parallel dilated convolution blocks with different dilation rates. The dilated convolution block 1006 may have a dilation rate of 5. The dilated convolution block 1008 may have a dilation rate of 7. The dilated convolution block 1010 may have a dilation rate of 9. The dilated convolution block 1012 may have a dilation rate of 11. The present disclosure contemplates other dilation rate sets. The outputs of the parallel dilated convolution blocks may be combined by an adder 1014. The adder 1014 may adjust and / or reduce the number of channels. In some embodiments, one or more projection layers may be included to process one or more outputs of the parallel dilated convolutional blocks. In some embodiments, a projection layer may be included to process the output of adder 1014. In some embodiments, adder 1014 may be combined by cascade blocks. Example Implementation of Spatial Attention Module

[0112] Fig.11 An exemplary implementation of the spatial attention module 1100 according to some embodiments of the present disclosure is depicted. The spatial attention module 1100 (e.g., Figure 1 The SAM 128 of the spatial attention module 1100 may perform background suppression. The spatial attention module 1100 may include an activation block 1102. The activation block 1102 may implement a suitable activation function, for example, a rectified linear function. The spatial attention module 1100 may include a 1×1 convolution block 1104. The 1×1 convolution block 1104 may adjust and / or reduce the number of channels. The spatial attention module 1100 may include a 3×3 convolution block 1106. The 3×3 convolution block 1106 may determine the correlation between pixels (for example, find spatial correlation information). The spatial attention module 1100 may include an activation block 1108. The activation block 1108 may implement a suitable activation function, for example, a sigmoid function. The output of the activation block 1108 may include a single channel feature map. The input of the spatial attention module 1100 may be fed forward and multiplied with the output of the activation block 1108 at the multiplier block 1114. The multiplier block 1114 may perform an element-wise multiplication of the input to the spatial attention module 1100 (e.g., having a single channel feature map) and the output of the activation block 1108 (e.g., having another single channel feature map). The element-wise multiplication performed by the multiplier block 1114 may generate an attention feature map. Exemplary methods for generating one or more outputs

[0113] Fig.121 is a flow chart illustrating a method 1200 for generating an output based on one or more images according to some embodiments of the present disclosure. The one or more images may be individual images that have no temporal relationship to each other. The one or more images may be a sequence of images of a video, such as a time series of images. The output may include inferences about the one or more images. The output may include predictions about the one or more images. Examples of the output may include edge detection, image segmentation, foreground detection, background detection, salient object detection, object classification, etc. The method 1200 may use a computing device (such as Fig.16 The computing device 1600 in the embodiment is used for execution.

[0114] At 1202, one or more images (e.g., image 102) may be input into a first backbone stage of a machine learning model (e.g., stage 1101 of backbone 106 of machine learning model 100, stage 1101 of backbone 106 of machine learning model 200, etc.). The first backbone stage may include a spatial difference layer for performing convolution using one or more first parameters trained based on different spatial convolution types. Figure 4 An exemplary spatially differential stratification is shown in .

[0115] At 1204 , the first backbone stage may generate a first feature map based on the one or more images.

[0116] In 1206, the first feature map can be input into a second backbone level of the machine learning model (e.g., level 1102 of backbone 106 of machine learning model 100, level 1102 of backbone 106 of machine learning model 200, etc.).

[0117] In 1208 , the second backbone stage may generate a second feature map based on the first feature map.

[0118] In 1210, the second feature map can be input into the first branch of the machine learning model (e.g., branch 1122 of machine learning model 100, branch 2122 of machine learning model 200, etc.).

[0119] In 1212, the first branch may generate a third feature map based on the second feature map.

[0120] In 1214, the first feature map and the third feature map can be input into the second branch of the machine learning model (e.g., branch 1121 of the machine learning model 100, branch 2121 of the machine learning model 200, etc.).

[0121] In 1216, the second branch may combine the first feature map with the third feature map to generate a first combined feature map (e.g., using the cascade block 1141 in branch 1121 of the machine learning model 100, using the cascade block 1141 in branch 2121 of the machine learning model 200, etc.).

[0122] In 1218, the second branch may generate one or more predictions regarding the one or more images based on the first combined feature map.

[0123] In some embodiments, generating the one or more predictions in 1218 may include processing the first combined feature map by a spatiotemporal differential convolution module in the second branch. The spatiotemporal differential convolution module may perform convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map. Figure 5A-Figure 5B An exemplary spatiotemporal differential convolution module is shown in . In some embodiments, the spatiotemporal differential convolution module performs convolution using one or more second parameters trained based on different spatiotemporal convolution types.

[0124] In some embodiments, the first backbone level may further include a second spatial difference layer for performing convolution on the output of the first spatial difference layer using one or more third parameters trained based on different spatial convolution types. The first backbone level may further include additional spatial difference layers.

[0125] In some embodiments, method 1200 may further include inputting the second feature map into a third backbone level of the machine learning model. Method 1200 may further include generating a fourth feature map based on the second feature map by the third backbone level. Method 1200 may further include inputting the fourth feature map into a third branch of the machine learning model. Method 1200 may further include generating a fifth feature map based on the fourth feature map by the third branch. Method 1200 may further include inputting the fifth feature map into the first branch. Method 1200 may further include combining the second feature map with the fifth feature map by the first branch to generate a second combined feature map. The first branch may generate a third feature map based on the second combined feature map.

[0126] In some embodiments, method 1200 may further include inputting the fourth feature map into a fourth backbone level of the machine learning model. Method 1200 may further include generating a sixth feature map based on the fourth feature map by the fourth backbone level. Method 1200 may further include inputting the sixth feature map into a fourth branch of the machine learning model. Method 1200 may further include generating a seventh feature map based on the sixth feature map by the fourth branch. Method 1200 may further include inputting the seventh feature map into a third branch. Method 1200 may further include combining the fourth feature map with the seventh feature map by the third branch to generate a third combined feature map. The third branch may generate a fifth feature map based on the third combined feature map.

[0127] In some embodiments, generating the one or more predictions in 1218 may include processing the first combined feature map by a dilated convolution module in a second branch (e.g., DCM 124 in branch 1121 of machine learning model 100, DCM 124 in branch 2121 of machine learning model 200, etc.). The dilated convolution module may perform one or more dilated convolutions. Fig.10 An exemplary implementation of the dilated convolution module is shown in .

[0128] In some embodiments, generating the one or more predictions in 1218 may include combining the output of the spatiotemporal difference convolution module and the output of the dilated convolution module in the second branch to generate a fourth combined feature map (e.g., cascade block 156 in branch 1121 of the machine learning model 100). The spatiotemporal difference convolution module may perform convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map. The dilated convolution module may perform one or more dilated convolutions on the first combined feature map.

[0129] In some embodiments, generating the one or more predictions in 1218 may include processing the fourth combined feature map (or if the spatiotemporal difference convolution module is omitted, the output of the dilated convolution module, such as in Figure 2 ). The spatial attention module can perform convolution to extract spatial correlation information. Fig.11 An exemplary implementation of the spatial attention module is shown in .

[0130] In some embodiments, generating the second feature map in 1218 may include extracting distance relationships between image regions by a visual transformer block in the second backbone stage. Exemplary method for determining parameters of a machine learning model

[0131] Fig.13is a flowchart illustrating a method 1300 for determining parameters of a machine learning model according to some embodiments of the present disclosure. The method 1300 may use a computing device (such as Fig.16 The method 1300 involves applying a reparameterization technique to reduce parameter and computational complexity while taking advantage of the advantages from multiple convolution types.

[0132] In 1302, parameters of a machine learning model may be updated using training data. The machine learning model may include a first layer having a first parallel operator implementing different convolution types. The different convolution types may include a pixel intensity difference convolution type that implements an inner product between a pixel intensity difference between a selected pair of pixels in a local area and a kernel weight. The different convolution types may include other convolution types, such as a standard convolution type. The different convolution types may include other pixel difference convolution types that use different pixel selection strategies. In 1302, the first parallel operator may have a corresponding first parameter set that is updated. In 1304, the first parameter sets (of the parallel operators) may be fused into a first fused parameter set. In 1306, the first fused parameter set in the first convolutional neural network layer is used to process one or more images. The first convolutional neural network layer may be Figure 1 Machine learning models 100 or Figure 2 The machine learning model has 200 layers in the backbone level. Fig. 9 An exemplary implementation of the first convolutional neural network layer is shown in FIG.

[0133] In some embodiments, each of the respective first parameter sets may have a corresponding weighting parameter.Method 1300 may include fusing the first parameter sets, which includes summing the parameters in the first parameter sets using the respective weighting parameters.

[0134] In some embodiments, the machine learning model may include a second layer that processes the output of the first layer. The second layer may have a second parallel operator that implements a different convolution type (e.g., a standard convolution type, a pixel difference convolution type, etc.). The second parallel operator may have a corresponding second parameter set that is updated. Method 1300 may further include fusing the second parameter set into a second fused parameter set, and using the second fused parameter set in a second convolutional neural network layer to process the output of the first convolutional neural network layer. The second convolutional neural network layer may be a layer in the backbone level of the machine learning model 100 or the machine learning model 200. Fig. 9 An exemplary implementation of the second convolutional neural network layer is shown in FIG.

[0135] In some embodiments, method 1300 may further include transforming one of the first parameter sets corresponding to one of the first parallel operators implementing a pixel intensity differential convolution type into an equivalent parameter set of an operator implementing an inner product between pixel intensities in the local region and kernel weights. Exemplary method for extracting spatiotemporal information

[0136] Fig.14 is a flowchart illustrating a method 1400 for extracting spatiotemporal information according to some embodiments of the present disclosure. The method 1400 may use a computing device (such as Fig.16 The computing device 1600 in the embodiment is used for execution.

[0137] At 1402, a first local region in a height-time plane of a first input feature having a height dimension, a width dimension, and a time dimension may be determined. At 1404, a height-time spatiotemporal convolution block (e.g., Figure 5A-Figure 5B The first spatiotemporal differential convolution block 512 is as follows: Figure 6 ) can process the first local area using first parameters trained based on different spatiotemporal convolution types (e.g., standard convolution, pixel difference convolution type, center pixel difference convolution type, corner pixel difference convolution type, radial pixel difference convolution type, etc.). In 1406, a second local area in a width-time plane of a second input feature having a height dimension, a width dimension, and a time dimension can be determined. In 1408, a width-time spatiotemporal convolution block (e.g., Figure 5A-Figure 5B The second spatiotemporal differential convolution block 522 is as follows: Figure 7 ) can use second parameters trained based on different spatiotemporal convolution types (e.g., standard convolution, pixel difference convolution type, center pixel difference convolution type, corner pixel difference convolution type, radial pixel difference convolution type, etc.) to process the second local area.

[0138] In some embodiments, the different spatiotemporal convolution types may include a first spatiotemporal convolution type that calculates a first inner product between a first pixel intensity difference and a first kernel weight between a first pair of pixels of an input local area (e.g., a first local area or a second local area) selected using a first selection strategy. The different spatiotemporal convolution types may include a second spatiotemporal convolution type that calculates a second inner product between a second pixel intensity difference and a second kernel weight between a second pair of pixels of an input local area (e.g., a first local area or a second local area) selected using a second selection strategy.

[0139] In some embodiments, the different spatiotemporal convolution types may include a first spatiotemporal convolution type that calculates a first inner product between a first pixel intensity of an input local area (e.g., a first local area or a second local area) and a first kernel weight. The different spatiotemporal convolution types may include a second spatiotemporal convolution type that calculates a second inner product between a second pixel intensity difference between a pair of pixels of an input local area (e.g., a first local area or a second local area) selected using a selection strategy and a second kernel weight.

[0140] In some embodiments, processing the first local region using the first parameter in 1404 may include determining a first inner product between the first local region and the first parameter.

[0141] In some embodiments, processing the second local region using the second parameter in 1408 may include determining a second inner product between the second local region and the second parameter.

[0142] In some embodiments, the first input features may be element-wise added to the output of the height-time spatiotemporal convolution block, and the second input features include the element-wise added features. This implementation places the height-time spatiotemporal convolution block upstream of the weight-time spatiotemporal convolution block, such as Figure 5A as shown in .

[0143] In some embodiments, the second input features can be added element-by-element to the output of the width-time spatiotemporal convolution block, and the first input features contain the features added element-by-element. This implementation places the height-time spatiotemporal convolution block downstream of the weight-time spatiotemporal convolution block, such as Figure 5B as shown in . Another exemplary method for determining parameters of a machine learning model

[0144] Fig.15 1 is a flow chart illustrating a method 1500 for determining parameters of a machine learning model according to some embodiments of the present disclosure. The method 1500 may use a computing device (e.g. Fig.16 The method 1500 involves applying a reparameterization technique to reduce parameters and computational complexity while taking advantage of the advantages from multiple convolution types.

[0145] At 1502, parameters of the machine learning model, using the training data, may be updated. The machine learning model may include a first spatiotemporal convolution block that processes a first local region in a first two-dimensional plane of a first input feature having at least three dimensions. Examples of the first spatiotemporal convolution block may include a height-time spatiotemporal convolution block and a width-time spatiotemporal convolution block, such as Figure 5A-Figure 5B and Figure 6-Figure 8As shown in . The first spatiotemporal convolution block may include a first parallel operator that implements a first convolution type, and a second parallel operator that implements a second convolution type that is different from the first convolution type. Different convolution types may include (for example, a standard convolution type, a pixel difference convolution type, a center pixel difference convolution type, an angle pixel difference convolution type, a radial pixel difference convolution type, etc.). The first parallel operator may have a first parameter set that is updated in 1502. The second parallel operator may have a second parameter set that is updated in 1502. In 1504, the first and second parameter sets may be fused into a first fused parameter set. In 1506, the first fused parameter set may be used in a first spatiotemporal convolutional neural network layer to process features extracted from one or more images. The first spatiotemporal convolutional neural network layer may be as shown in 1505. Figure 5A-Figure 5B A portion of a height-time spatiotemporal convolutional block and / or a width-time spatiotemporal convolutional block as shown in FIG. The first spatiotemporal convolutional neural network layer may be Figure 1 Part of STDM 122.

[0146] In some embodiments, the first parallel operator may determine a first inner product between a pixel intensity difference between a first pair of pixels in the first local region and a first kernel weight, the first pair being selected using a first selection strategy. The second parallel operator may determine a second inner product between a pixel intensity difference between a second pair of pixels in the first local region and a second kernel weight, the second pair being selected using a second selection strategy.

[0147] In some embodiments, the machine learning model may include a second spatiotemporal convolutional block that processes a second local region in a second two-dimensional plane of a second input feature having at least three dimensions; wherein the second two-dimensional plane is orthogonal to the first two-dimensional plane. The second spatiotemporal convolutional neural network layer may be as follows Figure 5A-Figure 5B A portion of the height-time spatiotemporal convolutional block and / or the width-time spatiotemporal convolutional block shown in . The second spatiotemporal convolutional neural network layer can be Figure 1 Part of STDM 122.

[0148] In some embodiments, the second spatiotemporal convolution block may include a third parallel operator implementing the first convolution type and a fourth parallel operator implementing the second convolution type. The third parallel operator may have a third parameter set updated in 1502. The fourth parallel operator may have a fourth parameter set updated in 1502.

[0149] In some embodiments, method 1500 may further include fusing the third and fourth parameter sets into a second fused parameter set, and using the second fused parameter set in a second spatiotemporal convolutional neural network layer to process features extracted from the one or more images.

[0150] In some embodiments, each of the respective first parameter sets may have a (corresponding) weighting parameter.Method 1500 may include fusing the first parameter sets, which includes summing the parameters in the first parameter sets using the respective weighting parameters.

[0151] In some embodiments, method 1500 may include transforming the first set of parameters into an equivalent set of parameters for an operator that implements an inner product between pixel intensities in the first local region and kernel weights.

[0152] In some embodiments, method 1500 may include transforming the second set of parameters into an equivalent set of parameters for an operator that implements an inner product between pixel intensities in the first local region and kernel weights.

[0153] In some embodiments, the first two-dimensional plane is a height-time plane of the first input feature having a height dimension, a width dimension, and a time dimension.

[0154] In some embodiments, the first two-dimensional plane is a width-time plane of the first input features having a height dimension, a width dimension, and a time dimension. Exemplary Computing Devices

[0155] Fig.16 is a block diagram of an apparatus or system (e.g., an exemplary computing device 1600) according to some embodiments of the present disclosure. One or more computing devices 1600 may be used to implement the functionality described by the figures and herein. The multiple components shown in the figure may be included in the computing device 1600, but any one or more of these components may be omitted or duplicated when appropriate for the application. In some embodiments, some or all of the components included in the computing device 1600 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-chip (SoC) die. Additionally, in various embodiments, the computing device 1600 may not include Fig.16 , and computing device 1600 may include an interface circuit module for coupling to the one or more components. For example, computing device 1600 may not include display device 1606, and may include a display device interface circuit module (e.g., a connector and a driver circuit module) to which display device 1606 may be coupled. In another set of examples, computing device 1600 may not include audio input device 1618 or audio output device 1608, and may include an audio input or output device interface circuit module (e.g., a connector and a supporting circuit module) to which audio input device 1618 or audio output device 1608 may be coupled.

[0156] The computing device 1600 may include a processing device 1602 (e.g., one or more processing devices, one or more of the same type of processing devices, one or more of different types of processing devices). The processing device 1602 may include an electronic circuit module that processes electronic data from a data storage element (e.g., a register, a memory, a resistor, a capacitor, a qubit unit) to transform the electronic data into other electronic data that can be stored in the register and / or the memory. Examples of the processing device 1602 may include a central processing unit (CPU), a graphics processing unit (GPU), a quantum processor, a machine learning processor, an artificial intelligence processor, a neural network processor, an artificial intelligence accelerator, an application specific integrated circuit (ASIC), an analog signal processor, an analog computer, a microprocessor, a digital signal processor, a field programmable gate array (FPGA), a tensor processing unit (TPU), a data processing unit (DPU), etc.

[0157] The computing device 1600 may include a memory 1604, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. The memory 1604 includes one or more non-transitory computer-readable storage media. In some embodiments, the memory 1604 may include a memory that shares a die with the processing device 1602. In some embodiments, the memory 1604 includes one or more non-transitory computer-readable media that stores executable instructions to perform the operations described by the figures and herein, such as Figure 12-Figure 15 . An exemplary portion that may be encoded as instructions and stored in memory 1604 is depicted. Memory 1604 may store instructions for encoding one or more exemplary portions. The instructions stored in the one or more non-transitory computer-readable media may be executed by processing device 1602. In some embodiments, memory 1604 may store data, such as data structures, binary data, bits, metadata, files, binary large objects (blobs), etc., as described by the accompanying drawings and herein. Exemplary data that may be stored in memory 1604 is depicted. As depicted, memory 1604 may store one or more data.

[0158] In some embodiments, memory 1604 may store one or more machine learning models (and / or portions thereof). Memory 1604 may store input data, output data, intermediate outputs, intermediate inputs. Memory 1604 may store instructions for performing one or more operations of a machine learning model. Memory 1604 may store one or more parameters used by the machine learning model. Memory may store information encoding how processing units are connected to each other. Examples of machine learning models or portions of machine learning models may include Figure 1 Machine learning models 100, Figure 2 Machine learning models 200, Figure 4 Level 402, Figure 2 Level 404, Figure 5A-Figure 5B The spatiotemporal difference module 502, Figure 6 The center pixel difference spatiotemporal convolution operator 650, Figure 6 Angular pixel difference spatiotemporal convolution operator 660, Figure 7 The center pixel difference spatiotemporal convolution operator 750, Figure 7 Angular pixel difference spatiotemporal convolution operator 760, Fig. 9 Layer 900, Fig.10 The dilated convolution module 1000, and Fig.11 The spatial attention module 1100.

[0159] In some embodiments, memory 1604 may include training data 1688, and update component 1690 may use training data 1688 to update parameters in machine learning model 100' and / or machine learning model 200'. Training data 1688 may include a labeled data set. Update component 1690 may optimize the parameters in machine learning model 100' and / or machine learning model 200' to minimize the loss function. Machine learning model 100' may represent the loss function achieved during training time. Figure 1 The machine learning model 200' may represent a version of the machine learning model 100 implemented during training time. Figure 2 The reparameterizer 1696 may be used during inference time (or at deployment time) to generate a version of the machine learning model 200. The machine learning model 100 and / or the machine learning model 200 may be used during inference time (or at deployment time). Using the reparameterization techniques described herein, the reparameterizer 1696 may fuse the parameter sets in the machine learning model 100' and / or the machine learning model 200' to form the machine learning model 100 and / or the machine learning model 200.

[0160] In some embodiments, the computing device 1600 may include a communication device 1612 (e.g., one or more communication devices). For example, the communication device 1612 may be configured to manage wired and / or wireless communications for transmitting data to and from the computing device 1600. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can transfer data by using modulated electromagnetic radiation through a non-solid medium. The term does not mean that the associated devices do not contain any wires, although in some embodiments they may not contain any wires. The communication device 1612 may implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects together with any amendments, updates and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as "3GPP2"), etc.). IEEE 802.16 compatible broadband wireless access (BWA) networks are often referred to as WiMAX networks (an acronym for Worldwide Interoperability for Microwave Access), which is a certification mark for products that have passed the conformance and interoperability tests of the IEEE 802.16 standard. The communication device 1612 may operate according to a global system for mobile communications (GSM), a general packet radio service (GPRS), a universal mobile telecommunications system (UMTS), a high speed packet access (HSPA), an evolved HSPA (E-HSPA), or an LTE network. The communication device 1612 may operate according to enhanced data GSM evolution (EDGE), a GSM EDGE radio access network (GERAN), a universal terrestrial radio access network (UTRAN), or an evolved UTRAN (E-UTRAN). The communication device 1612 may operate according to code division multiple access (CDMA), time division multiple access (TDMA), digital enhanced cordless telecommunications (DECT), evolution data optimized (EV-DO), and derivative protocols thereof, and any other wireless protocols designated as 3G, 4G, 5G, and above. In other embodiments, the communication device 1612 may operate according to other wireless protocols. The computing device 1600 may include an antenna 1622 for facilitating wireless communications and / or for receiving other wireless communications (e.g., radio frequency transmissions). The computing device 1600 may include receiver circuitry and / or transmitter circuitry. In some embodiments, the communication device 1612 may manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication device 1612 may include multiple communication chips.For example, the first communication device 1612 may be dedicated to shorter range wireless communications such as Wi-Fi or Bluetooth, and the second communication device 1612 may be dedicated to longer range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO or others. In some embodiments, the first communication device 1612 may be dedicated to wireless communications, and the second communication device 1612 may be dedicated to wired communications.

[0161] Computing device 1600 may include power source / power circuit module 1614. Power source / power circuit module 1614 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuit modules for coupling components of computing device 1600 to an energy source (e.g., DC power, AC power, etc.) separate from computing device 1600.

[0162] Computing device 1600 may include a display device 1606 (or a corresponding interface circuit module, as discussed above). Display device 1606 may include any visual indicator, such as, for example, a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.

[0163] Computing device 1600 may include audio output device 1608 (or corresponding interface circuit modules, as discussed above). Audio output device 1608 may include any device that generates an audible indicator, such as, for example, a speaker, headphones, or earbuds.

[0164] Computing device 1600 may include an audio input device 1618 (or a corresponding interface circuit module, as discussed above). Audio input device 1618 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0165] Computing device 1600 may include a GPS device 1616 (or a corresponding interface circuit module, as discussed above). As is known in the art, GPS device 1616 may communicate with a satellite-based system and may receive the location of computing device 1600.

[0166] Computing device 1600 may include sensor 1630 (or one or more sensors). Computing device 1600 may include corresponding interface circuit modules, as discussed above). Sensor 1630 may sense physical phenomena and translate the physical phenomena into electrical signals that can be processed by, for example, processing device 1602. Examples of sensor 1630 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, stress sensors, moisture sensors, humidity sensors, distance sensors, range sensors, time of flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0167] Computing device 1600 may include another output device 1610 (or a corresponding interface circuit module, as discussed above). Examples of other output devices 1610 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, a tactile output device, a gas output device, a vibration output device, a lighting output device, a home automation controller, or an additional storage device.

[0168] Computing device 1600 may include another input device 1620 (or a corresponding interface circuit module, as discussed above). Examples of another input device 1620 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (such as a mouse), a stylus, a touchpad, a barcode reader, a quick response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0169] The computing device 1600 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a personal digital assistant (PDA), a mobile personal computer, a remote controller, a wearable device, a helmet, glasses, footwear, electronic clothing, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, an Internet of Things device, or a wearable computer system. In some embodiments, the computing device 1600 may be any other electronic device that processes data. Select Example

[0170] Example 1 provides a method, comprising: inputting one or more images into a first backbone level of a machine learning model, the first backbone level comprising a first spatial difference layer, the first spatial difference layer being used to perform convolution using one or more first parameters trained based on different spatial convolution types; generating a first feature map based on the one or more images by the first backbone level; inputting the first feature map into a second backbone level of the machine learning model; generating a second feature map based on the first feature map by the second backbone level; inputting the second feature map into a first branch of the machine learning model; generating a third feature map based on the second feature map by the first branch; inputting the first feature map and the third feature map into a second branch of the machine learning model; combining the first feature map with the third feature map by the second branch to generate a first combined feature map; and generating one or more predictions about the one or more images based on the first combined feature map by the second branch.

[0171] Example 2 provides a method as described in Example 1, wherein: generating the one or more predictions by the second branch includes processing the first combined feature map by a spatiotemporal differential convolution module in the second branch; and the spatiotemporal differential convolution module performs convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map.

[0172] Example 3 provides a method as described in Example 2, wherein the spatiotemporal differential convolution module performs convolution using one or more second parameters trained based on different spatiotemporal convolution types.

[0173] Example 4 provides a method as described in any of Examples 1-3, wherein the first backbone level further includes: a second spatial difference layer, the second spatial difference layer being used to perform convolution on the output of the first spatial difference layer using one or more third parameters trained based on different spatial convolution types.

[0174] Example 5 provides a method as described in any one of Examples 1-4, further comprising: inputting the second feature map into the third backbone level of the machine learning model; generating a fourth feature map by the third backbone level based on the second feature map; inputting the fourth feature map into the third branch of the machine learning model; generating a fifth feature map by the third branch based on the fourth feature map; inputting the fifth feature map into the first branch; and combining the second feature map with the fifth feature map by the first branch to generate a second combined feature map; wherein the first branch generates the third feature map based on the second combined feature map.

[0175] Example 6 provides a method as described in Example 5, further comprising: inputting the fourth feature map into the fourth backbone level of the machine learning model; generating a sixth feature map by the fourth backbone level based on the fourth feature map; inputting the sixth feature map into the fourth branch of the machine learning model; generating a seventh feature map by the fourth branch based on the sixth feature map; inputting the seventh feature map into the third branch; and combining the fourth feature map with the seventh feature map by the third branch to generate a third combined feature map; wherein the third branch generates the fifth feature map based on the third combined feature map.

[0176] Example 7 provides a method as described in any of Examples 1-6, wherein: generating the one or more predictions by the second branch includes processing the first combined feature map by a dilated convolution module in the second branch; and the dilated convolution module performs one or more dilated convolutions.

[0177] Example 8 provides a method as described in any of Examples 1-7, wherein: generating the one or more predictions by the second branch includes: combining the output of a spatiotemporal differential convolution module and the output of a dilated convolution module in the second branch to generate a fourth combined feature map; and processing the fourth combined feature map by a spatial attention module in the second branch; the spatiotemporal differential convolution module performs convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map; the dilated convolution module performs one or more dilated convolutions; and the spatial attention module performs convolution to extract spatial correlation information.

[0178] Example 9 provides a method as described in any of Examples 1-8, wherein generating the second feature map by the second backbone stage includes: extracting distance relationships between image regions by a visual transformer block in the second backbone stage.

[0179] Example 10 provides an apparatus comprising: one or more processors; and one or more storage devices, the one or more storage devices storing a machine learning model having processing operations performed by the one or more processors, the machine learning model comprising: a first backbone stage, the first backbone stage receiving one or more images and generating a first feature map, the first backbone stage comprising a first spatial difference layer, the first spatial difference layer being used to perform convolution using one or more first parameters trained based on different spatial convolution types; a second backbone stage, the second backbone stage receiving the first feature map from the first backbone stage and generating a second feature map based on the first feature map; a first branch, the first branch receiving the second feature map from the second backbone stage and generating a third feature map based on the second feature map; and a second branch, the second branch receiving the first feature map from the first backbone stage and the third feature map from the first branch, combining the first feature map with the third feature map to generate a first combined feature map, and generating one or more predictions about the one or more images based on the first combined feature map.

[0180] Example 11 provides a device as described in Example 10, wherein the second branch includes a spatiotemporal differential convolution module, and the spatiotemporal differential convolution module is used to perform convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map.

[0181] Example 12 provides an apparatus as described in Example 10 or 11, wherein the spatiotemporal differential convolution module comprises: a first spatiotemporal differential convolution block, which performs convolution on a first local area in a height-time plane; and a second spatiotemporal differential convolution block, which processes a second local area in a width-time plane.

[0182] Example 13 provides an apparatus as described in Example 11 or 12, wherein the spatiotemporal differential convolution module performs convolution using one or more second parameters trained based on different spatiotemporal convolution types.

[0183] Example 14 provides a device as described in Example 13, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity difference between a first pixel pair of a local area selected using a first selection strategy and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between a second pixel pair of the local area selected using a second selection strategy and a second kernel weight.

[0184] Example 15 provides an apparatus as described in Example 13 or 14, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity of a local area and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a pixel intensity difference between pixel pairs of the local area selected using a selection strategy and a second kernel weight.

[0185] Example 16 provides a device as described in any of Examples 10-15, wherein the first backbone level further includes one or more additional spatial difference layers, each additional spatial difference layer being used to perform convolution using one or more additional parameters trained based on different spatial convolution types.

[0186] Example 17 provides a device as described in any of Examples 10-16, wherein the different spatial convolution types include: a first convolution type, which calculates a first inner product between a first pixel intensity difference between a first pixel pair selected using a first selection strategy and a first kernel weight; and a second convolution type, which calculates a second inner product between a second pixel intensity difference between a second pixel pair selected using a second selection strategy different from the first selection strategy and a second kernel weight.

[0187] Example 18 provides a device as described in any of Examples 10-17, wherein the different spatial convolution types include: a first convolution type, which calculates a first inner product between a first pixel intensity and a first kernel weight; and a second convolution type, which calculates a second inner product between a pixel intensity difference between a pair of pixels selected using a selection strategy and a kernel weight.

[0188] Example 19 provides a device as described in any of Examples 10-18, wherein the machine learning model further includes: a third backbone stage, the third backbone stage receiving the second feature map from the second backbone stage and generating a fourth feature map based on the second feature map; and a third branch, the third branch receiving the fourth feature map from the third backbone stage and generating a fifth feature map based on the fourth feature map; wherein the first branch further receives the fifth feature map, combines the second feature map with the fifth feature map to generate a second combined feature map, and generates the third feature map based on the second combined feature map.

[0189] Example 20 provides a device as described in Example 19, wherein the machine learning model further includes: a fourth backbone stage, which receives the fourth feature map from the third backbone stage and generates a sixth feature map based on the fourth feature map; and a fourth branch, which receives the sixth feature map from the fourth backbone level and generates a seventh feature map based on the sixth feature map; wherein the third branch further receives the seventh feature map, combines the fourth feature map with the seventh feature map to generate a third combined feature map, and generates the fifth feature map based on the third combined feature map.

[0190] Example 21 provides a device as described in any of Examples 10-20, wherein the second branch further includes a dilated convolution module, which is used to receive the first combined feature map and perform one or more dilated convolutions.

[0191] Example 22 provides a device as described in any of Examples 10-21, wherein the second branch further includes a spatial attention module, which is used to perform convolution and extract spatial correlation information.

[0192] Example 23 provides an apparatus as described in any of Examples 10-22, wherein the second backbone stage includes a visual transformer block for distance relations between image regions.

[0193] Example 24 provides a method comprising updating parameters of a machine learning model using training data, wherein: the machine learning model comprises a first layer, the first layer having a first parallel operator implementing different convolution types, the different convolution types comprising a pixel intensity difference convolution type that implements an inner product between pixel intensity differences and kernel weights between selected pixel pairs in a local area, and the first parallel operator having a corresponding first parameter set that is updated; fusing the first parameter set into a first fused parameter set; and using the first fused parameter set in a first convolutional neural network layer to process one or more images.

[0194] Embodiment 25 provides a method as described in Embodiment 24, wherein: each of the corresponding first parameter sets has a weighting parameter; and fusing the first parameter sets includes summing the parameters in the first parameter sets using the corresponding weighting parameters.

[0195] Example 26 provides a method as described in Example 24 or 25, wherein: the machine learning model includes a second layer that processes the output of the first layer; the second layer has a second parallel operator that implements a different convolution type; the second parallel operator has a corresponding second parameter set that is updated; the method also includes fusing the second parameter set into a second fused parameter set; and using the second fused parameter set in a second convolutional neural network layer to process the output of the first convolutional neural network layer.

[0196] Example 27 provides a method as described in any of Examples 24-26, further comprising transforming one of the first parameter sets corresponding to one of the first parallel operators that implement the pixel intensity differential convolution type into an equivalent parameter set for an operator that implements the inner product between the pixel intensity in the local area and the kernel weight.

[0197] Example 28 provides a method comprising: determining a first local area in a height-time plane of a first input feature having a height dimension, a width dimension, and a time dimension; processing the first local area by a height-time spatiotemporal convolution block using one or more first parameters trained based on different spatiotemporal convolution types; determining a second local area in a width-time plane of a second input feature having a height dimension, a width dimension, and a time dimension; and processing the second local area by a width-time spatiotemporal convolution block using one or more second parameters trained based on different spatiotemporal convolution types.

[0198] Example 29 provides a method as described in Example 28, wherein the different spatiotemporal convolution types include a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity difference between a first pixel pair of an input local area selected using a first selection strategy and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between a second pixel pair of the input local area selected using a second selection strategy and a second kernel weight.

[0199] Example 30 provides a method as described in Example 28 or 29, wherein the different spatiotemporal convolution types include a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity of an input local area and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between pixel pairs of the input local area selected using a selection strategy and a second kernel weight.

[0200] Example 31 provides a method as described in any of Examples 28-30, wherein processing the first local region using the first parameter includes determining a first inner product between the first local region and the first parameter.

[0201] Example 32 provides a method as described in any of Examples 28-31, wherein processing the second local region using the second parameter includes determining a second inner product between the second local region and the second parameter.

[0202] Example 33 provides a method as described in any of Examples 28-32, wherein: the first input features are added element-by-element to the output of the height-time spatiotemporal block; and the second input features include features added element-by-element.

[0203] Example 34 provides a method as described in any of Examples 28-33, wherein: the second input features are added element-by-element to the output of the width-time spatiotemporal block; and the first input features include features added element-by-element.

[0204] Example 35 provides a method for determining parameters of a machine learning model, comprising: updating the parameters of the machine learning model using training data, wherein: the machine learning model includes a first spatiotemporal convolution block that processes a first local area in a first two-dimensional spatiotemporal plane that processes a first input feature having at least three dimensions; the first spatiotemporal convolution block includes a first parallel operator that implements a first convolution type and a second parallel operator that implements a second convolution type different from the first convolution type; the first parallel operator has an updated first parameter set; and the second parallel operator has an updated second parameter set; fusing the first and second parameter sets into a first fused parameter set; and using the first fused parameter set in a first spatiotemporal convolutional neural network layer to process features extracted from one or more images.

[0205] Example 36 provides a method as described in Example 35, wherein: the first parallel operator determines a first inner product between a pixel intensity difference between a first pair of pixels in the first local area and a first kernel weight, the first pair being selected using a first selection strategy; and the second parallel operator determines a second inner product between a pixel intensity difference between a second pair of pixels in the first local area and a second kernel weight, the second pair being selected using a second selection strategy.

[0206] Example 37 provides a method as described in Example 35 or 36, wherein: the machine learning model includes a second spatiotemporal convolution block, which processes a second local area in a second two-dimensional spatiotemporal plane of a second input feature having at least three dimensions; and the second two-dimensional spatiotemporal plane is orthogonal to the first two-dimensional spatiotemporal plane.

[0207] Example 38 provides a method as described in Example 37, wherein: the second spatiotemporal convolution block includes a third parallel operator that implements the first convolution type and a fourth parallel operator that implements the second convolution type; the third parallel operator has an updated third parameter set; and the fourth parallel operator has an updated fourth parameter set.

[0208] Example 39 provides a method as described in Example 38, further comprising fusing the third and fourth parameter sets into a second fused parameter set; and using the second fused parameter set in a second spatiotemporal convolutional neural network layer to process features extracted from one or more images.

[0209] Embodiment 40 provides a method as described in any of Embodiments 35-39, wherein: the first parameter set has a first weighting parameter; the second parameter set has a second weighting parameter; and fusing the first and second parameter sets includes summing the parameters in the first and second parameter sets using the first and second weighting parameters.

[0210] Example 41 provides a method as described in any of Examples 35-40, further comprising transforming the first set of parameters into an equivalent set of parameters for an operator that implements an inner product between pixel intensities and kernel weights in the first local region.

[0211] Example 42 provides a method as described in any of Examples 35-41, further comprising transforming the second set of parameters into an equivalent set of parameters for an operator that implements an inner product between pixel intensities and kernel weights in the first local region.

[0212] Example 43 provides a method as described in any of Examples 35-42, wherein the first two-dimensional space-time plane is a height-time plane of the first input feature having a height dimension, a width dimension, and a time dimension.

[0213] Example 44 provides a method as described in any of Examples 35-42, wherein the first two-dimensional space-time plane is a width-time plane of the first input feature having a height dimension, a width dimension, and a time dimension.

[0214] Example 45 provides a device comprising one or more processors; and one or more storage devices, wherein the one or more storage devices store a spatiotemporal differential convolution module having processing operations performed by the one or more processors, the spatiotemporal differential convolution module comprising a first spatiotemporal differential convolution block, which processes a first local area in a height-time plane of an input feature space having a height dimension, a width dimension, and a time dimension; and a second spatiotemporal differential convolution block, which processes a second local area in a width-time plane of the input feature space.

[0215] Example 46 provides an apparatus as described in Example 45, wherein the first spatiotemporal differential convolution block includes one or more first parameters trained based on different spatiotemporal convolution types.

[0216] Example 47 provides an apparatus as described in Example 46, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity difference between a first pixel pair of the first local area selected using a first selection strategy and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between a second pixel pair of the first local area selected using a second selection strategy and a second kernel weight.

[0217] Example 48 provides an apparatus as described in Example 46 or 47, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity of the first local area and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between pixel pairs of the first local area selected using a selection strategy and a second kernel weight.

[0218] Example 49 provides an apparatus as described in any of Examples 45-48, wherein the second spatiotemporal differential convolution block includes one or more second parameters trained based on different spatiotemporal convolution types.

[0219] Example 50 provides an apparatus as described in Example 49, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity difference between a first pixel pair of the first local area selected using a first selection strategy and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between a second pixel pair of the first local area selected using a second selection strategy and a second kernel weight.

[0220] Example 51 provides an apparatus as described in Example 49 or 50, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type, which calculates a first inner product between a first pixel intensity of the first local area and a first kernel weight; and a second spatiotemporal convolution type, which calculates a second inner product between a second pixel intensity difference between pixel pairs of the first local area selected using a selection strategy and a second kernel weight.

[0221] Example 52 provides a device as described in any of Examples 45-51, wherein: the first spatiotemporal differential convolution block processes a first input and generates a first output; the first input is added element-by-element to the first output; the result of the element-by-element addition is provided as a second input to the second spatiotemporal differential convolution block; the second spatiotemporal differential convolution block generates a second output; and the second input is added element-by-element to the second output.

[0222] Example 53 provides a device as described in any of Examples 45-51, wherein: the second spatiotemporal differential convolution block processes a first input and generates a first output; the first input is added element-by-element to the first output; the result of the element-by-element addition is provided to the first spatiotemporal differential convolution block as a second input; the first spatiotemporal differential convolution block generates a second output; and the second input is added element-by-element to the second output.

[0223] Example 54 includes one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods provided in Examples 1-9 and 24-44.

[0224] Example 55 is an apparatus comprising means for performing or means for performing any of the computer-implemented methods provided in Examples 1-9 and 24-44. Variants and other notes

[0225] The above description of the illustrated implementation of the present disclosure (including the content described in the abstract) is not intended to be exhaustive or limit the present disclosure to the disclosed precise form. Although specific implementations and examples of the present disclosure are described herein for illustrative purposes, as those skilled in the relevant art will recognize, various equivalent modifications are possible within the scope of the present disclosure. Based on the above detailed description, these modifications can be made to the present disclosure.

[0226] For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details, and / or the present disclosure may be practiced with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative implementations.

[0227] In addition, reference is made to the accompanying drawings which form a part of this document, and in which are shown by way of illustration embodiments that may be practiced. It will be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be understood in a limiting sense.

[0228] Various operations can be described as multiple discrete actions or operations in a manner that is most helpful for understanding the disclosed subject matter. However, the order of description should not be interpreted as implying that these operations are necessarily sequence-dependent. In particular, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed, or the described operations may be omitted.

[0229] For purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B or C" or the phrase "A, B and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range.

[0230] This description uses the phrases "in one embodiment" or "in an embodiment", each of which may refer to one or more of the same or different embodiments. As used with respect to the embodiments of the present disclosure, the terms "including", "comprising", "having" and the like are synonymous. The present disclosure may use perspective-based descriptions such as "upper", "lower", "top", "bottom" and "side" to explain the various features of the accompanying drawings, but these terms are merely for ease of discussion and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise stated, the use of the sequential adjectives "first", "second", and "third", etc., which describe common objects, merely indicates that different instances of the same object are referenced, and is not intended to imply that the objects so described must be in a given sequence in time, space, rank, or in any other way.

[0231] In the following detailed description, various aspects of the illustrative implementations will be described using terms that are commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0232] The terms "substantially," "close," "approximately," "near," and "about" generally refer to within + / - 20% of a target value, as described herein or as known in the art. Similarly, terms indicating the orientation of various elements (e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements) generally refer to within + / - 5-20% of a target value, as described herein or as known in the art.

[0233] Furthermore, the terms "comprise," "comprising," "include," "including," "have," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, or apparatus. Furthermore, the term "or" refers to an inclusive "or" rather than an exclusive "or."

[0234] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein.The details of one or more implementations of the subject matter described in this specification are set forth in the description and the drawings. The present disclosure thus provides the following technical solutions: Technical solution 1. A method comprising: Inputting one or more images into a first backbone stage of a machine learning model, the first backbone stage comprising a first spatial difference layer for performing convolution using one or more first parameters trained based on different spatial convolution types; generating, by the first backbone stage, a first feature map based on the one or more images; inputting the first feature map into a second backbone stage of the machine learning model; generating, by the second backbone level, a second feature map based on the first feature map; Inputting the second feature map into the first branch of the machine learning model; Generating, by the first branch, a third feature map based on the second feature map; Inputting the first feature map and the third feature map into a second branch of the machine learning model; combining, by the second branch, the first feature map and the third feature map to generate a first combined feature map; and One or more predictions are generated, by the second branch, regarding the one or more images based on the first combined feature map. Technical solution 2. The method according to technical solution 1, wherein: Generating the one or more predictions by the second branch includes processing the first combined feature map by a spatiotemporal difference convolution module in the second branch; and The spatiotemporal differential convolution module performs convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map. Technical Solution 3. A method as described in Technical Solution 2, wherein the spatiotemporal differential convolution module uses one or more second parameters trained based on different spatiotemporal convolution types to perform convolution. Technical Solution 4. The method according to Technical Solution 1, wherein the first backbone level further comprises: A second spatial difference layer is used to perform a convolution on the output of the first spatial difference layer using one or more third parameters trained based on a different spatial convolution type. Technical solution 5. The method according to technical solution 1 further includes: inputting the second feature map into a third backbone stage of the machine learning model; Generating, by the third backbone level, a fourth feature map based on the second feature map; Inputting the fourth feature map into a third branch of the machine learning model; Generating a fifth feature map based on the fourth feature map by the third branch; Inputting the fifth feature map into the first branch; and The first branch combines the second feature map with the fifth feature map to generate a second combined feature map; The first branch generates the third feature map based on the second combined feature map. Technical Solution 6. The method according to Technical Solution 5 further includes: inputting the fourth feature map into a fourth backbone stage of the machine learning model; Generating, by the fourth backbone level, a sixth feature map based on the fourth feature map; Inputting the sixth feature map into a fourth branch of the machine learning model; The fourth branch generates a seventh feature map based on the sixth feature map; Inputting the seventh feature map into the third branch; and The third branch combines the fourth feature map with the seventh feature map to generate a third combined feature map; The third branch generates the fifth feature map based on the third combined feature map. Technical solution 7. The method according to technical solution 1, wherein: Generating the one or more predictions by the second branch includes processing the first combined feature map by a dilated convolution module in the second branch; and The dilated convolution module performs one or more dilated convolutions. Technical solution 8. The method according to technical solution 1, wherein: Generating, by the second branch, the one or more predictions comprises: combining, in the second branch, the output of the spatiotemporal difference convolution module and the output of the dilated convolution module to generate a fourth combined feature map; and Processing the fourth combined feature map by a spatial attention module in the second branch; The spatiotemporal differential convolution module performs convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map; The dilated convolution module performs one or more dilated convolutions; and The spatial attention module performs convolution to extract spatial correlation information. Technical solution 9. A device comprising: one or more processors; and one or more storage devices storing a machine learning model having processing operations executed by the one or more processors, the machine learning model comprising: a first backbone stage that receives one or more images and generates a first feature map, the first backbone stage comprising a first spatial difference layer for performing a convolution using one or more first parameters trained based on different spatial convolution types; a second backbone stage, the second backbone stage receiving the first feature map from the first backbone stage and generating a second feature map based on the first feature map; a first branch, the first branch receiving the second feature map from the second backbone stage and generating a third feature map based on the second feature map; and a second branch that receives the first feature map from the first backbone level and the third feature map from the first branch, combines the first feature map with the third feature map to generate a first combined feature map, and generates one or more predictions about the one or more images based on the first combined feature map. Technical Solution 10. An apparatus as described in Technical Solution 9, wherein the second branch includes a spatiotemporal differential convolution module, and the spatiotemporal differential convolution module is used to perform convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map. Technical Solution 11. The device according to Technical Solution 10, wherein the spatiotemporal differential convolution module comprises: a first spatiotemporal differential convolution block that performs convolution on a first local region in a height-time plane; and A second spatiotemporal differential convolution block, wherein the second spatiotemporal differential convolution block processes a second local area in a width-time plane. Technical Solution 12. An apparatus as described in Technical Solution 10, wherein the spatiotemporal differential convolution module performs convolution using one or more second parameters trained based on different spatiotemporal convolution types. Technical Solution 13. The device according to Technical Solution 12, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type that computes a first inner product between a first pixel intensity difference and a first kernel weight between a first pair of pixels of a local region selected using a first selection strategy; and A second spatiotemporal convolution type that calculates a second inner product between a second pixel intensity difference between a second pair of pixels of the local area selected using a second selection strategy and a second kernel weight. Technical Solution 14. The device according to Technical Solution 12, wherein the different spatiotemporal convolution types include: a first spatiotemporal convolution type that calculates a first inner product between a first pixel intensity of a local area and a first kernel weight; and A second spatiotemporal convolution type that calculates a second inner product between pixel intensity differences between pairs of pixels of the local area selected using a selection strategy and a second kernel weight. Technical solution 15. The device as described in Technical Solution 9, wherein the different spatial convolution types include: a first convolution type that computes a first inner product between a first pixel intensity difference between a first pair of pixels selected using a first selection strategy and a first kernel weight; and A second convolution type that computes a second inner product between a second pixel intensity difference between a second pair of pixels selected using a second selection strategy different from the first selection strategy and a second kernel weight. Technical solution 16. The device as described in Technical Solution 9, wherein the different spatial convolution types include: a first convolution type that computes a first inner product between a first pixel intensity and a first kernel weight; and A second convolution type that computes a second inner product between a kernel weight and a pixel intensity difference between pairs of pixels selected using a selection strategy. Technical Solution 17. An apparatus as described in Technical Solution 9, wherein the second backbone level includes a visual transformer block, which is used for distance relationships between image regions. Technical solution 18. The device according to technical solution 9, wherein the second branch further includes one or more of the following modules: a dilated convolution module, configured to receive the first combined feature map and perform one or more dilated convolutions; and A spatial attention module, which is used to perform convolution and extract spatial correlation information. Technical solution 19. One or more non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to: Inputting one or more images into a first backbone stage of a machine learning model, the first backbone stage comprising a first spatial difference layer for performing convolution using one or more first parameters trained based on different spatial convolution types; generating, by the first backbone stage, a first feature map based on the one or more images; inputting the first feature map into a second backbone stage of the machine learning model; generating, by the second backbone level, a second feature map based on the first feature map; Inputting the second feature map into the first branch of the machine learning model; Generating, by the first branch, a third feature map based on the second feature map; Inputting the first feature map and the third feature map into a second branch of the machine learning model; combining, by the second branch, the first feature map and the third feature map to generate a first combined feature map; and One or more predictions are generated, by the second branch, regarding the one or more images based on the first combined feature map. Technical solution 20. One or more non-transitory computer-readable media as described in Technical Solution 19, wherein generating the second feature map by the second backbone level includes: The distance relationship between image regions is extracted by the visual transformer block in the second backbone stage.

Claims

1. A method comprising: Inputting one or more images into a first backbone stage of a machine learning model, the first backbone stage comprising a first spatial difference layer for performing convolution using one or more first parameters trained based on different spatial convolution types; generating, by the first backbone stage, a first feature map based on the one or more images; inputting the first feature map into a second backbone stage of the machine learning model; generating, by the second backbone level, a second feature map based on the first feature map; Inputting the second feature map into the first branch of the machine learning model; Generating, by the first branch, a third feature map based on the second feature map; Inputting the first feature map and the third feature map into a second branch of the machine learning model; The second branch combines the first feature map with the third feature map to generate a first combined feature map; as well as One or more predictions are generated, by the second branch, regarding the one or more images based on the first combined feature map.

2. The method of claim 1, wherein: Generating the one or more predictions by the second branch includes processing the first combined feature map by a spatiotemporal difference convolution module in the second branch; and The spatiotemporal differential convolution module performs convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map.

3. The method of claim 2, wherein: The spatiotemporal differential convolution module performs convolution using one or more second parameters trained based on different spatiotemporal convolution types.

4. The method according to any one of claims 1 to 3, wherein: The first backbone level also includes: A second spatial difference layer is used to perform a convolution on the output of the first spatial difference layer using one or more third parameters trained based on a different spatial convolution type.

5. The method according to any one of claims 1 to 3, further comprising: inputting the second feature map into a third backbone stage of the machine learning model; Generating, by the third backbone level, a fourth feature map based on the second feature map; Inputting the fourth feature map into a third branch of the machine learning model; Generating a fifth feature map based on the fourth feature map by the third branch; Inputting the fifth feature map into the first branch; as well as The first branch combines the second feature map with the fifth feature map to generate a second combined feature map; The first branch generates the third feature map based on the second combined feature map.

6. The method of claim 5, further comprising: inputting the fourth feature map into a fourth backbone stage of the machine learning model; Generating, by the fourth backbone level, a sixth feature map based on the fourth feature map; Inputting the sixth feature map into a fourth branch of the machine learning model; The fourth branch generates a seventh feature map based on the sixth feature map; Inputting the seventh feature map into the third branch; as well as The third branch combines the fourth feature map with the seventh feature map to generate a third combined feature map; The third branch generates the fifth feature map based on the third combined feature map.

7. The method of any one of claims 1 to 3, wherein: Generating the one or more predictions by the second branch includes processing the first combined feature map by a dilated convolution module in the second branch; and The dilated convolution module performs one or more dilated convolutions.

8. The method of any one of claims 1 to 3, wherein: Generating, by the second branch, the one or more predictions comprises: combining, in the second branch, the output of the spatiotemporal difference convolution module and the output of the dilated convolution module to generate a fourth combined feature map; and Processing the fourth combined feature map by a spatial attention module in the second branch; The spatiotemporal differential convolution module performs convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map; The dilated convolution module performs one or more dilated convolutions; and The spatial attention module performs convolution to extract spatial correlation information.

9. The method according to any one of claims 1 to 3, wherein: Generating the second feature map by the second backbone level includes: The distance relationship between image regions is extracted by the visual transformer block in the second backbone stage.

10. A device comprising: one or more processors; as well as one or more storage devices storing a machine learning model having processing operations executed by the one or more processors, the machine learning model comprising: a first backbone stage that receives one or more images and generates a first feature map, the first backbone stage comprising a first spatial difference layer for performing a convolution using one or more first parameters trained based on different spatial convolution types; a second backbone stage, the second backbone stage receiving the first feature map from the first backbone stage and generating a second feature map based on the first feature map; a first branch, the first branch receiving the second feature map from the second backbone stage and generating a third feature map based on the second feature map; and a second branch that receives the first feature map from the first backbone level and the third feature map from the first branch, combines the first feature map with the third feature map to generate a first combined feature map, and generates one or more predictions about the one or more images based on the first combined feature map.

11. The device of claim 10, wherein: The second branch comprises a spatiotemporal differential convolution module, which is used to perform convolution on one or more two-dimensional spatiotemporal planes of the first combined feature map.

12. The device of claim 11, wherein: The spatiotemporal differential convolution module includes: a first spatiotemporal differential convolution block that performs convolution on a first local region in a height-time plane; and A second spatiotemporal differential convolution block, wherein the second spatiotemporal differential convolution block processes a second local area in a width-time plane.

13. The apparatus of claim 11 or 12, wherein: The spatiotemporal differential convolution module performs convolution using one or more second parameters trained based on different spatiotemporal convolution types.

14. The device of claim 13, wherein: The different types of spatiotemporal convolution include: a first spatiotemporal convolution type that computes a first inner product between a first pixel intensity difference and a first kernel weight between a first pair of pixels of a local region selected using a first selection strategy; and A second spatiotemporal convolution type that calculates a second inner product between a second pixel intensity difference between a second pair of pixels of the local area selected using a second selection strategy and a second kernel weight.

15. The apparatus of claim 13, wherein: The different types of spatiotemporal convolution include: a first spatiotemporal convolution type that calculates a first inner product between a first pixel intensity of a local area and a first kernel weight; and A second spatiotemporal convolution type that calculates a second inner product between pixel intensity differences between pairs of pixels of the local area selected using a selection strategy and a second kernel weight.

16. The device according to any one of claims 10 to 12, wherein: The first backbone level further comprises one or more additional spatial difference layers, each additional spatial difference layer being used to perform convolution using one or more additional parameters trained based on different spatial convolution types.

17. The apparatus of any one of claims 10 to 12, wherein: The different spatial convolution types include: a first convolution type that computes a first inner product between a first pixel intensity difference between a first pair of pixels selected using a first selection strategy and a first kernel weight; and A second convolution type that computes a second inner product between a second pixel intensity difference between a second pair of pixels selected using a second selection strategy different from the first selection strategy and a second kernel weight.

18. The apparatus of any one of claims 10 to 12, wherein: The different spatial convolution types include: a first convolution type that computes a first inner product between a first pixel intensity and a first kernel weight; and A second convolution type that computes a second inner product between a kernel weight and a pixel intensity difference between pairs of pixels selected using a selection strategy.

19. The apparatus of any one of claims 10 to 12, wherein: The machine learning model also includes: a third backbone stage, the third backbone stage receiving the second feature map from the second backbone stage and generating a fourth feature map based on the second feature map; and a third branch, the third branch receiving the fourth feature map from the third backbone level, and generating a fifth feature map based on the fourth feature map; The first branch further receives the fifth feature map, combines the second feature map with the fifth feature map to generate a second combined feature map, and generates the third feature map based on the second combined feature map.

20. The apparatus of claim 19, wherein: The machine learning model also includes: a fourth backbone stage, the fourth backbone stage receiving the fourth feature map from the third backbone stage and generating a sixth feature map based on the fourth feature map; and a fourth branch, the fourth branch receiving the sixth feature map from the fourth backbone level, and generating a seventh feature map based on the sixth feature map; The third branch further receives the seventh feature map, combines the fourth feature map with the seventh feature map to generate a third combined feature map, and generates the fifth feature map based on the third combined feature map.

21. The apparatus of any one of claims 10 to 12, wherein: The second branch also includes a dilated convolution module, which is used to receive the first combined feature map and perform one or more dilated convolutions.

22. The apparatus of any one of claims 10 to 12, wherein: The second branch also includes a spatial attention module, which is used to perform convolution and extract spatial correlation information.

23. The apparatus of any one of claims 10 to 12, wherein: The second backbone stage includes a visual transformer block for distance relations between image regions.

24. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods of claims 1-3.

Images