Neural network with point mesh convolutional layer

Abstract

The PGConv layer extracts features from the data sample of the grid structure. The PGConv layer may receive an input feature map including a grid representation of the object, the grid representation being generated from a graph representation of the object. The grid representation includes node elements arranged in a grid pattern. The PGConv layer may perform padding on the grid representation. The result of the padding is an IFM including node elements and additional node elements. The additional node elements may have a zero value or a value of a node element in the grid representation. The PGConv layer may also generate an attention kernel that includes an attention weight determined based on the IFM. The PGConv layer may generate a dynamic kernel based on the kernel of interest and a convolutional kernel generated by training. The PGConv layer may also perform MAC operations on the IFM and the dynamic kernel and generate an OFM.

Description

Technical Field

[0001] The present disclosure relates generally to deep neural networks (DNNs), and more specifically to DNNs with point grid convolutional (PGConv) layers. Background Art

[0002] DNNs are widely used in various artificial intelligence (AI) applications including computer vision, speech recognition, and natural language processing due to their ability to achieve high accuracy. One type of DNN is the graph convolutional network (GCN). GCN is one of the popular solutions for various AI applications such as human pose lifting, skeleton-based human action recognition, mesh reconstruction, traffic navigation, social network analysis, recommendation systems, scientific computing, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To aid this description, similar reference numerals designate similar structural elements. The embodiments are illustrated in the accompanying drawings by way of example and not by way of limitation.

[0004] Figure 1 An example layer structure of a convolutional neural network (CNN) is illustrated according to various embodiments.

[0005] Figure 2 is a block diagram of an example PGConv layer according to various embodiments.

[0006] Figure 3 Example point grid convolutions are illustrated in accordance with various embodiments.

[0007] Figures 4A-4C Example semantic mesh transformations are illustrated in accordance with various embodiments.

[0008] Figures 5A-5D Various filling methods are illustrated according to various embodiments.

[0009] Figure 6 Example dynamic convolutions are illustrated in accordance with various embodiments.

[0010] Figure 7 An example PGConv layer including multiple branches is illustrated according to various embodiments.

[0011] Figure 8 is a block diagram of a point grid system according to various embodiments.

[0012] Fig. 9 According to various embodiments, example three-dimensional (3D) pose estimation based on two-dimensional (2D) images is illustrated.

[0013] Fig.10 A deep learning (DL) environment is illustrated according to various embodiments.

[0014] Fig.11 is a flow chart illustrating a method of point grid convolution with padding assistance according to various embodiments.

[0015] Fig.12 is a flow chart illustrating a method of dynamic point grid convolution according to various embodiments.

[0016] Fig.13 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION

[0017] Overview

[0018] GCN is a variant of CNN. GCN is adopted to operate on data samples represented in the form of irregular graph structures (such as images). Take pose lifting network as an example, which is a specific type of GCN. Pose lifting network is usually trained to estimate 3D human pose given the positions of body joints detected from 2D input. Estimating 3D human pose from images and videos has a wide range of applications, such as human action recognition, human robot / computer interaction, augmented reality, animation and games. In general, existing pose lifting networks can be grouped into four families of solutions: (1) Fully Connected Network (FCN); (2) Semantic Graph Convolution Network (SGCN); (3) Locally Connected Network (LCN); and (4) other variants of FCN, SGCN and LCN. All of these pose lifting networks operate on data samples represented in the form of irregular graph structures.

[0019] However, the use of such data samples may limit the performance of GCN in some image-based applications. In addition, it can also lead to irregular workloads, such as irregular sparse tensor operations. These irregular workloads prevent the efficient execution of GCN on many AI processors, such as GPU (Graphics Processing Unit), CPU (Central Processing Unit), VPU (Vision Processing Unit), TPU (Tensor Processing Unit), etc. Therefore, improved techniques for convolution operations on graph-structured data are needed.

[0020] Embodiments of the present disclosure can improve at least some of the above challenges and problems by providing a PGConv layer for DNN. In various embodiments of the present disclosure, graph structure data is converted into data with a regular structure, for example, grid structure data. The PGConv layer can perform convolution on the grid structure data samples to extract features from the grid structure data samples. The PGConv layer can perform padding on the grid structure data samples to maintain the spatial size of the grid structure data samples.

[0021] For example, the PGConv layer may receive an IFM that includes a data sample of a grid structure that includes grid nodes (also referred to as "node elements") arranged in a grid pattern. The PGConv layer may add additional node elements to the IFM to generate an expanded feature map (EFM). The spatial size of the EFM is larger than that of the IFM. In some embodiments, the PGConv layer generates the EFM by zero padding, wherein additional node elements with zero values ​​are added to one or more edges of the IFM. In other embodiments, the PGConv layer generates the EFM by circular padding, wherein additional node elements with node element values ​​on an edge of the IFM are added to the opposite edge of the IFM. The PGConv layer may use a combination of zero padding and circular padding. The grid pattern of the EFM may be different from that of the IFM. In one example, the IFM may have a 2D grid pattern, while the EFM may have a 3D grid pattern. The PGConv layer may further perform convolution with the EFM and the kernel to generate an output feature map (OFM). The spatial size of the OFM may be the same as that of the IFM.

[0022] Padding can solve the problem caused by the shrinkage of feature maps during convolution. Due to the shrinkage of feature maps during convolution, the number of convolutions that can be performed is limited, especially in scenarios where the spatial size of the IFM is relatively small. In addition, the node elements on the corners and edges of the IFM are used much less frequently than the node elements in the middle. Padding can maintain the size of the IFM and increase the contribution of node elements at the border of the IFM by moving them to the middle of the EFM.

[0023] The PGConv layer can perform dynamic point grid convolution, also referred to as dynamic convolution below. The PGConv layer can generate attention for IFM or EFM using an attention function. Attention can include one or more attention kernels, including attention weights determined based on the IFM or EFM. The PGConv layer can use attention to scale the convolution kernel, which is determined from the training of the PGConv layer. Convolution kernels are static because their weights do not change with changes in IFM. Since the attention kernel changes with changes in IFM, the scaled convolution kernel adopts the dynamic nature of attention. The PGConv layer can perform dynamic convolution using IFM (or EFM generated from IFM) and scaled convolution kernels. This dynamic convolution is input-dependent and position-aware, and can have better feature learning capabilities than traditional convolution. Thus, dynamic convolution may be more advantageous in various applications (e.g., image-based applications).

[0024] For purposes of illustration, specific numbers, materials, and configurations are described to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be implemented without the specific details, or / and may be implemented with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative implementations.

[0025] In addition, reference is made to the accompanying drawings, which form a part of this document, in which embodiments that can be implemented are illustrated in a graphical manner. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be understood in a limiting sense.

[0026] Various operations may be described in turn as a plurality of discrete actions or operations in a manner that is most helpful for understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these operations must be dependent on the order. Specifically, these operations may not be performed in the order presented. The operations described may be performed in an order different from the described embodiments. Various additional operations may be performed or the operations described may be omitted in additional embodiments.

[0027] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a measurement range includes both ends of the measurement range.

[0028] The description uses the phrases "in one embodiment" or "in an embodiment", each of which refers to one or more identical or different embodiments. The terms "including", "comprising", "having", etc., used for the embodiments of the present disclosure, are synonymous. The present disclosure may use perspective-based descriptions, such as "above", "below", "top", "bottom", and "side", to illustrate various features of the drawings, but these terms are only for ease of discussion and do not imply a desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise indicated, the use of ordinal adjectives "first", "second", and "third", etc. to describe common objects simply indicates that different instances of similar objects are referenced, and is not intended to imply that the objects described in this way must be in a given sequence in time, space, ranking, or in any other way.

[0029] In the detailed description that follows, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0030] The terms "substantially," "close," "roughly," "near," and "approximately" generally refer to input operating objects based on specific values ​​as described herein or as known in the art, within + / -20% of target values. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to input operating objects based on specific values ​​as described herein or as known in the art, within + / -5-20% of target values.

[0031] Furthermore, the terms "includes," "comprising," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Additionally, the term "or" refers to an inclusive "or" rather than an exclusive "or."

[0032] The DNN systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is entirely responsible for all of the desirable properties disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the following description and accompanying drawings.

[0033] Example CNN layer structure

[0034] Figure 1 An example layer structure of CNN 100 is illustrated according to various embodiments. For illustration, CNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1 In the embodiment of the present invention, CNN 100 receives an input image 105 including objects 115, 125, and 135. CNN 100 includes a sequence of layers, including a plurality of convolutional layers 110 (individually referred to as "convolutional layers 110"), a plurality of pooling layers 120 (individually referred to as "pooling layers 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layers 130"). In other embodiments, CNN 100 may include fewer, more, or different layers.

[0035] The convolutional layer 110 summarizes the presence of features in the input image 105. Figure 1 In an embodiment of the present invention, the first layer of CNN 100 is a convolutional layer 110. Convolutional layer 110 functions as a feature extractor. Convolutional layer 110 may receive an input and output features extracted from the input. In one example, convolutional layer 110 performs a convolution on IFM 140 using kernel 150, generates OFM 160 from the convolution, and passes OFM 160 to the next layer in the sequence. IFM 140 may include multiple IFM arrays, each IFM array corresponding to a channel (i.e., input channel) of IFM 140. The IFM array includes multiple input elements arranged in rows and columns. The number of rows and columns may define the spatial size of IFM 140. For the first convolutional layer 110, which is also the first layer of CNN 100, IFM 140 is input image 105. For other convolutional layers, IFM 140 may be the output of another convolutional layer 110 or the output of pooling layer 120.

[0036] Kernel 150 may be a convolution kernel. Figure 1 In the embodiment of the present invention, the kernel 150 includes three convolution filters 155 (individually referred to as convolution filters 155). In other embodiments, the kernel 150 may include a different number of convolution filters 155. The number of convolution filters 155 can be equal to the number of channels in the OFM 160. Each convolution filter 155 includes a plurality of weight arrays, which can correspond to a plurality of channels of the IFM 140. The number of weight arrays in the convolution filter 155 can be equal to the number of channels in the IFM 140. The weight array includes weights arranged in rows and columns. The number of rows and columns can define the spatial size of the kernel 150. The value of the weight can be determined by training the CNN 100.

[0037] OFM 160 may include a plurality of IFM arrays, each IFM array corresponding to a channel (i.e., output channel) of OFM 160. OFM arrays include a plurality of output elements arranged in rows and columns. The number of rows and columns may define the spatial size of OFM 160. The spatial size of OFM 160 may be smaller than the spatial size of IFM 140. In some embodiments, the spatial size of IFM 140 may be maintained by padding.

[0038] Convolution can be a linear operation that involves multiplying a weight operation object in the kernel 150 with a patch of the size of the weight operation object of the IFM 140. The weight operation object can be a weight matrix in the kernel 150, such as a 2-dimensional array of weights, where the weights are arranged in columns and rows. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of the kernel 150 in extracting features from the IFM 140. The weight operation object can be smaller than the IFM 140. The multiplication can be an element-wise multiplication between a patch of the size of the weight operation object of the IFM 140 and the corresponding weight operation object, followed by addition, always producing a single value. Because its result is a single value, this operation is often called a "scalar product."

[0039] In some embodiments, using a weight operation object that is smaller than the IFM 140 is intentional because it allows the same weight operation object (set of weights) to be multiplied with the IFM 140 multiple times at different points on the IFM 140. Specifically, the weight operation object is systematically applied to each overlapping portion of the IFM 140, or patch of the weight operation object size, from left to right and from top to bottom. The result of multiplying the weight operation object with the IFM 140 once is a single value. Since the weight operation object is applied to the IFM 140 multiple times, the result of the multiplication is a two-dimensional array representing the output values ​​of the weight operation of the IFM 140. Therefore, the 2-dimensional output array from this operation is called a "feature map."

[0040] In some embodiments, OFM 160 is passed through an activation function. An example activation function is a rectified linear activation function (ReLU). ReLU is a calculation that directly returns the value provided as an input, or returns a value of zero if the input is zero or less. Convolutional layer 110 can receive several images as input and calculate the convolution of each of them with each weight operation object. This process can be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 after the convolutional layer 110 that generates OFM 160 in the sequence). The subsequent convolutional layer 110 performs convolution on OFM 160 with the new weight operation object and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be weighted again by a further subsequent convolutional layer 110, and so on and so forth.

[0041] In some embodiments, the convolution layer 110 has four hyperparameters: the number of weight operation objects, the size F of the weight operation object (e.g., the size of the weight operation object is F×F×D pixels), the step size S of dragging the window corresponding to the weight operation object on the image (e.g., a step size of one means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thick to the input image of the convolution layer 110). The convolution layer 110 can perform various types of convolutions, such as 2-dimensional convolutions, dilated or hollow convolutions, spatially separable convolutions, depthwise separable convolutions, transposed convolutions, and the like. The CNN 100 includes 16 convolution layers 110. In other embodiments, the CNN 100 may include a different number of convolution layers.

[0042] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in patches of the feature map. The pooling layer 120 is placed between two convolutional layers 110: the preceding convolutional layer 110 (the convolutional layer 110 before the pooling layer 120 in the sequence of layers) and the subsequent convolutional layer 110 (the convolutional layer 110 after the pooling layer 120 in the sequence of layers). In some embodiments, for example, after an activation function (e.g., ReLU) is applied to the OFM 160, the pooling layer 120 is added after the convolutional layer 110.

[0043] The pooling layer 120 receives the feature map generated by the previous convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while retaining their important characteristics. Therefore, the pooling operation improves the efficiency of the CNN and avoids over-learning. The pooling layer 120 can perform a pooling operation by average pooling (calculating the average value for each patch on the feature map), maximum pooling (calculating the maximum value for each patch of the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2×2 pixels applied with a span of two pixels, so that the pooling operation reduces the size of the feature map by 2 times, for example, the number of pixels or values ​​in the feature map is reduced to a quarter of the size. In one example, the pooling layer 120 applied to a 6×6 feature map results in a 3×3 output pooled feature map. The output of the pooling layer 120 is input into the subsequent convolutional layer 110 to further extract features. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of pooled feature maps of the same number.

[0044] The fully connected layer 130 is the last layer of the CNN. The fully connected layer 130 can be convolutional or not. The fully connected layer 130 receives an input operation object. The input operation object defines the output of the convolutional layer 110 and the pooling layer 120, and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies linear combinations and activation functions to the input operation objects and generates individual partial sums. The individual partial sums can contain as many elements as categories: element i represents the probability that the image belongs to category i. Therefore, each element is between 0 and 1, and the sum of all values ​​is one. These probabilities are calculated by the last fully connected layer 130 using a logical function (binary classification) or a softmax function (multi-class classification) as an activation function.

[0045] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operation object indicates the probability that the input image 105 belongs to a certain category. To calculate the probability, the fully connected layer 130 multiplies each input element by the weight, obtains the sum, and then applies an activation function (e.g., if N=2, it is logical, if N>2, it is softmax). This is equivalent to multiplying the input operation object with the matrix containing the weights. In one example, the individual partial sums include three probabilities: the first probability indicates that the object 115 is a tree, the second probability indicates that the object 125 is a car, and the third probability indicates that the object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual partial sums can be different.

[0046] Example PGConv layer

[0047] Figure 2 2 is a block diagram of an example PGConv layer 200 according to various embodiments. The PGConv layer 200 may extract features from a grid-structured data sample by a convolution operation (e.g., a dynamic convolution). The grid-structured data sample may be converted from a graph-structured data sample (e.g., an image) by a semantic grid transformation. The PGConv layer 200 may be Figure 1 One or more convolutional layers 110 in an embodiment. Figure 2 As shown, the PGConv layer 200 includes an interface module 210, a padding module 220, a focus module 230, a convolution operator 240, and a convolution operator 250. In other embodiments, alternative configurations, different or additional components may be included in the PGConv layer 200. For example, the PGConv layer 200 may include more than one convolution operator. In addition, the functions attributed to the components of the PGConv layer 200 may be performed by different components included in the PGConv layer 200 or by different systems.

[0048] The interface module 210 assists in implementing the communication between the PGConv layer 200 and other systems. In some embodiments, the interface module 210 establishes the communication between the PGConv layer 200 to receive the IFMs, and the PGConv layer 200 will extract features from these IFMs. The IFMs can be grid-structured data samples, which can be converted from graph-structured data through semantic grid transformation, such as Figures 4A-4C In some embodiments, the IFM includes a mesh representation of an object converted from an image of the object. The object may be a person, an animal, a plant, a tree, a building, a vehicle, a street, or other types of objects. The PGConv layer 200 may be derived from a transformation module (e.g., Figure 8The interface module 210 may also support the PGConv layer 200 to send the OFM generated by the PGConv layer 200 to other systems or devices. For example, the interface module 210 may send the feature map to another layer of the DNN including the PGConv layer 200, to another DNN, or to a module that can use the feature map for various applications, such as posture estimation, action recognition, 3D mesh reconstruction, traffic navigation, social network analysis, recommendation system, scientific computing, etc. In one example, the interface module 210 may send the feature map to an inverse transformation module (e.g., described below in conjunction with Figure 8 The inverse transformation module 860 described above is used to generate a graph, such as a 3D graph representing a pose.

[0049] The filling module 220 performs filling on the IFM received by the interface module 210 to generate an EFM. In some embodiments, the filling module 220 receives an IFM including a mesh representation of an object. The mesh representation (an example of which is shown in FIG. 1 ) Figure 4C 2 (shown in FIG. 2 ) includes a plurality of node elements arranged in a grid pattern. The grid pattern may be a 2D pattern, for example, a pattern including elements arranged in rows and columns. The node elements may be assigned to one or more elements of the grid structure. The fill module 220 adds one or more additional node elements to the grid representation to form an EFM for the convolution to be performed by the convolution operator 240. The EFM includes the node elements of the grid representation as well as the additional node elements.

[0050] Padding ensures that the OFM produced from the convolution has the same spatial size as the IFM. Without padding, the spatial size of the OFM will be smaller than the IFM. For example, for a convolution with spatial size H×P and C in Channel and C with spatial size K×K out The kernel of the filter The OFM resulting from the convolution operation has dimensions (H–K+1)×(P–K+1). In the scenario where H=8, P=8, and K=3, the OFM will have a spatial size of (6x 6), which is smaller than the spatial size of the IFM. Since the feature map shrinks as the convolution is performed, there may be an upper limit on the number of convolutions that can be performed before the input shrinks to nothing, which may prevent the development of deeper CNNs, such as CNNs with more convolutional layers. In addition, the input elements on the corners and edges of the IFM are used much less frequently than the middle part, and thus provide less contribution to the OFM than the middle part. Padding can solve these problems because padding can maintain the size of the IFM and increase the contribution of input elements at the border of the IFM by moving them to the middle of the EFM. In the example of adding additional input elements to all four sides of the IFM, the IFM can be expanded to an EFM with a spatial size of (H+2)x(P+2). Accordingly, the spatial size of the OFM is (H+2–K+1)×(P+2–K+1). In the scenario where H=8, P=8, and K=3, the spatial size of OFM will be (8 x 8), which is the same as the spatial size of IFM. In other scenarios, H or P may have other values. In addition, the value of H may be different from the value of P.

[0051] In some embodiments, the filling module 220 can perform zero filling on the IFM by adding zero-value node elements to the IFM. In one embodiment, the filling module 220 can add zero-value node elements to one or more edges of the IFM. For example, the filling module 220 can add two rows of zero-value node elements and two columns of zero-value node elements to the IFM, so that the original input elements of the IFM are surrounded by zero-value node elements. In another embodiment, the filling module 220 can add zero-value node elements to other positions of the IFM. As an addition or replacement of the zero-value node elements, the filling module 220 can add non-zero-value node elements to the IFM. For example, the filling module 220 can generate additional node elements based on the value of the node elements of one side (e.g., edge) of the IFM, and add the additional node elements to the opposite side of the IFM. The value of the additional node element may be equal to the value of the node element on the edge. The filling may cause two edges to have the same value, so that the two edges can be connected to form a 3D circular structure, such as a torus or a hollow cylinder. This filling is called "circular filling".

[0052] The filling module 220 can use different filling methods to generate an EFM with a pattern different from the IFM. In some embodiments, the IFM is a 2D grid with four edges (e.g., top, bottom, left, and right), and the filling module 220 can perform zero filling on all four edges of the IFM, and the resulting EFM will also have a 2D grid pattern, but the spatial size is larger than the IFM. The filling module 220 can perform circular filling on all four edges and generate an EFM with a torus grid pattern. The filling module 220 can also perform a mixture of zero filling and circular filling on the IFM. For example, the filling module 220 performs circular filling on the left and right sides, but performs zero filling on the top and bottom sides, and generates an EFM with a barrel-shaped grid pattern. For another example, the filling module 220 performs zero filling on the left and right sides, but performs circular filling on the top and bottom sides, and generates an EFM with a tubular grid pattern. In other embodiments, the filling module 220 can generate EFMs with other shapes. The following is combined with Figure 3 and Figures 5A-5D More details about padding are described.

[0053] The attention module 230 generates an attention kernel based on the IFM or EFM to assist in implementing dynamic convolution. The attention module 230 can generate an attention α for the IFM or EFM. The attention α can be spatially adaptive. For example, the attention α may include one or more attention kernels, and different attention kernels in the attention α may have different dimensions. The attention kernel includes an attention weight. The attention module 230 can determine the value of the attention weight based on the IFM or EFM. The attention module 230 can provide attention to the convolution operator 240, and the convolution operator 240 can use the attention α to scale the convolution kernel. The scaled convolution kernel and the IFM or EFM can be used by, for example, the convolution operator 250 to perform dynamic convolution.

[0054] Some of the following descriptions of the generation of attention α relate to IFM. However, attention module 230 may also generate attention α based on EFM from fill module 220. In addition to IFM, attention module 230 may also generate attention kernels based on convolution kernels. In some embodiments, attention module 230 may generate multiple attention kernels along multiple dimensions of the convolution kernel. For example, attention module 230 generates three attention kernels, including spatial attention kernel α s , input channel kernel α c and the output channel kernel α f . Spatial attention kernel α s Corresponding to the spatial size of the convolution filter in the convolution kernel, the input channel kernel α c Corresponding to the input channel dimension of the convolution kernel, the output channel kernel α f Corresponds to the output channel dimension of the convolution kernel. The convolution kernel is represented as In an example, three kernels of interest can be represented as:

[0055] α s ∈R K×K×1×1

[0056]

[0057] In some embodiments (e.g., embodiments where PGConv layer 200 performs dynamic convolution), attention module 230 may generate an α s ∈R K×K×1×1 For an example IFM with a spatial size of H×P (i.e., H×P node elements per channel), the attention module 230 may generate α∈R K×K×H×P The dimensions of the kernel of interest are examples. In other embodiments, the kernel of interest may have other dimensions.

[0058] In some embodiments, the attention module 230 is in the IFM D in Apply the attention function π on it, and the attention function returns the attention α, which can be expressed as:

[0059] α=π(D in )

[0060] The attention function may include one or more parameters. These parameters may be determined during the training of the PGConv layer 200. In some embodiments, the attention function includes a pooling function, an attention function, a batch normalization function, a function for the operation of a fully connected layer, other functions, or some combination of these.

[0061] In one example process of generating an attention α, the attention module 230 may generate a sequence of transformed feature maps (TFMs) through various operations. In some embodiments, the attention module 230 performs a pooling operation on the IFM to generate a first TFM, which is also referred to as a channel descriptor. The pooling operation may be an average pooling operation, such as a global pooling operation. In the average pooling operation, the attention module 230 may perform spatial aggregation of the IFM by replacing the input elements in each channel of the IFM with the average value of the input elements in the channel. The attention module 230 may further perform a transformation on the channel descriptor, for example, by using a fully connected layer (e.g., Figure 1 The fully connected layer 130 in ), batch normalizer and activation function. The fully connected layer can reduce the dimension of the channel descriptor, for example, the channel size C in Reduce to C in / r, where r is the compression ratio. The batch normalizer can perform a batch normalization operation on the output of the fully connected layer. Additionally, the activation function can apply an activation function (e.g., ReLU, etc.) to the output of the batch normalizer. The output of the activation function is a second TFM, which is also referred to as an abstract descriptor.

[0062] The abstract descriptor can be further provided to a set of fully connected layers and activation functions to generate an attention kernel. In one example, the attention module 230 includes three pairs of fully connected layers and activation functions. Each pair includes a fully connected layer and an activation function (e.g., a sigmoid activation function) and calculates a different attention kernel. The fully connected layers in a pair can convert the abstract description from a reduced channel size C to a reduced channel size C. in / r maps to the corresponding reducibility of the corresponding attention kernel. The output of the fully connected layer can be sent to the activation function in the pair. The activation function can scale the output of the fully connected layer to produce a value in the corresponding attention kernel.

[0063] The parameters used by the attention module 230 to generate the attention kernel can be determined during the process of training the PGConv layer 200. These parameters may include the parameters of the pooling operations, the operations of the fully connected layers, the activation functions, and the batch normalization described above. The attention kernel is dynamic because the value of the attention weight may change if a different IFM is input to the PGConv layer 200. Although the convolution kernel can be static (for example, because the value of the convolution weight is determined by training the PGConv layer 200 and is independent of the IFM), the kernel generated by scaling the convolution kernel with one or more attention kernels can also adopt the dynamic characteristics of the attention kernel and therefore can also be dynamic. More details about the attention kernel will be described below in conjunction with Figure 6 Give a description.

[0064] The convolution operator 240 performs convolution based on the IFM (or EFM), the attention, and the convolution kernel. For example, the convolution operator 240 can perform convolution based on the EFM from the padding module 220, the attention generated by the attention module 230 based on the EFM, and the convolution kernel. The result of the dynamic convolution is the OFM. The convolution kernel includes one or more convolution filters. The number of convolution filters in the convolution kernel can be equal to the channel size of the OFM. Each convolution filter includes a convolution weight, whose value is determined by training the PGConv layer 200, for example, training a DNN including the PGConv layer 200. An example of a convolution kernel is the above combined Figure 1The kernel 150 described in the embodiment of the present invention. The value of the convolution weight is independent of the EFM, so the convolution kernel is static. In contrast, the focus kernel in the focus is dynamic because the value of the focus weight may change as the EFM changes. The convolution operator 240 can scale the convolution kernel with the focus kernel and generate a scaled convolution kernel. The scaled convolution kernel adopts the dynamic characteristics of the focus kernel and is also called a dynamic kernel. The convolution operator 240 can perform convolution using the dynamic kernel and EFM, which is called dynamic convolution. Dynamic convolution can be expressed as:

[0065]

[0066] α∈R K×K×H×P

[0067] α ij ∈R K×K

[0068] where α ij represents the attention kernel of attention α and is multiplied element-wise with the convolution kernel W in the channel dimension, δ ij represents the grid index vector of the local patch centered at position (i,j), where i∈[1,H] and j∈[1,P].

[0069] In some embodiments, the convolution operator 240 scales the convolution kernel by element-wise multiplication on the convolution kernel and the focus kernel. s , α c and α f In an example, element-wise multiplication can be expressed as:

[0070]

[0071] in, is a dynamic kernel. In some embodiments, the convolution operator performs a series of scaling operations. Each scaling operation scales the convolution kernel with the kernel of interest. In one example, the first scaling operation is expressed as:

[0072] For i = 1...C out , j = 1...C in , k=1...s

[0073] The second scaling operation is expressed as:

[0074] For i = 1...C out , j = 1...C in , k=1...sThe second scaling operation is expressed as:

[0075] For i = 1...C out , j = 1...C in , k=1...s In other embodiments, the number or order of scaling operations may be different.

[0076] After generating the dynamic kernel, the convolution operator 240 can perform convolution based on the EFM and the dynamic kernel. In some embodiments, the convolution operator 240 includes processing elements (PEs), which can be arranged into one or more arrays. The PE can perform multiplication-accumulation (MAC) operations on the dynamic kernel and the EFM. The weights in the dynamic kernel and the elements in the EFM can be distributed to the PE, for example, based on a bitmap. Each PE may have two input signals: a weight operation object including the weights from the dynamic kernel, and an input operation object including the elements from the EFM. The PE can perform a MAC operation on the weight operation object and the input operation object, and generate an output signal. The output signal can be provided to one or more other PEs for further processing. The result of the MAC operation of the PE is an OFM. Although some of the examples provided above relate to EFM, the convolution operator 240 can also perform dynamic convolution on the IFM. In addition, the convolution operator 240 can perform conventional convolution, for example, convolution based on a convolution kernel and an IFM or an EFM.

[0077] Example point grid convolution

[0078] Figure 3 An example point grid convolution 300 is illustrated according to various embodiments. The point grid convolution 300 is a convolution of grid structured data. The point grid convolution 300 may include a multiply-accumulate (MAC) operation on the grid IFM 320 and the kernel 350. The result of the MAC operation is the grid OFM 330. In some embodiments, the kernel 350 is a dynamic kernel generated based on the convolution kernel and the attention, and the point grid convolution 300 may be a dynamic convolution. In other embodiments, the kernel 350 is a convolution kernel, and the point grid convolution 300 may be a conventional convolution, such as the above combined Figure 1 Describe the convolution.

[0079] exist Figure 3 In the embodiment of the invention, the mesh IFM 320 is mesh structure data generated from the person's graphical representation 310, for example, by Figure 8The mesh OFM 330 is generated by the semantic mesh transformation performed by the transformation module 820 in the mesh. The mesh IFM 320 can be a mesh representation of a person and can include a plurality of input channels. Each channel can include an array including a number of rows and a number of columns. The kernel 350 includes a number of filters. The mesh OFM 330 includes a plurality of output channels. The number of output channels in the mesh OFM 330 can be equal to the number of filters in the kernel 350.

[0080] In some embodiments, the point grid convolution 300 can be expressed as

[0081] D out =W*D in

[0082] in, Represents a space with size H×P and C in IFM 320 with channels, Represents C with a spatial size of K×K out filter kernels 350, for example, K = {1, 3, 5 ...} based on the spatial size of the grid IFM 320, and Represents a space with size H×P and C out OFM 330 of channels.

[0083] The grid IFM 320 is transformed into the EFM 340 by filling. For simplicity and for illustration, Figure 3 One channel of EFM 340 is shown. The number of channels in EFM 340 may be equal to the number of channels in grid IFM 320. Figure 3 As shown, the EFM 340 includes the input node elements 325 (individually referred to as “input node elements 325”) in the grid IFM 320, such as all the input node elements 325 in the grid IFM 320, and the additional node elements 345 (individually referred to as “additional node elements 345”) that are not present in the grid IFM 320. The additional node elements 345 are placed on four sides (i.e., the top side, the bottom side, the left side, and the right side) of the grid IFM 320, so that the spatial size of the EFM 340 (i.e., as shown in FIG. 1 ) is reduced. Figure 3 7×7) is larger than the grid IFM 320 (i.e., Figure 35×5 as shown). In other embodiments, the additional node elements 345 may be arranged on a subset of the four edges, rather than on all four edges. In one embodiment, the additional node elements 345 may be arranged on the top edge, but not on the bottom edge, and / or on the left side, but not on the right side. The additional node elements 345 may have a zero value or a non-zero value. In some embodiments, the additional node elements 345 on one side may have the value of the input node element 325 on the other side. In one example (e.g., an example of performing circular filling on the mesh IFM 320), the additional node elements 345 at the top of the EFM 340 have the same value as the input node elements 325 at the bottom of the mesh IFM 320. The number and order of the additional node elements 345 may also be the same as the number and order of the input node elements 325. Since the additional node elements 345 and the input node elements 325 have the same value, they may be connected, which may produce a 3D circular structure.

[0084] The point grid convolution 300 is performed on the EFM 340 and the kernel 350, which produces the grid OFM 330. In some embodiments, the kernel 350 is a dynamic kernel that includes weights determined based on the grid IFM 320, such as Figure 3 . The dynamic kernel can be generated by a convolution kernel and an element-wise multiplication of the attention generated based on EFM 340 or grid IFM 320. In other embodiments, kernel 350 is a convolution kernel. Point grid convolution 300 can be performed by a PGConv layer (e.g., PGConv layer 200). The PGConv layer can be a layer (e.g., the first layer) in a point grid network.

[0085] like Figure 3 As shown, the spatial size of the grid OFM 330 (ie, Figure 3 5×5) is the same as the grid IFM 320, but smaller than the grid EFM 340. The grid OFM 330 may also be grid structured data. The grid IFM 320 and the grid OFM 330 may have different numbers of channels. In other embodiments, the grid IFM 320 and the grid OFM 330 have different spatial sizes. The grid OFM 330 may be further processed, for example, by an activation function, another convolution, or other functions.

[0086] Example semantic mesh transformation

[0087] Figures 4A-4C An example semantic mesh transformation is illustrated according to various embodiments. The semantic mesh transformation may be performed by Figure 8 The transformation module 820 in is executed. Figure 4A Graph nodes identified from a graph representation of a person are shown, e.g. Figure 3The graph in 310. Each graph node is Figure 4A are shown as solid circles and represent different parts of the human body. Figure 4A 18 graph nodes are shown, representing the head, nose, chin, neck, right shoulder ("R.Shoulder" or "R.Shou."), left shoulder ("L.Shoulder" or "L.Shou."), right elbow ("R.Elbow"), torso, left elbow ("L.Elbow"), right wrist ("R.Wrist"), right hip ("R.Hip"), pelvis, left hip ("L.Hip"), left wrist ("L.Wrist"), right knee ("R.Knee"), left knee ("L.Knee"), right ankle ("R.Ankle"), and left ankle ("L.Ankle"). In other embodiments, more, fewer, or different graph nodes are identified from the graph representation of a person.

[0088] The trunk graph node is used as an anchor node for semantic grid transformation, for example, based on determining that the trunk graph node is connected to most other graph nodes and / or the trunk graph node is located at or near the center of the graph representation of the person. As described above, the spatial relationship between the trunk graph node and each other graph node can be determined. The spatial relationship can be a vertical relationship (e.g., an ancestor-descendant relationship), a horizontal relationship (e.g., a peer relationship), or a combination of the two. In some embodiments, the spatial relationship can be one or more distances from the trunk graph node to other graph nodes, such as a linear distance from the trunk graph node to other graph nodes, a horizontal distance along the X-axis (e.g., a distance from the projection of the trunk graph node on the X-axis to the projection of other graph nodes on the X-axis), a distance along the Y-axis (e.g., a distance from the projection of the trunk graph node on the Y-axis to the projection of other graph nodes on the Y-axis), and the like.

[0089] In addition, a hierarchy is determined based on the relative positions of other graph nodes relative to the trunk graph node. The hierarchy includes a series of levels, each level including one or more graph nodes. Figure 4A , the levels are represented by dashed shapes (circles and ovals). The first level includes the torso graph node, and is represented by the smallest dashed shape surrounding the torso graph node. The second level includes the neck graph node and the pelvis graph node, and is represented by the second smallest dashed shape. Similarly, the third level includes the right hip, right shoulder, chin, left shoulder, and left hip graph nodes; the fourth level includes the right knee, right elbow, nose, left elbow, and left knee graph nodes, and the fifth level includes the right ankle, right wrist, head, left wrist, and left ankle graph nodes.

[0090] exist Figure 4B In FIG. 4 , a pyramid structure 410 is formed based on the hierarchy. The pyramid includes five layers, each layer corresponds to a level of the hierarchy, and the graph nodes in the level are arranged in the corresponding layer of the pyramid. Figure 4B As shown, the trunk graph nodes are in the first / top level, the second level graph nodes are in the second level, the third level graph nodes are in the third level, and so on. Figure 4B The arrangement of the graph nodes in the pyramid structure 410 reflects Figure 4A The relationships between the graph nodes are shown in . Although Figure 4B A pyramid structure 410 is shown, but in other embodiments, the graph nodes may be arranged in a different structure, such as a cylinder, etc.

[0091] exist Figure 4C , the graph nodes arranged in a pyramid structure 410 are "flattened" and a 2D grid representation 420 of a person is generated from the pyramid structure 410. The grid representation 420 has a grid structure including five rows and five columns, i.e., a total of 25 elements. The number of rows is equal to the number of levels in the hierarchy. The number of columns is equal to the number of graph nodes in the level with the most graph nodes (i.e., the fifth level). Each graph node is in one of the 25 elements. Graph nodes in the same level are in the same row. The torso graph node is in the middle element of the first row. The grid representation 420 may be Figure 3 4. An embodiment of a grid IFM 320 in FIG. 4. The grid representation 420 includes empty grid elements, for example, in the first row and the second row. The empty grid elements can be filled with graph nodes in the same row. For example, some or all of the four empty elements in the first row can be filled with trunk graph nodes.

[0092] Figures 4A-4C The semantic mesh transformation shown in can be defined as

[0093]

[0094] in, represents the transformation function used to map node indices, G∈R J×C Denote an input graph sample (i.e., graph representation) with J nodes and each node has C feature channels, and D∈R H×P×C The output represents a woven mesh (e.g., mesh representation 420) of spatial size H×P. The woven mesh may have a rectangular shape or a square shape, depending on the value of Hand P. Although the semantic mesh transformation includes Figure 4B However, in other embodiments, the mesh representation 420 may be generated directly from the person graph representation and may be skipped. Figure 4B Follow the steps shown in .

[0095] Example Fill

[0096] Figures 5A-5D Various filling methods are illustrated according to various embodiments. Figures 5A-5DThe filling shown in can be performed on a grid representation converted from a graph representation by a semantic grid transformation. Given a graph-structured data representation, a data sample can be represented as a graph G = {V, E}, where nodes V store node features and edges define node connections. The degree of a graph node depends on the connected edges E, which forms irregular neighborhoods of local areas. In some cases, the original graph representation may have a limited number of nodes, for example, a 2D human pose with N = 17 body joints. This may result in a grid representation that will have a smaller spatial size, such as 5×5, or 7×5. Thus, it may be important to perform filling on grid-structured data samples.

[0097] Figures 5A-5D The filling shown in can be performed on the grid representation 420 in Figure 4. For illustration, Figures 5A-5D Four EFMs 510, 520, 530, and 540 generated based on four different filling methods are shown. In other embodiments, other filling methods may be used, or other shapes of EFMs may be formed.

[0098] Figure 5A EFM 510 is shown transformed from grid representation 420 by zero padding. Zero padding places extra node elements with zero values ​​to some or all of the four edges of grid representation 420. Similar to grid representation 420, EFM 510 also has a two-dimensional grid pattern, such as a flat pattern. The spatial scale of EFM 510 is larger than the spatial scale of grid representation 420. EFM 510 may be Figure 3 An embodiment of EFM 340 in FIG.

[0099] Figure 5B An EFM 520 transformed from the mesh representation 420 by circle filling is shown. Figure 5B The circular fill in places additional node elements on the four edges of the grid representation 420. Figure 5A The zero padding in is different, Figure 5B The values ​​of the additional node elements in are determined based on the values ​​of the node elements in the grid representation 420. For example, the additional node elements at the top (or bottom) edge have the same value as the node elements at the bottom (or top) of the grid representation 420, and the additional node elements at the right (or left) edge have the same value as the node elements at the left (or right) of the grid representation 420. The circle fill may connect the top edge to the bottom edge, and the right edge to the left edge, producing Figure 5A The 3D torus mesh pattern shown in FIG. 520 has a torus shape.

[0100] Figure 5CAn EFM 530 is shown which is transformed from the grid representation 420 through a combination of circle filling and zero filling. Figure 5C The circular padding in places an extra node element at the right (or left) edge of the grid representation 420, and the extra node element at the right (or left) edge has the same value as the node element at the left (or right) edge of the grid representation 420. The circular padding can connect the right edge with the left edge. The zero padding places an extra node element with a zero value at the top or bottom edge, and does not connect the top edge and the bottom edge. Thus, the combination of the circular padding and the zero padding produces a hollow cylindrical grid pattern that looks like a bucket.

[0101] Figure 5D An EFM 540 is shown which is transformed from the grid representation 420 through a combination of circle filling and zero filling. Figure 5D The circular fill in places an extra node element at the top (or bottom) edge of the grid representation 420, and the extra node element has the same value as the node element at the bottom (or top) of the grid representation 420. The circular fill can connect the top edge with the bottom edge. The zero fill places an extra node element with a zero value at the right or left edge, and does not connect the right edge and the left edge. Thus, the combination of the circular fill and the zero fill produces a hollow cylindrical grid pattern that looks like a tube.

[0102] Each of the four padding designs can maintain the spatial size of the grid representation 420, for example, the OFM generated from the grid representation 420 can have the same spatial size as the grid representation 420. In addition, the padding design can keep the convolution operations between internal node elements and boundary node elements consistent. In addition, the different connection states on the horizontal and vertical boundaries make the convolution on one grid pattern complementary to the convolution on other grid patterns. With a variety of padding methods, powerful PGConv-driven architectures can be built to cope with different applications, such as the applications mentioned above. The padding methods can be used with other techniques, such as dynamic convolution. For example, dynamic kernels can be generated based on EFMs, which are formed from IFMs through various padding methods. This paves the technical path for designing powerful PGConv-based networks via novel multi-branch building blocks. The following is combined with Figure 7 More information is provided on the multi-branch design of the PGConv layer.

[0103] Example Dynamic Convolution

[0104] Figure 6 An example dynamic convolution 600 is illustrated according to various embodiments. The dynamic convolution 600 may be performed by Figure 2The dynamic convolution 600 is a convolution performed on the EFM 610 and the dynamic kernel 640. In other embodiments, the dynamic convolution 600 may be performed on the IFM and the dynamic kernel 640. The EFM 610 may be represented as D in ∈R H′×P′×C , where C is the number of channels in EFM 610 and H′×P′ is the spatial size of EFM 610, i.e., the number of node elements per channel is H′×P′, where H′ is the number of rows and P′ is the number of columns.

[0105] The dynamic kernel 640 is based on the attention 620 (α) and the convolution kernel 630 (W∈R K×K×C ). In some embodiments, the dynamic kernel 640 may be the result of an element-wise multiplication of the attention kernel α and the convolution kernel W. The dynamic kernel 640 may be represented as α⊙W. The attention 620 is generated based on the EFM 610 by using the attention function 625 (π(·)). Figure 6 In the embodiment of the invention, the function 625 outputs α∈R K×K×H×P The attention function 625 can determine α for each node element in the channel of EFM 610. s ∈R K×K×1×1 Since EFM 610 has H×P node elements in a single channel, the overall attention can be expressed as α∈R K×K×H×P The kernel has the dimensions of the convolution weights (K×K) and the dimensions of the EFM 610 (H′×P′).

[0106] Dynamic convolution generates OFM 650 (D out ∈R H×P ). The spatial size of OFM 650 may be the same as the spatial size of the OFM from which EFM 610 is generated by padding. For simplicity and for illustration, convolution kernel 630 includes one filter and OFM 650 includes one output channel. In other embodiments, convolution kernel 630 may include multiple filters and OFM 650 may include multiple output channels.

[0107] Example PGConv layer with multi-branch design

[0108] Figure 7 An example PGConv layer 700 including a plurality of branches 730 and 730 is illustrated according to various embodiments. The PGConv layer 700 may be Figure 2 An embodiment of the PGConv layer 200 in FIG. Figure 7 As shown, PGConv layer 700 receives IFM 710 and outputs OFM 770. An example of IFM 710 is Figure 3Grid IFM 320 or Figure 4C The grid in 420 is represented. For simplicity and for illustration, Figure 7 Two branches 730 and 740 are shown. In other embodiments, the PGConv layer 700 may include more branches. Figure 7 In an embodiment, each branch performs dynamic convolution. In addition to branches 730 and 730, PGConv layer 700 also includes a padding module 720, an attention module 750, and an accumulator 760. In other embodiments, alternative configurations, different or additional components may be included in PGConv layer 700. In addition, the functions attributed to the components of PGConv layer 700 may be performed by different components included in PGConv layer 700 or by different systems.

[0109] The fill module 720 applies fill on the IFM 710. One embodiment of the fill module 720 is Figure 2 In the filling module 220. Figure 7 In some embodiments, fill module 720 generates EFMs 735 and 745 based on IFM 710, for example, by using different fill methods. EFMs 735 and 745 may have different grid patterns. In some embodiments, EFMs 735 or 745 may have Figures 5A-5D Fill module 720 provides EFM 735 to branch 730 and provides EFM 745 to branch 740. Fill module 720 also provides EFMs 735 and 745 to focus module 750.

[0110] Attention module 750 receives EFMs 735 and 745 and generates attention 753 and 754 based on EFMs 735 and 745, respectively. Each attention includes one or more attention kernels, including attention weights determined based on the corresponding EFMs. Attention 753 and 754 may be different, for example, due to differences between EFMs 735 and 745. Attention module 750 provides attention 753 to branch 730 and provides attention 754 to branch 740. Attention module 750 may be Figure 2 An embodiment of the attention module 230 in.

[0111] Branch 730 includes a convolution operator 733. Convolution operator 733 may perform dynamic convolution based on EFM 735, attention 753, and a convolution kernel determined by training of PGConv layer 700. The result of the convolution is branch OFM (BOFM) 737. Similarly, branch 740 includes a convolution operator 743. Convolution operator 743 may perform dynamic convolution based on EFM 745, attention 754, and a convolution kernel determined by training of PGConv layer 700. The result of the convolution is BOFM 747. BOFM 737 may be different from BOFM 747 due to differences between EFM 735 and 745, between attention 753 and 754, between the two convolution kernels, or some combination of these. In some embodiments, the convolution kernels in the two branches 730 and 740 may be the same.

[0112] BOFMs 737 and 747 are provided to accumulator 760. Accumulator 760 aggregates BOFMs 737 and 747 and generates OFM 770. In some embodiments, accumulator 760 sums BOFMs 737 and 747. For example, accumulator 760 may perform element-wise summation on BOFMs 737 and 747. Element-wise summation may include element-wise summation of each channel. An example of element-wise summation may be the summation of an element in BOFM 737 with a corresponding element in BOFM 747. The spatial coordinates of the two elements match. For example, both elements may be in the first row and first column of the first channel of the corresponding BOFM. The generated OFM 770 may have the same spatial size and channel dimension (i.e., the number of channels) as BOFMs 737 and 747. The spatial size of OFM 770 may be the same as IFM 710, but the channel dimension may be different from IFM 710. The channel dimension of OFM 770 depends on the number of filters in the dynamic kernel.

[0113] Example Point Grid System

[0114] Figure 88 is a block diagram of a point grid system 800 according to various embodiments. The point grid system 800 assists in implementing convolution of grid structure data generated from a semantic grid transformation of graph structure data. The point grid system 800 includes an interface module 810, a transformation module 820, a training module 830, a verification module 840, a point grid model 850, and an inverse transformation module 860. In other embodiments, alternative configurations, different or additional components may be included in the point grid system 800. For example, the point grid system 800 may include more than one point grid model. In addition, the functions attributed to the components of the point grid system 800 may be performed by different components included in the point grid system 800 or by different systems. For example, some or all of the functions attributed to the transformation module 820 or the inverse transformation module 860 may be performed by the point grid model 850.

[0115] The interface module 810 assists in implementing the communication between the point grid system 800 and other systems. In some embodiments, the interface module 810 establishes communication between the point grid system 800 and an external database to receive graph structure data that can be used to generate grid structure data, which is used to train the point grid model 850 or for reasoning of the point grid model 850. The external database can be an image library that stores multiple images, such as 8D images, 3D images, and the like. The interface module 810 can support the point grid system 800 to distribute the point grid model 850 to other systems, such as a computing device configured to apply the point grid model 850 to perform a task. The computing device can be an edge device, a client device, and the like. The interface module 810 can also support the point grid system 800 to distribute the output of the inverse transformation module 860 to other systems.

[0116] The transformation module 820 performs a semantic grid transformation on the graph structure data sample. The transformation module 820 may receive the graph structure data sample from the interface module 810. The graph structure data sample may be a graphical representation of one or more objects, such as an image of one or more objects. The objects may be people, animals, plants, trees, buildings, vehicles, streets, or other types of objects. The graphical representation may have an irregular structure. For example, a graphical representation of a person has an outline of the person's body, which may be different from a graphical representation of another person, or even different from another graphical representation of the same person with a different posture. The transformation module 820 may transform a graphical representation with an irregular structure into a grid representation with a regular structure.

[0117] In some embodiments, the transformation module 820 transforms the graphical representation of the object into a grid representation of the object based on a grid pattern. The grid pattern has a regular structure that can be adopted by the grid representation through a semantic grid transformation. The structure of the grid pattern can be fixed and can be 2D or 3D. The example grid pattern includes a number of elements, each element defined by a fixed boundary. These elements can be arranged in rows and / or columns. For example, a row or a column can have one or more elements. The transformation module 820 can obtain (e.g., generate or retrieve from a database) grids with different structures. The transformation module 820 can select a grid based on a graph structure data sample, for example, based on the category of the object illustrated in the graph structure data sample. For example, compared to the graphical representation of a tree, the transformation module 820 can use a different grid to transform the graphical representation of a person.

[0118] In a semantic mesh transformation of a graphical representation of an object, the transformation module 820 may identify graph nodes in the graphical representation. A graph node is a graphical representation of one or more components of an object. In an example where the object is a person, the transformation module 820 may identify graph nodes that represent different parts of the person's body, such as the head, neck, torso, arms, legs, and so on. The transformation module 820 may identify graph nodes based on the body joints illustrated in the graphical representation of the person. The transformation module 820 may assign graph nodes to different elements of a grid to form a grid representation of the object. The grid representation employs both information from the graphical representation (e.g., relationships between graph nodes) and the regular structure of the grid pattern.

[0119] In some embodiments, the transformation module 820 identifies the anchor node from the graph node and first assigns the anchor node. The transformation module 820 can randomly select the anchor node. Alternatively, the transformation module 820 can select the anchor node based on a rule. In one embodiment, the transformation module 820 can select a graph node connected to some or all other graph nodes as an anchor node. For example, the transformation module 820 can select a graph node representing a person's torso as an anchor node for a graphical representation of a person, or select a graph node representing the central trunk of a tree as an anchor node for a graphical representation of a tree. In another embodiment, the anchor node can be a root node. After determining the anchor node, the transformation module 820 assigns the anchor node to an element of a grid. The element can be predetermined. In one example, the transformation module 820 assigns the anchor node to a specific row of a grid (or a specific element in a specific row).

[0120] The transformation module 820 further assigns other graph nodes ("secondary nodes") based on the assignment of the anchor node. The transformation module 820 can determine the relationship between the secondary node and the anchor node, and assign the secondary node based on the relationship. The relationship can be a spatial relationship, such as the distance between the component represented by the secondary node and the component represented by the anchor node. The transformation module 820 can measure the distance based on the graphical representation of the object. The spatial relationship can be multi-dimensional. For example, the transformation module 820 can determine a vertical node relationship (the relationship can indicate the distance, orientation, or both between the two components along a first direction) and a horizontal node relationship (the relationship can indicate the distance, orientation, or both between the two components along a second direction perpendicular to or substantially perpendicular to the first direction).

[0121] The transformation module 820 assigns the secondary node based on its relationship with the anchor node. For example, the transformation module 820 selects an element of the grid for the secondary node based on the relationship and the predetermined element to which the anchor node is assigned. The spatial relationship between the two elements may match the spatial relationship between the secondary node and the anchor node. In an example where the secondary node is located below the anchor node in the graphical representation, the transformation module 820 assigns the secondary node to the element below the predetermined element in the grid. In another example where the secondary node is located to the left of the anchor node in the graphical representation, the transformation module 820 assigns the secondary node to the element to the left of the predetermined element in the grid.

[0122] In some embodiments, the transformation module 820 may determine a hierarchy of graph nodes, where an anchor node is a first level, one or more secondary nodes closest to the anchor node are a second level, one or more secondary nodes second closest to the anchor node are a third level, and so on and so forth. In the example of assigning an anchor node to a first row of a grid, secondary nodes in the second level are assigned to the second row, secondary nodes in the third level are assigned to the third row, and so on and so forth. The transformation module 820 may determine the hierarchy based on spatial relationships in one dimension. The transformation module 820 may further determine additional spatial relationships between secondary nodes in the same level in different dimensions, and assign secondary nodes to different elements in the same row based on the additional spatial relationships.

[0123] The training module 830 trains the point grid model 850, which performs machine learning tasks using grid structure data samples. In the process of training the point grid model 850, the training module 830 can form a training data set. The training data set includes training samples and ground truth labels. The training samples can be grid structure data samples provided by the transformation module 820. Each training sample can be associated with one or more ground truth labels. The ground truth labels of the training samples can be known or verified labels that answer questions or queries that the point grid model 850 will use to answer. In the example where the point grid model 850 is used to estimate the posture, the ground truth label can indicate the ground truth posture of the object in the training sample. The ground truth label can be a numerical value indicating the posture of the object or the possibility that the object has a certain posture. In some embodiments, the training module 830 can also form a verification data set for verifying the performance of the trained DNN by the verification module 840. The verification data set may include verification samples and ground truth labels of the verification samples. The verification data set may include different samples from the training data set used to train the point grid model 850. In one embodiment, a portion of the training data set may be used to initially train the point grid model 850, while the remainder of the training data set may be reserved as a validation subset for the validation module 840 to validate the performance of the point grid model 850. The portion of the training data set that does not include the validation subset may be used to train the point grid model 850.

[0124] The training module 830 also determines the hyperparameters for training the point grid model 850. Hyperparameters are variables that specify the training process. Hyperparameters are different from parameters inside the point grid model 850 ("internal parameters", such as the weights in the convolution kernel of the PGConv layer). In some embodiments, the hyperparameters include variables that determine the architecture of the point grid model 850, such as the number of hidden layers, etc. Hyperparameters also include variables that determine how the point grid model 850 is trained, such as batch size, number of rounds, etc. The batch size defines the number of training samples to be worked through before updating the parameters of the point grid model 850. The batch size is the same as or smaller than the number of samples in the training data set. The training data set can be divided into one or more batches. The number of rounds defines how many times the entire training data set is passed forward and backward throughout the network. The number of rounds defines the number of times the DL algorithm works through the entire training data set. One round means that each training sample in the training data set has a chance to update the internal parameters of the point grid model 850. A round may include one or more batches. The number of rounds can be 15, 150, 500, 1500, or even larger.

[0125] The training module 830 defines the architecture of the point grid model 850, for example, based on some hyperparameters. The architecture of the point grid model 850 includes an input layer, an output layer, and multiple hidden layers. The input layer of the point grid model 850 may include a tensor (e.g., a multidimensional array) specifying the attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of the pixels in the input image). The output layer includes the labels of the objects in the input layer. The hidden layer is a layer between the input layer and the output layer. The hidden layer includes one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, normalization layers, softmax or logical layers, etc. The convolutional layer of the point grid model 850 converts the input image into a feature map, which is represented by a tensor specifying the feature map height, feature map width, and feature map channels (e.g., red, green, and blue images include three channels). The pooling layer is used to reduce the spatial volume of the input image after convolution. It is used between two convolutional layers. A fully connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories through training.

[0126] In the process of defining the architecture of the point grid model 850, the training module 830 also adds an activation function to the hidden layer or the output layer. The activation function of the layer transforms the weighted sum of the input of the layer into the output of the layer. The activation function can be, for example, a ReLU activation function, a tangent activation function, or an activation function of other types. After the training module 830 defines the architecture of the point grid model 850, the training module 830 inputs the training data set into the point grid model 850. The training module 830 modifies the internal parameters of the point grid model 850 so that the error between the label of the training sample generated by the point grid model 850 and the ground truth label is minimized. In some embodiments, the training module 830 uses a cost function or a loss function to minimize the error.

[0127] The training module 830 may train the point grid model 850 for a predetermined number of rounds. The number of rounds is a hyperparameter that defines the number of times the DL algorithm will work through the entire training data set. One round means that each sample in the training data set has a chance to update the internal parameters of the point grid model 850. After the training module 830 completes the predetermined number of rounds, the training module 830 may stop updating the internal parameters of the point grid model 850, and the point grid model 850 is considered to have been trained.

[0128] After the point grid model 850 is trained, the verification module 840 verifies the accuracy of the point grid model 850. In some embodiments, the verification module 840 inputs samples in the verification data set into the point grid model 850 and uses the output of the point grid model 850 to determine the model accuracy. In some embodiments, the verification data set may be formed by some or all samples in the training data set. Additionally or alternatively, the verification data set includes additional samples in addition to the samples in the training set. In some embodiments, the verification module 840 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The verification module 840 may determine the accuracy score using the following metrics: precision = TP / (TP+FP) and recall = TP / (TP+FN), where precision may be the number of correctly predicted (TP or true positives) by the reference classification model in its total number of predictions (TP+FP or false positives), and recall may be the number of correctly predicted (TP) by the reference classification model in the total number of objects that do have the relevant attribute (TP+FN or false negatives). F-score (F-score = 8*PR / (P+R)) unifies precision and recall into a single metric.

[0129] The verification module 840 may compare the accuracy score to a threshold score. In examples where the verification module 840 determines that the accuracy score is below the threshold score, the verification module 840 instructs the training module 830 to retrain the point grid model 850. In one embodiment, the training module 830 may iteratively retrain the point grid model 850 until a stopping condition occurs, such as an accuracy measurement indicating that the point grid model 850 may be sufficiently accurate, or a number of training rounds have occurred.

[0130] The point grid model 850 performs machine learning tasks using grid structured data. A machine learning task is a task of performing reasoning. Reasoning is the process of running available data (e.g., grid structured data) into the point grid model 850 to generate an output. The output provides a solution to the problem or question posed. The point grid model 850 can perform machine learning tasks for a variety of applications, including applications that traditionally rely on graph structured data, such as 2D to 3D human posture lifting, skeleton-based human action recognition, 3D grid reconstruction, traffic navigation, social network analysis, recommendation systems, and scientific computing. In some embodiments, the point grid model 850 is a convolutional network that includes multiple hidden layers, for example, one or more convolutional layers. The point grid model 850 may include one or more PGConv layers, such as PGConv layers 200 or 700. An embodiment of the point grid model 850 may be the above in combination with Figure 1 CNN 100 described.

[0131] In some embodiments, the point grid model 850 receives a grid structured data sample from the transformation module 820 and processes the grid structured data sample to make a decision. The convolution layer of the point grid model 850 may extract features from the grid structured data sample or from the output of another layer of the point grid model 850. In one embodiment, the convolution layer may generate a variant of the grid structured data sample and extract features based on the variant. The variant of the grid structured data sample may include some or all of the graph nodes in the grid structured data sample, but has a different structure from the grid structured data sample or other variants. The output of the point grid model 850 may be grid structured data, such as a feature map of a grid structure. This will be discussed below in conjunction with Figure 3 and Figure 5- Figure 7 More details are provided regarding the point grid model 850 .

[0132] Inverse transformation module 860 can transform the grid structure output of point grid model 850 into graph structure data. In one example, point grid model 850 outputs a grid representation of the estimated posture of the object, and sends the output to inverse transformation module 860. Inverse transformation module 860 converts the grid representation into a graphical representation of the estimated posture, for example, by the inverse transformation of the semantic grid transformation. This transformation is called semantic graph transformation. In some applications where another system or user needs graph structure data instead of grid structure data, semantic graph transformation may be necessary. The graphical representation generated by inverse transformation module 860 can be a 3D graph showing the estimated posture. In some embodiments, inverse transformation module 860 can generate a 3D image or animation showing the estimated posture.

[0133] Example 3D pose estimation

[0134] Fig. 9 According to various embodiments, an example 3D pose estimation based on a 2D input 910 is illustrated. The 2D input 910 is a graphical representation of a person, which may show the 2D pose of the person. Fig. 9 As shown, the 2D input 910 is converted to a mesh IFM 920, for example, by a semantic mesh transformation, The semantic mesh transformation can be done by Figure 8 The transformation module 820 in 9D is completed. In some embodiments, the mesh IFM 920 is the result of applying padding to the mesh representation of the 2D input 910.

[0135] like Fig. 9 As shown, the mesh IFM 920 is input to the mesh boosting network 930. The mesh boosting network 930 may include one or more PGConv layers, such as the PGConv layer 200 or 700. The mesh boosting network 930 may be Figure 8. The grid lifting network 930 has been trained to process grid-structured data samples, for example, by point grid convolution, to estimate 3D pose. The grid lifting network 930 may be trained with a training data set that includes grid-structured data samples and ground truth grid poses of the samples. In some embodiments, the predicted 3D grid pose D may be estimated by minimizing the predicted 3D grid pose D on the training data set. 3D With ground truth The error L between them is used to train the grid lifting network 930:

[0136]

[0137] Where L represents the 2-norm distance.

[0138] After training, the mesh lifting network 930 can estimate the 3D pose by inference. Inference is the process of processing an input (e.g., mesh IFM 920) and generating an output (e.g., mesh OFM 940) that indicates a 3D pose. In some embodiments, the spatial size of the mesh IFM 920 is the same as the spatial size of the mesh OFM 940, but the number of channels may be different. For example, the mesh IFM 920 may have two channels representing 2D joint positions, while the mesh OFM 940 may have three channels representing 3D joint positions. In other embodiments (e.g., embodiments that use dense 2D pose input to estimate sparse 3D poses, and estimate one or more 3D joint positions from all 2D joint positions), the spatial size of the mesh IFM 920 and the spatial size of the mesh OFM 940 may be different.

[0139] The grid OFM 940 is then transformed by inverse is converted to a 3D output 950. The inverse transform can be done by Figure 2 The 3D output 950 is a 3D graph representation of the estimated pose. In some embodiments, if a specific The inverse transformation that converts D back to G can then be easily obtained by mapping the grid back to J unique nodes For repetitive meshes, feature merging is applied in the inverse transformation.

[0140] like Fig. 9 As shown, the lifting process includes semantic mesh transformation Inference and inverse transformation of the grid lifting network 930 The lifting process can be expressed as:

[0141]

[0142] Among them, G 3Drepresents the 3D output 950, F represents the reasoning of the mesh lifting network 930, and G 2D Represents 2D input 910 .

[0143] Example DL Environment

[0144] Fig.10 The DL environment 1000 is illustrated according to various embodiments. The DL environment 1000 includes a DL server 1010 and a plurality of client devices 1020 (individually referred to as client devices 1020). The DL server 1010 is connected to the client devices 1020 via a network 1030. In other embodiments, the DL environment 1000 may include fewer, more, or different components.

[0145] DL server 1010 uses neural network to train DL model. The structure of neural network is like human brain, and is composed of artificial neurons, also known as nodes. These nodes are stacked on each other in three types of layers: input layer, (one or more) hidden layer, and output layer. Data provides information to each node in the form of input. The node multiplies the input with random weights, performs calculations, and adds biases. Finally, a nonlinear function, also known as an activation function, is applied to determine which neuron to start. DL server 1010 can use various types of neural networks, such as DNN, recurrent neural network (RNN), generative adversarial network (GAN), long short-term memory network (LSTMN), etc. During the process of training DL model, neural network uses unknown elements in input distribution to extract features, group objects, and find useful data patterns. DL model can be used to solve various problems, such as making predictions, classifying images, etc. DL server 1010 can build DL model specifically for specific types of problems that need to be solved. DL model is trained to receive input and output solutions to specific problems.

[0146] exist Fig.10 In the example, DL server 1010 includes DNN system 1040, database 1050, and distributor 1060. DNN system 1040 trains DNN. DNN can be used to process images, such as images captured by autonomous vehicles, medical equipment, satellites, etc. In one embodiment, DNN receives input images and outputs classifications of objects in the input images. Examples of DNN are described above in conjunction with Figure 1 CNN 100 described Fig. 9 The grid in the network 930, or the above combined Figure 8The point grid model 850 described above. One embodiment of the DNN system 1040 is a combination of the above Fig. 9 A point grid system 800 is described.

[0147] The database 1050 stores data received, used, generated, or otherwise associated with the DL server 1010. For example, the database 1050 stores a training data set used by the DNN system 1040 to train the DNN. In one embodiment, the training data set is an image library that can be used to train the DNN to classify images, estimate poses, and the like. The training data set may include data received from the client device 1020. As another example, the database 1050 stores hyperparameters of a neural network constructed by the DL server 1010.

[0148] The distributor 1060 distributes the DL model generated by the DL server 1010 to the client device 1020. In some embodiments, the distributor 1060 receives a request for a DNN from the client device 1020 via the network 1030. The request may include a description of a problem that the client device 1020 needs to solve. The request may also include information about the client device 1020, such as information describing the available computing resources on the client device. The information describing the available computing resources on the client device 1020 may be information indicating the network bandwidth, information indicating the available memory size, information indicating the processing power of the client device 1020, and the like. In one embodiment, the distributor may instruct the DNN system 1040 to generate a DNN according to the request. The DNN system 1040 may generate a DNN based on the information in the request. For example, the DNN system 1040 may determine the structure of the DNN and / or train the DNN according to the request.

[0149] In another embodiment, the distributor 1060 may select a DNN from a set of pre-existing DNNs based on the request. The distributor 1060 may select a DNN for a particular client device 1020 based on the size of the DNN and the available resources of the client device 1020. In an embodiment where the distributor 1060 determines that the memory or processing power of the client device 1020 is limited, the distributor 1060 may select a compressed DNN for the client device 1020 instead of an uncompressed DNN with a larger size. The distributor 1060 then transmits the DNN generated or selected for the client device 1020 to the client device 1020.

[0150] In some embodiments, the distributor 1060 may receive feedback from the client device 1020. For example, the distributor 1060 receives new training data from the client device 1020 and may send the new training data to the DNN system 1040 to further train the DNN. As another example, the feedback includes an update of available computer resources on the client device 1020. The distributor 1060 may send a different DNN to the client device 1020 based on the update. For example, after receiving feedback indicating that the computing resources of the client device 1020 have been reduced, the distributor 1060 sends a smaller DNN to the client device 1020.

[0151] The client device 1020 receives the DNN from the distributor 1060 and applies the DNN to perform machine learning tasks, for example, to solve a problem or answer a question. In various embodiments, the client device 1020 inputs an image into the DNN and uses the output of the DNN for various applications, for example, visual reconstruction, augmented reality, robot positioning and navigation, medical diagnosis, weather prediction, and the like. The client device 1020 may be one or more computing devices that are capable of receiving user input and sending and / or receiving data via the network 1030. In one embodiment, the client device 1020 is a conventional computer system, such as a desktop or laptop computer. Alternatively, the client device 1020 may be a device with computer functionality, such as a personal digital assistant (PDA), a mobile phone, a smart phone, an autonomous vehicle, or other appropriate device. The client device 1020 is configured to communicate via the network 1030. In one embodiment, the client device 1020 executes an application that allows a user of the client device 1020 to interact with the DL server 1010 (e.g., the distributor 1060 of the DL server 1010). The client device 1020 may request the DNN through the application or send feedback to the distributor 1060. For example, the client device 1020 executes a browser application to enable interaction between the client device 1020 and the DL server 1010 via the network 1030. In another embodiment, the client device 1020 executes a browser application in a native operating system (e.g., or ANDROID TM ) to interact with the DL server 1010.

[0152] In one embodiment, client device 1020 is an integrated computing device that operates as an independent device with network capabilities. For example, client device 1020 includes a display, a speaker, a microphone, a camera, and an input device. In another embodiment, client device 1020 is a computing device for coupling to an external media device, such as a television or other external display and / or an audio output system. In this embodiment, client device 1020 can be coupled to an external media device via a wireless interface or a wired interface (e.g., an HDMI cable), and various functions of the external media device can be utilized, such as its display, speaker, microphone, camera, and input device. Here, client device 1020 can be configured to be compatible with a general external media device that does not have specialized software, firmware, or hardware specifically for interacting with client device 1020.

[0153] The network 1030 supports communication between the DL server 1010 and the client device 1020. The network 1030 may include any combination of a local area network and / or a wide area network, using wired and / or wireless communication systems. In one embodiment, the network 1030 may use standard communication technologies and / or protocols. For example, the network 1030 may include communication links using technologies such as Ethernet, 10010.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, code division multiple access (CDMA), digital subscriber line (DSL), etc. Examples of networking protocols for communicating via the network 1030 may include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged through network 1030 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). In some embodiments, all or some of the communication links of network 1030 may be encrypted using any one or more suitable techniques.

[0154] Example method for point grid convolution with padding assistance

[0155] Fig.11 is a flow chart showing a method 1100 for implementing point grid convolution with padding assistance according to various embodiments. The method 1100 may be Figure 2 PGConv layer 200 in . Although the reference Fig.11 The flowchart shown in describes method 1100, but many other methods for point grid convolution with padding assistance may be used instead. For example, Fig.11 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.

[0156] The PGConv layer 200 receives (1110) an IFM. The IFM includes a grid representation of an object. The grid representation includes a plurality of node elements arranged in a grid pattern. The grid pattern may have a 2D grid pattern including a plurality of edges.

[0157] The PGConv layer 200 generates (1120) an EFM by adding one or more additional node elements to the grid representation. The EFM includes a plurality of node elements and one or more additional node elements. The plurality of node elements and the one or more additional node elements may be arranged in a grid pattern different from the grid pattern of the IFM. The grid pattern of the EFM may be 3D, while the grid pattern of the IFM may be 2D. The PGConv layer 200 may add one or more additional node elements on one or more edges of the grid pattern of the IFM. In some embodiments, one of the one or more additional node elements has a zero value. In other embodiments, one of the one or more additional node elements has the value of one of the plurality of node elements. In one embodiment, the grid pattern has a 2D grid pattern, including a first edge and a second edge opposite to the first edge. The node elements are arranged on the first edge, and the additional node elements may be added on the second edge.

[0158] The PGConv layer 200 generates (1130) an OFM based on the EFM and the convolution kernel. The convolution kernel includes convolution weights, whose values ​​are determined by training the DNN. In some embodiments, the EFM is a first EFM, and the PGConv layer 200 can generate a second EFM by adding one or more other node elements to the IFM. The PGConv layer 200 can perform other MAC operations based on the second EFM and the convolution kernel. For example, the PGConv layer 200 can perform a first convolution operation based on the first EFM and the convolution kernel, and a second convolution operation based on the second EFM and another convolution kernel. The PGConv layer 200 can further accumulate the result of the first convolution operation and the result of the second convolution operation.

[0159] Example method for dynamic point grid convolution

[0160] Fig.12 is a flow chart illustrating a method 1200 for dynamic point grid convolution according to various embodiments. The method 1200 may be Figure 2 PGConv layer 200 in . Although the reference Fig.12 The flowchart shown in describes method 1200, but many other methods for dynamic point grid convolution may be used instead. For example, Fig.12 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.

[0161] The PGConv layer 200 obtains (1210) an EFM generated from a mesh representation of the object. The mesh representation includes a plurality of node elements arranged in a grid pattern. The EFM can be generated by adding one or more additional node elements to the mesh representation. The EFM includes a plurality of node elements and one or more additional node elements.

[0162] The PGConv layer 200 generates a focus kernel. The focus kernel includes a focus weight, whose value is determined based on the EFM. In some embodiments, the focus kernel is a first focus kernel, and the PGConv layer 200 generates a second focus kernel and a third focus kernel. The three focus kernels may have different dimensions. For example, the PGConv layer 200 may determine the dimension of the first focus kernel based on the dimension of the EFM, determine the dimension of the second focus kernel based on the dimension of the convolution kernel, and determine the dimension of the third focus kernel based on the dimension of the OFM.

[0163] The PGConv layer 200 generates OFM based on the EFM, the attention kernel and the convolution kernel. The convolution kernel includes convolution weights, whose values ​​are determined by training the DNN.

[0164] In an embodiment where multiple kernels of interest are generated, the PGConv layer 200 may generate an OFM based on the multiple kernels of interest. The PGConv layer 200 may generate an OFM by performing element-wise multiplication on the first kernel of interest, the second kernel of interest, the third kernel of interest, and the convolution kernel. The PGConv layer 200 may further generate an OFM by performing a MAC operation on the EFM and the result of the element-wise multiplication.

[0165] In some embodiments, the EFM is a first EFM, and the attention kernel is a first attention kernel including a first attention weight, wherein the value of the first attention weight is determined based on the first EFM. The PGConv layer 200 may generate a second EFM based on the grid representation, and generate a second attention kernel including a second attention weight, wherein the value of the second attention weight is determined based on the second EFM. The PGConv layer 200 may further generate an OFM based on the second EFM and the second attention kernel. In some embodiments, the first EFM includes a plurality of node elements arranged in a first grid pattern, the second EFM includes a plurality of node elements arranged in a second grid pattern, and the first grid pattern has a shape different from that of the second grid pattern. In addition to the plurality of node elements, the first feature map also includes a first node element. In addition to the plurality of node elements, the second EFM also includes a second node element, and the first node element has a value different from that of the second node element.

[0166] The PGConv layer 200 can generate an OFM by accumulating the results of a first convolution operation and a second convolution operation. The first convolution operation is based on a first EFM, a first focus kernel, and a convolution kernel. The second convolution operation is based on a second EFM, a second focus kernel, and a convolution kernel.

[0167] Example computing device

[0168] Fig.13 is a block diagram of an example computing device 1300 in accordance with various embodiments. Fig.13 1300, but any one or more of these components may be omitted or repeated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1300 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. In addition, in various embodiments, computing device 1300 may not include a processor or a processor. Fig.13 , but computing device 1300 may include interface circuitry for coupling to the one or more components. For example, computing device 1300 may not include display device 1306, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 1306 may be coupled. In another set of examples, computing device 1300 may not include audio input device 1318 or audio output device 1308, but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which audio input device 1318 or audio output device 1308 may be coupled.

[0169] The computing device 1300 may include a processing device 1302 (e.g., one or more processing devices). As used herein, the term "processing device" or "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. The processing device 1302 may include one or more digital signal processors (DSPs), application-specific ICs (ASICs), CPUs, GPUs, cryptographic processors (specialized processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices. The computing device 1300 may include a memory 1304, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard disk. In some embodiments, the memory 1304 may include a memory that shares a die with the processing device 1302. In some embodiments, the memory 1304 includes one or more non-transitory computer-readable media storing instructions that are executable to perform operations for point grid convolution, such as the above-mentioned Fig.11 The method 1100 described above is combined with Fig.12 The method 1200 described above or in combination with Figure 2 The operations performed by the PGConv layer 200 are described. The instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 1302.

[0170] In some embodiments, computing device 1300 may include communication chip 1312 (e.g., one or more communication chips). For example, communication chip 1312 may be configured to manage wireless communications for transmitting data to and from computing device 1300. The term "wireless" and its derivatives may be used to describe circuits, devices, DNN accelerators, methods, techniques, communication channels, and the like that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated devices do not contain any wires, although in some embodiments they may not.

[0171] The communication chip 1312 can implement any of several wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards, including Wi-Fi (IEEE 1302.13 series), IEEE 1302.16 standards (e.g., IEEE 1302.16-2005 amendment), Long-Term Evolution (LTE) project and any amendments, updates and / or revisions (e.g., Advanced LTE project, ultramobile broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE1302.16 are generally referred to as WiMAX networks. This abbreviation stands for Worldwide Interoperability for Microwave Access, which is a certification mark for products that have passed the compliance and interoperability tests of the IEEE 1302.16 standard. The communication chip 1312 can operate according to the Global system for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications system (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA) or LTE network. The communication chip 1312 can operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN) or Evolved UTRAN (E-UTRAN).The communication chip 1312 may operate according to CDMA, Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, and any other wireless protocols designated as 3G, 4G, 5G and above. In other embodiments, the communication chip 1312 may operate according to other wireless protocols. The computing device 1300 may include an antenna 1322 to facilitate wireless communication and / or receive other wireless communications (e.g., AM or FM radio transmissions).

[0172] In some embodiments, the communication chip 1312 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1312 may include multiple communication chips. For example, the first communication chip 1312 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, while the second communication chip 1312 may be dedicated to longer-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1312 may be dedicated to wireless communications, while the second communication chip 1312 may be dedicated to wired communications.

[0173] Computing device 1300 may include battery / power circuitry 1314. Battery / power circuitry 1314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of computing device 1300 to an energy source separate from computing device 1300 (e.g., AC line power).

[0174] Computing device 1300 may include a display device 1306 (or corresponding interface circuitry, as described above). Display device 1306 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.

[0175] Computing device 1300 may include audio output device 1308 (or corresponding interface circuitry, as described above). Audio output device 1308 may include any device that generates audible indications, such as a speaker, headphones, or earbuds.

[0176] Computing device 1300 may include audio input device 1318 (or corresponding interface circuitry, as described above). Audio input device 1318 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output).

[0177] Computing device 1300 may include GPS device 1316 (or corresponding interface circuitry, as described above). GPS device 1316 may communicate with a satellite-based system and may receive the location of computing device 1300, as is known in the art.

[0178] Computing device 1300 may include other output devices 1313 (or corresponding interface circuitry, as described above). Examples of other output devices 1313 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.

[0179] The computing device 1300 may include other input devices 1320 (or corresponding interface circuits, as described above). Examples of other input devices 1320 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0180] The computing device 1300 may have any desired form factor, such as a handheld or mobile computing system (e.g., a cellular phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultra-mobile personal computer, etc.), a desktop computing system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computing system. In some embodiments, the computing device 1300 may be any other electronic device that processes data.

[0181] Selected Examples

[0182] The following paragraphs provide various examples of the embodiments disclosed herein.

[0183] Example 1 provides a DNN, which includes a filling module configured to: receive an IFM including a grid representation of an object, the grid representation including a plurality of node elements arranged in a grid pattern, and generate an EFM by adding one or more additional node elements to the grid representation, the EFM including the plurality of node elements and the one or more additional node elements; and a convolution operator configured to generate an output feature map based on the EFM and a convolution kernel, the convolution kernel including convolution weights, the values ​​of the convolution weights being determined by training the DNN.

[0184] Example 2 provides the DNN of Example 1, wherein the grid pattern is a two-dimensional grid pattern including a plurality of edges, and the filling module is configured to generate the EFM by adding the one or more additional node elements on at least one of the plurality of edges.

[0185] Example 3 provides the DNN of Example 1, wherein an additional node element of the one or more additional node elements has a value of zero.

[0186] Example 4 provides the DNN of Example 1, wherein an additional node element of the one or more additional node elements has a value of a node element of the plurality of node elements.

[0187] Example 5 provides a DNN as described in Example 4, wherein the grid pattern is a two-dimensional grid pattern including a first edge and a second edge opposite to the first edge, the node elements are arranged on the first edge, and the filling module is configured to generate the EFM by adding the additional node elements on the second edge.

[0188] Example 6 provides the DNN of Example 1, wherein the EFM has a grid pattern different from the grid, and the plurality of node elements and the one or more additional node elements are arranged in the different grid pattern.

[0189] Example 7 provides the DNN of Example 6, wherein the grid pattern is a two-dimensional grid pattern and the EFM has a three-dimensional grid pattern.

[0190] Example 8 provides the DNN of Example 6, wherein the EFM has a grid pattern, and the grid pattern has a torus shape or a hollow cylindrical shape.

[0191] Example 9 provides a DNN as described in Example 1, wherein the EFM is a first EFM having a first grid pattern, the filling module is further configured to generate a second EFM by adding one or more other node elements to the grid representation, the second EFM having a second grid pattern different from the first grid pattern, and the convolution operator is further configured to perform other MAC operations based on the second EFM and the convolution kernel.

[0192] Example 10 provides the DNN as described in Example 9, wherein the convolution operator is configured to generate the OFM by performing a first convolution operation based on the first EFM and the convolution kernel; performing a second convolution operation based on the second EFM and another convolution kernel; and accumulating a result of the first convolution operation and a result of the second convolution operation.

[0193] Example 11 provides the DNN as described in Example 1, wherein the convolution weights are determined by training the DNN.

[0194] Example 12 provides the DNN of Example 11, wherein the convolution operator is configured to generate the output feature map further based on an attention kernel, and the attention kernel includes an attention weight determined based on the expanded feature map.

[0195] Example 13 provides the DNN of Example 12, wherein the convolution operator is configured to generate the output feature map by performing element-wise multiplication on the convolution kernel and the attention kernel.

[0196] Example 14 provides a method for performing DL by a DNN, the method comprising receiving an IFM comprising a grid representation of an object, the grid representation comprising a plurality of node elements arranged in a grid pattern; generating an EFM by adding one or more additional node elements to the grid representation, the EFM comprising the plurality of node elements and the one or more additional node elements; and generating an OFM based on the EFM and a convolution kernel, the convolution kernel comprising convolution weights, the values ​​of the convolution weights being determined by training the DNN.

[0197] Example 15 provides a method as described in Example 14, wherein the grid pattern is a two-dimensional grid pattern including a plurality of edges, and adding the one or more additional node elements to the grid representation includes adding the one or more additional node elements on at least one of the plurality of edges.

[0198] Example 16 provides the method of Example 14, wherein an additional node element of the one or more additional node elements has a value of zero.

[0199] Example 17 provides the method of Example 14, wherein an additional node element of the one or more additional node elements has a value of a node element of the plurality of node elements.

[0200] Example 18 provides a method as described in Example 17, wherein the grid pattern is a two-dimensional grid pattern including a first edge and a second edge opposite to the first edge, the node elements are arranged on the first edge, and adding the one or more additional node elements to the grid representation includes adding the additional node elements on the second edge.

[0201] Example 19 provides the method of Example 14, wherein the EFM has a different grid pattern than the grid, and the plurality of node elements and the one or more additional node elements are arranged in the different grid pattern.

[0202] Example 20 provides the method of Example 19, wherein the grid pattern is a two-dimensional grid pattern and the EFM has a three-dimensional grid pattern.

[0203] Example 21 provides the method of Example 19, wherein the EFM has a grid pattern, and the grid pattern has a torus shape or a hollow cylindrical shape.

[0204] Example 22 provides a method as described in Example 14, wherein the EFM is a first EFM having a first grid pattern, and the method further includes generating a second EFM by adding one or more other node elements to the grid representation, the second EFM having a second grid pattern different from the first grid pattern, and performing other MAC operations based on the second EFM and the convolution kernel.

[0205] Example 23 provides a method as described in Example 22, wherein generating the OFM includes performing a first convolution operation based on the first EFM and the convolution kernel; performing a second convolution operation based on the second EFM and another convolution kernel; and accumulating a result of the first convolution operation and a result of the second convolution operation.

[0206] Example 24 provides a method as described in Example 14, wherein the convolution weights are determined by training the DNN.

[0207] Example 25 provides a method as described in Example 14, wherein generating the output feature map includes further generating the output feature map based on an attention kernel, wherein the attention kernel includes an attention weight determined based on the expanded feature map.

[0208] Example 26 provides one or more non-transitory computer-readable media storing instructions that are executable to perform operations for DL, the operations comprising receiving an IFM comprising a mesh representation of an object, the mesh representation comprising a plurality of node elements arranged in a grid pattern; generating an EFM by adding one or more additional node elements to the mesh representation, the EFM comprising the plurality of node elements and the one or more additional node elements; and generating an OFM based on the EFM and a convolution kernel, the convolution kernel comprising convolution weights, the values ​​of the convolution weights being determined by training the DNN.

[0209] Example 27 provides one or more non-transitory computer-readable media as described in Example 26, wherein the grid pattern is a two-dimensional grid pattern including a plurality of edges, and adding the one or more additional node elements to the grid representation includes adding the one or more additional node elements on at least one of the plurality of edges.

[0210] Example 28 provides one or more non-transitory computer-readable media as described in Example 26, wherein an additional node element of the one or more additional node elements has a value of zero.

[0211] Example 29 provides one or more non-transitory computer-readable media as described in Example 26, wherein an additional node element of the one or more additional node elements has a value of a node element of the plurality of node elements.

[0212] Example 30 provides one or more non-transitory computer-readable media as described in Example 26, wherein the EFM is a first EFM having a first grid pattern, and the operation also includes generating a second EFM by adding one or more other node elements to the grid representation, the second EFM having a second grid pattern different from the first grid pattern, and performing other MAC operations based on the second EFM and the convolution kernel.

[0213] Attach Selected Examples

[0214] Example 1 provides a DNN, which includes an attention module, which is configured to receive an EFM generated from a grid representation of an object, the grid representation including a plurality of node elements arranged in a grid pattern, and generate an attention kernel, the attention kernel including an attention weight, the value of which is determined based on the EFM; and a convolution operator, which is configured to receive the EFM, receive the attention kernel from the attention module, and generate an OFM based on the EFM, the attention kernel and a convolution kernel, the convolution kernel including a convolution weight, the value of which is determined by training the DNN.

[0215] Example 2 provides a method as described in Example 1, wherein the attention module is configured to generate the attention kernel based on the dimension of the convolution kernel.

[0216] Example 3 provides a method as described in Example 2, wherein the convolution operator is configured to generate the OFM by performing element-wise multiplication on the focus kernel and the convolution kernel.

[0217] Example 4 provides a method as described in Example 3, wherein the focus module is configured to determine the dimension of the focus kernel based on one or more dimensions of the EFM and the dimension of the convolution kernel.

[0218] Example 5 provides the method of Example 3, wherein the convolution operator is configured to generate the OFM by performing a multiply-accumulate (MAC) operation on the EFM and a result of the element-wise multiplication.

[0219] Example 6 provides a method as described in Example 1, wherein the EFM is a first EFM, the attention kernel is a first attention kernel including a first attention weight, the value of the first attention weight is determined based on the first EFM, the attention module is also configured to receive a second EFM generated based on the grid representation and generate a second attention kernel including a second attention weight, the value of the second attention weight is determined based on the second EFM, and the convolution operator is configured to further generate the OFM based on the second EFM and the second attention kernel.

[0220] Example 7 provides a method as described in Example 6, wherein the first EFM includes the plurality of node elements arranged in a first grid pattern, the second EFM includes the plurality of node elements arranged in a second grid pattern, and the first grid pattern has a different shape from the second grid pattern.

[0221] Example 8 provides a method as described in Example 6, wherein the first EFM includes a first node element in addition to the multiple node elements, the second EFM includes a second node element in addition to the multiple node elements, and the first node element has a different value from the second node element.

[0222] Example 9 provides a method as described in Example 6, wherein the convolution operator is configured to generate the OFM by accumulating results of a first convolution operation and a second convolution operation, wherein the first convolution operation is based on the first EFM, the first focus kernel and the convolution kernel, and the second convolution operation is based on the second EFM, the second focus kernel and the convolution kernel.

[0223] Example 10 provides the method of Example 1, wherein the EFM is generated by adding one or more additional node elements to the mesh representation, and the EFM includes the plurality of node elements and the one or more additional node elements.

[0224] The above description of the illustrated implementations of the present disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although specific implementations of the present disclosure and examples thereof are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure in accordance with the above detailed description.

Claims

1. A deep neural network (DNN), the DNN comprising: The fill module is configured as follows: receiving an input feature map, the input feature map comprising a grid representation of an object, the grid representation comprising a plurality of node elements arranged in a grid pattern, and generating an extended feature graph by adding one or more additional node elements to the mesh representation, the extended feature graph comprising the plurality of node elements and the one or more additional node elements; as well as A convolution operator is configured to generate an output feature map based on the expanded feature map and a convolution kernel, wherein the convolution kernel includes convolution weights.

2. The DNN of claim 1, wherein: The grid pattern is a two-dimensional grid pattern including a plurality of edges, and the filling module is configured to generate the extended feature graph by adding the one or more additional node elements on at least one of the plurality of edges.

3. The DNN of claim 1, wherein: An additional node element of the one or more additional node elements has a value of zero.

4. The DNN of claim 1, wherein: An additional node element of the one or more additional node elements has a value of a node element of the plurality of node elements.

5. The DNN of claim 4, wherein: The grid pattern is a two-dimensional grid pattern including a first edge and a second edge opposite to the first edge, the node elements are arranged on the first edge, and the filling module is configured to generate the extended feature map by adding the additional node elements on the second edge.

6. The DNN of claim 1, wherein: The extended feature map has a grid pattern different from the grid, and the plurality of node elements and the one or more additional node elements are arranged in the different grid pattern.

7. The DNN of claim 6, wherein: The grid pattern is a two-dimensional grid pattern, and the extended feature map has a three-dimensional grid pattern.

8. The DNN of claim 6, wherein: The different grid patterns have a torus shape or a hollow cylindrical shape.

9. The DNN of claim 1, wherein: The extended feature map is a first extended feature map having a first grid pattern, The filling module is further configured to generate a second extended feature map by adding one or more other node elements to the grid representation, the second extended feature map having a second grid pattern different from the first grid pattern, and The convolution operator is further configured to perform other MAC operations based on the second expanded feature map and the convolution kernel.

10. The DNN of claim 9, wherein: The convolution operator is configured to generate the output feature map by the following steps: Performing a first convolution operation based on the first extended feature map and the convolution kernel; Performing a second convolution operation based on the second expanded feature map and another convolution kernel; and Accumulate the result of the first convolution operation and the result of the second convolution operation.

11. The DNN of claim 1, wherein: The convolution weights are determined by training the DNN.

12. The DNN of claim 11, wherein: The convolution operator is configured to generate the output feature map further based on an attention kernel, and the attention kernel includes an attention weight determined based on the expanded feature map.

13. The DNN of claim 12, wherein: The convolution operator is configured to generate the output feature map by performing element-wise multiplication on the convolution kernel and the attention kernel.

14. A method for deep learning by a deep neural network (DNN), the method comprising: receiving an input feature map, the input feature map comprising a grid representation of an object, the grid representation comprising a plurality of node elements arranged in a grid pattern; and generating an extended feature graph by adding one or more additional node elements to the mesh representation, the extended feature graph comprising the plurality of node elements and the one or more additional node elements; and An output feature map is generated based on the expanded feature map and a convolution kernel, wherein the convolution kernel includes convolution weights.

15. The method of claim 14, wherein: The grid pattern is a two-dimensional grid pattern including a plurality of edges, and adding the one or more additional node elements to the grid representation includes adding the one or more additional node elements on at least one of the plurality of edges.

16. The method of claim 14, wherein: An additional node element of the one or more additional node elements has a value of zero.

17. The method of claim 14, wherein: An additional node element of the one or more additional node elements has a value of a node element of the plurality of node elements.

18. The method of claim 17, wherein: The grid pattern is a two-dimensional grid pattern including a first edge and a second edge opposite to the first edge, the node elements are arranged on the first edge, and adding the one or more additional node elements to the grid representation includes adding the additional node elements on the second edge.

19. The method of claim 14, wherein: The extended feature map has a grid pattern different from the grid, and the plurality of node elements and the one or more additional node elements are arranged in the different grid pattern.

20. The method of claim 19, wherein: The grid pattern is a two-dimensional grid pattern, and the extended feature map has a three-dimensional grid pattern.

21. The method of claim 19, wherein: The different grid patterns have a torus shape or a hollow cylindrical shape.

22. The method of claim 14, wherein: The extended feature map is a first extended feature map having a first grid pattern, and the method further comprises: generating a second extended feature graph by adding one or more other node elements to the grid representation, the second extended feature graph having a second grid pattern different from the first grid pattern, and Other MAC operations are performed based on the second expanded feature map and the convolution kernel.

23. The method of claim 22, wherein: Generating the output feature map includes: Performing a first convolution operation based on the first extended feature map and the convolution kernel; Performing a second convolution operation based on the second expanded feature map and another convolution kernel; and Accumulate the result of the first convolution operation and the result of the second convolution operation.

24. The method of claim 14, wherein: The convolution weights are determined by training the DNN.

25. The method of claim 14, wherein: Generating the output feature map includes: Further generating the output feature map based on the attention kernel, Wherein, the attention kernel includes an attention weight determined based on the extended feature map.

Images