Feature extractor and method, point cloud processing network and chip

Abstract

The application discloses a feature extractor and method, a point cloud processing network and a chip. In order to solve the problems of large resource consumption, low precision and difficulty in hardware implementation when processing the output information of an event camera, the application relates to a point cloud processing network, directly processes an event stream, and uses a tensor sequence decomposition method to compress weight parameters of a feature extractor when extracting features. In the case of reaching the same precision as a conventional processing method, the network size, parameter quantity and power consumption can be effectively reduced, and the hardware implementation is easy, which provides an effective solution for low-computing, high-precision and generalized motion recognition tasks. The application is suitable for the field of event cameras or computer vision.

Description

Technical Field

[0001] This invention relates to a feature extractor and method, a point cloud processing network and chip, and more particularly to a lightweight feature extractor and method, a point cloud processing network and chip based on tensor sequence decomposition. Background Technology

[0002] Unlike traditional frame image sensors that sample at equal time intervals, event cameras or DVS (Dynamic Vision Sensors) generate events (also known as event streams or pulse streams) based on the relative changes in light intensity. Event output can reach millions of events per second, and has advantages such as ultra-low power consumption, low latency, and high dynamic range.

[0003] The DVS output includes coordinates, time, and polarity, but not intensity. For Artificial Neural Networks (ANNs), this means they cannot analyze the cause of events, potentially providing incorrect feature information. Furthermore, ANNs typically require compressing events over a period of time into image or voxel data for processing. Figure 1 As shown, this method consumes a lot of computing resources, increases data density and processing volume, weakens the advantage of sparse event camera data, and is not suitable for scenarios with fast-moving action.

[0004] To complete processing with limited computing resources, point-based networks (PointNet) make it possible to directly use event data as input, such as... Figure 2 As shown. However, existing point cloud networks still fall short in terms of accuracy, and most existing technologies focus on optimizing the network structure, paying relatively little attention to sampling points. Since point cloud networks sample only a small fraction of DVS events (typically 0.1% to 1%), and the point density of DVS events varies over time, the random sampling used in existing technologies may lose valuable information and struggle to maintain high accuracy in action recognition. Furthermore, existing point cloud processing network models are still not streamlined enough, making them difficult to deploy on battery-powered or resource-constrained devices.

[0005] Spiking neural networks are third-generation neural networks that can uniquely handle this type of sparse and asynchronous information, making them particularly suitable for event-based vision. However, they are difficult to design because they require design based on asynchronous circuits, and there is a lack of commercial EDA tools specifically for asynchronous design. Hardware implementation and testing are also challenging.

[0006] In view of the above-mentioned problems in the existing technology, there is a need for a lightweight event-based action recognition method with small number of parameters and computational load, low power consumption, uniform and effective sampling, high accuracy, low resource requirements and easy hardware implementation. Summary of the Invention

[0007] To solve or alleviate some or all of the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A feature extractor applied to a neural network includes a weight compression module for compressing the weight parameters of the neural network using a tensor sequence decomposition method.

[0009] The tensor sequence decomposition method is used to decompose the n-dimensional original weight parameters. Reconstructed into n tensor cores The product of n, where n is a positive integer, l1, l2, ..., l n For index, m∈[1, n], where the tensor core It is a three-dimensional tensor r m-1 ×l m ×r m r m Tensor Core Rank.

[0010] Use double-indexed vector o m and p m Indicate l m :l m =o m ×p m , where o m p m These represent the widths of the two neural network layers before and after the synaptic weights, respectively.

[0011] In one type of embodiment, r m It is greater than or equal to the third threshold and less than or equal to the fourth threshold.

[0012] In one type of embodiment, the l m Double-indexed vectors can be decomposed into higher-order or higher-dimensional vectors.

[0013] In one type of embodiment, the l m The order or dimension of the double-indexed vector is greater than or equal to the fifth threshold and less than or equal to the sixth threshold.

[0014] In one embodiment, the feature extractor includes at least one level of mutually coupled local feature extraction module and global feature extraction module;

[0015] The local feature extraction module is used to group the input of the feature extractor to achieve local feature extraction. Then, the global feature extraction module integrates the extracted local features to establish global relationships.

[0016] In one embodiment, the local feature extraction module includes a grouping unit, a residual local feature extraction unit, and a pooling unit;

[0017] The grouping unit is used to group the input of the feature extractor;

[0018] The residual local feature extraction unit is coupled to the grouping unit, and the pooling unit is coupled to the residual local feature extraction unit. By performing residual feature extraction and pooling on each group of data respectively, local features corresponding to each group are obtained.

[0019] The global feature extraction module is used to merge the local features of each group to obtain a tensor representing the global feature dimension.

[0020] In one embodiment, the pooling is one of max pooling, average pooling, or summation pooling.

[0021] In one embodiment, S key points are selected, and at least one data point is obtained around each key point as the same group, thereby obtaining S groups of data, where S is a positive integer.

[0022] In one type of embodiment, at least S key points are selected based on random sampling or farthest point sampling methods;

[0023] The K-nearest neighbor method selects at least one data point around a key point as a group, or selects at least one data point uniformly or randomly within a certain spatial domain around the key point as a group.

[0024] In one embodiment, the global feature extraction module performs global feature extraction through residual connections.

[0025] In one embodiment, the residual local feature extraction unit includes an upscaling unit and a residual feature extraction unit;

[0026] The dimension-up unit is used to increase the dimension;

[0027] The residual feature extraction unit is coupled to the dimensionality-upgrading unit, and their input and output dimensions are the same.

[0028] In one embodiment, the residual feature extraction unit and / or the global feature extraction module includes:

[0029] Several clusters of neurons;

[0030] After being weighted by the first weight matrix, the input of the residual feature extraction unit or the global feature extraction module is projected onto the first neuron cluster;

[0031] After being weighted by the second weight matrix, the output of the first neuron cluster is projected onto the second neuron cluster, and so on.

[0032] The output of the (M-1)th neuron cluster is weighted by the second weight matrix and summed with the input of the residual feature extraction unit or the global feature extraction module. The summation result is then projected onto the Mth neuron cluster, and the output of the Mth neuron cluster is used as the output of the residual feature extraction unit or the global feature extraction module, where M is a positive integer greater than or equal to 2.

[0033] In one type of embodiment, the number of subsequent groups is half the number of preceding groups.

[0034] In one embodiment, the neural network is a point cloud processing network.

[0035] A feature extraction method applied to neural networks uses tensor sequence decomposition to compress the weight parameters of the neural network;

[0036] The tensor sequence decomposition method is used to decompose the n-dimensional original weight parameters. Reconstructed into n tensor cores The product of n, where n is a positive integer, l1, l2, ..., l n For index, m∈[1,n]; where, the tensor core It is a three-dimensional tensor r m-1 ×l m ×r m r m Tensor Core Rank.

[0037] In one type of embodiment, a double-index vector o is used. m and p m Indicate l m :l m =o m ×p m , where o m p m These represent the widths of the two neural network layers before and after the synaptic weights, respectively.

[0038] In one type of embodiment, the l m Double-indexed vectors can be decomposed into higher-order or higher-dimensional vectors.

[0039] In one type of embodiment, feature extraction is performed based on at least one level of mutually coupled local feature extraction module and global feature extraction module;

[0040] The local feature extraction module is used to group the input of the feature extractor to achieve local feature extraction. Then, the global feature extraction module integrates the extracted local features to establish global relationships.

[0041] In one embodiment, the local feature extraction module includes a grouping unit, a residual local feature extraction unit, and a pooling unit;

[0042] The grouping unit is used to group the input of the feature extractor;

[0043] The residual local feature extraction unit is coupled to the grouping unit, and the pooling unit is coupled to the residual local feature extraction unit. By performing residual feature extraction and pooling on each group of data respectively, local features corresponding to each group are obtained.

[0044] The global feature extraction module obtains a tensor representing the global feature dimension based on the local features of each group.

[0045] A point cloud processing network, the point cloud processing network comprising:

[0046] The spatiotemporal preprocessing module is used to sample the received event cloud or event stream to obtain preprocessed point cloud data;

[0047] The hierarchical structure, whose input is coupled to the output of the spatiotemporal preprocessing module, includes a feature extractor as described above for feature extraction;

[0048] The classification module, whose input is coupled to the output of the hierarchical structure, uses the extracted point cloud features for training or inference to obtain the classification result.

[0049] In one embodiment, the spatiotemporal event stream is segmented using a sliding window, and the sampled data within each sliding window is normalized.

[0050] In one embodiment, each sliding window is evenly divided into at least two sub-windows, and the same number of data points are randomly sampled in each sub-window to obtain point cloud data corresponding to the sliding window.

[0051] In one type of embodiment, the number of child windows is greater than or equal to a first quantity threshold and less than or equal to a second quantity threshold.

[0052] In one embodiment, the hierarchical structure extracts features based on the point cloud data corresponding to each sliding window, and then the classification module obtains the initial classification result.

[0053] Multiple sliding windows correspond to multiple initial classification results, and the classification module further uses a voting mechanism to obtain the final classification result.

[0054] In one type of embodiment, adjacent sliding windows overlap.

[0055] In one embodiment, the overlapping area between two adjacent sliding windows is half the length of the sliding window.

[0056] In one embodiment, the classification module is a deep neural network.

[0057] A chip, comprising:

[0058] A feature extractor, as described above, is used to extract features;

[0059] Alternatively, a point cloud processing network, as described above.

[0060] In one embodiment, the chip is an AI chip.

[0061] Some or all of the embodiments of the present invention have the following beneficial technical effects:

[0062] 1) This invention is based on point cloud processing, which requires a small amount of data, can make full use of the characteristics of events, and has the advantages of being lightweight due to its small network size and small number of parameters.

[0063] 2) This invention maintains and optimizes spatiotemporal features by reconstructing data representation and sampling. The sub-window makes sampling more uniform, and can capture enough points even in areas with low time point density, so that slow motion can also be represented, avoiding the loss of valuable information, thereby improving accuracy and maintaining the lightweight nature of the network.

[0064] 3) The lightweight point cloud network of the present invention further compresses the number of parameters and simplifies the size of the neural network through the Tensor train decomposition strategy, which greatly reduces the computational complexity and power consumption, while the accuracy is almost unaffected.

[0065] 4) Each tensor kernel decomposed by this invention is very small, and the size of multiple tensor kernels is uniform, making it easy to implement in hardware and suitable for edge computing.

[0066] 5) The lightweight point cloud network of this invention, abbreviated as TTPOINT, is flexible and universal, and can achieve good results on data of different sizes without requiring specific design of the network structure. It provides a solution for point cloud processing of low-computation, high-precision, and generalized action recognition tasks.

[0067] Further beneficial effects will be described in the preferred embodiments.

[0068] The technical solutions / features disclosed above are intended to summarize the technical solutions and features described in the Detailed Embodiments section, and therefore the scope of the description may not be entirely the same. However, these new technical solutions disclosed in this section are also part of the numerous technical solutions disclosed in this invention document. The technical features disclosed in this section, together with the technical features disclosed in the subsequent Detailed Embodiments section and some contents in the drawings not explicitly described in the specification, disclose more technical solutions in a reasonable combination.

[0069] The technical solution formed by combining all the technical features disclosed at any position in this invention is used to support the summary of the technical solution, the modification of the patent document, and the disclosure of the technical solution. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of how a traditional artificial neural network (ANN) processes the DVS output event stream;

[0071] Figure 2 This is a schematic diagram illustrating the use of Point-based networks (PointNet) in existing technologies to process DVS output event streams;

[0072] Figure 3 This is a block diagram of a point cloud processing network in a certain embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of second-order sampling performed by the spatiotemporal preprocessing module of the present invention;

[0074] Figure 5 This is a schematic diagram illustrating the point cloud density statistics within a local area before and after using a sub-window;

[0075] Figure 6 Visualization results of point cloud data with and without child windows and with child windows sampled;

[0076] Figure 7 This is a schematic diagram of the hierarchical structure of the present invention;

[0077] Figure 8 This is a schematic diagram of tensor sequence decomposition in a certain embodiment of the present invention;

[0078] Figure 9 This is a schematic diagram of tensor sequence decomposition in a preferred embodiment of the present invention. Detailed Implementation

[0079] Since it is impossible to exhaustively describe all alternative solutions, the key points of the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Other technical solutions and details not disclosed in detail below generally belong to technical objectives or features that can be achieved by conventional means in the art, and due to space limitations, they will not be described in detail here.

[0080] Unless it refers to division, the " / " in any position in this invention represents logical "OR". The serial numbers "first", "second", etc., in any position in this invention are merely descriptive distinguishing marks and do not imply an absolute temporal or spatial order, nor do they imply that terms prefixed with such serial numbers necessarily refer to different things than the same terms prefixed with other modifiers.

[0081] This invention describes various key points used to combine into various specific embodiments, which will be incorporated into various methods and products. In this invention, even if a key point is described only when introducing a method / product solution, it means that the corresponding product / method solution also explicitly includes that technical feature.

[0082] The description of the existence or inclusion of a step, module, or feature at any location in this invention does not imply that such existence is exclusive or unique. Those skilled in the art can obtain other embodiments by supplementing the technical solutions disclosed in this invention with other technical means. The embodiments disclosed in this invention are generally for the purpose of disclosing preferred embodiments, but this does not imply that opposite embodiments of the preferred embodiments are excluded by this invention. As long as such opposite embodiments at least solve one of the technical problems of this invention, they are intended to be covered by this invention. Based on the key points described in the specific embodiments of this invention, those skilled in the art can substitute, delete, add, combine, or change the order of certain technical features to obtain a technical solution that still follows the concept of this invention. These solutions that do not depart from the technical concept of this invention are also within the protection scope of this invention.

[0083] Definitions:

[0084] Event imaging devices are a new type of biomimetic visual sensor, also known as neuromorphic sensors, such as event cameras, dynamic vision sensors (DVS, DAVIS), and event-based imaging fusion sensors. The following discussion will use event cameras as an example, but is not limited to them.

[0085] The event camera captures changes / motion information in the scene, and the output event stream is a time-series data recording changes in image spatial intensity in chronological order. Events in the event stream are based on Address Event Representation (AER) or similar methods. Each event includes the coordinates of the event and the timestamp t (typically accurate to µs / ns), as well as the polarity p of the light intensity change (i.e., brightening or darkening) and / or the photovoltage value of the pixel (i.e., grayscale value). Other representations including coordinate and time information can also be used. In some cases, polarity or grayscale value can be ignored. Furthermore, the coordinates of the event correspond to the sensor dimension. The following description uses a two-dimensional event imaging device and its corresponding action recognition dataset as an example, where the event coordinates are (x, y). However, the sensor can be one-dimensional or three-dimensional, and the dataset corresponding to the sensor can be any other dataset for any purpose; this invention does not limit this.

[0086] Point cloud processing networks are deep learning-based networks that can convert event stream data into point cloud data, automatically extract features, and perform tasks such as target recognition, tracking, and pose estimation.

[0087] This invention relates to a lightweight event-based action recognition method that leverages the advantages of point clouds in maintaining the sparsity of event data and low computational load. Figure 3 This is a block diagram of a point cloud processing network in a certain embodiment of the present invention, including a spatiotemporal preprocessing module, a hierarchical structure module, and a classification module.

[0088] The spatiotemporal preprocessing module is used to sample the received event cloud or event stream to obtain preprocessed point cloud data. The event cloud is the spatiotemporal event stream output by the event imaging device.

[0089] The hierarchical structure, whose input is coupled to the output of the spatiotemporal preprocessing module, is used for point cloud feature extraction.

[0090] The classification module, which is coupled to the output of the hierarchical structure, uses the extracted point cloud features for training or inference to obtain the classification result.

[0091] Figure 4 This is a schematic diagram of the second-order sampling of the spatiotemporal preprocessing module of the present invention. First, the point cloud data is segmented by a sliding window, and the data in each sliding window is further evenly divided into multiple sub-windows, and sampling is performed in each sub-window.

[0092] Unlike existing random sampling methods, the spatiotemporal preprocessing module of this invention uses a second-order sampling method to uniformly sample over time at different motion speeds. It includes the following steps:

[0093] Step S11: Segment the spatiotemporal event stream using a sliding window, also known as editing or slicing.

[0094] Step S12: Divide each sliding window into multiple sub-windows on an equal footing, and sample from each sub-window. Let N be the number of points sampled in each sliding window, and d be the number of sub-windows in each sliding window. Sample N / d points in each sub-window.

[0095] The event stream output by an event camera is a time-series data record of spatial intensity changes in an image in chronological order. For example, a simple event can be represented by e. m =(x m y m , t m p m The statement indicates that, in some cases, the polarity of an event can be ignored, and this invention does not impose any limitations on this. Where m represents the index of the m-th element in the identifier sequence, the event set in action recognition AR data can be represented as:

[0096] AR raw ={e m =(x m y m , t m (1) | m = 1, 2, ...

[0097] Among them, AR raw This represents the set of event data contained in the action recognition dataset, where m represents the sequence number of the current event, and x represents the sequence number of the event. m y m The coordinates of the current event, t m p m These represent the timestamp and polarity of the current event, respectively.

[0098] This invention segments the spatiotemporal event stream using a sliding window, with the point cloud data being three-dimensional spatiotemporal events without polarity distinction. Since the length of each sample in the action recognition dataset is different, this invention normalizes the sampled data within each sliding window so that the spatiotemporal coordinates are normalized to [0, 1].

[0099] AR point ={e′ m =(x′) m y′ m ,t′ m )|m=1,2,......}(2)

[0100]

[0101]

[0102]

[0103] Among them, AR point e represents the set of point cloud data contained in the action recognition dataset. m =(x m y m , t m ) and e′ m =(x′) m y′ m ,t′ m ) represent the current event information and its corresponding normalized event information, respectively. m t represents the timestamp of the current event. n t0 and t0 represent the current endpoint and starting point of the sliding window, respectively. The ordinate, x... m -x0 represents the distance or difference between the maximum and minimum values ​​of the x-coordinates of all events within the current sliding window. m-y0 represents the distance or difference between the maximum and minimum values ​​of the y-coordinates of all events within the current sliding window.

[0104] Alternatively, x0 = 0, y0 = 0, x m y m This is the value corresponding to the sensor's maximum resolution. For example, for a DVS with a resolution of 256×128, x m =256, y m =128.

[0105] This invention uses a sliding window to separate a single action from a set of repetitive actions within the entire sample. The normalized point cloud dataset within each sliding window can be represented as:

[0106] AR clip =clip i {e k→l}|i∈(1,n win )|t l -t k =L (6)

[0107] Where L is the length of the sliding window, n win The number of sliding windows, clip i {e k→l} indicates that the i-th sliding window includes events numbered k to l (e k to e l ), t k and t l For each event e k and e l The timestamp, the window length L of the i-th sliding window = t l -t k .

[0108] Optionally, adjacent sliding windows may overlap. Preferably, the overlap between adjacent sliding windows is L / 2.

[0109] To avoid the limitations of random sampling within a sliding window, the inventors further employ a segmentation strategy, uniformly dividing each sliding window into multiple sub-windows (e.g., d sub-windows), and randomly sampling the same number of points (e.g., Q / d points, where Q is the total number of sampling points within the sliding window) within each sub-window.

[0110] clip = {sub i {e a→b}|i∈[0,n sub ]|t b -t a =L sub} (7)

[0111] Where clip represents any sliding window, sub i {e a→b} indicates that the i-th sub-window includes events numbered a to b (e a to e b ), n sub Number the child window, L sub The length of the child window.

[0112] Figure 5 This is a schematic diagram illustrating the point cloud density statistics within a local area before and after using a sub-window, where... Figure 5 (a) shows how the DVS output events change over time with the drumming action. Figure 5 (b) shows the statistical structure of point cloud density in a local area before and after using a sub-window. Figure 6 The visualization results are shown for point cloud data sampled with and without child windows. Figure 6 (a) Visualization of point cloud data after sampling using only a sliding window. Figure 6 (b) Visualization of the point cloud data after further sampling using sub-windows based on the sliding window. It can be seen that sampling using only the sliding window yields a higher probability of obtaining points from moments with high motion speed, while fewer sampling points are obtained from moments with low motion speed.

[0113] A larger 'd' results in more sub-windows, leading to the collection of more points in sparse event regions and insufficient points in dense event regions, resulting in decreased accuracy. Conversely, a smaller 'd' concentrates the collected points in dense event regions, which is not conducive to capturing low-density events generated by slow movements and makes it difficult to fully represent slow movements. To ensure uniform sampling over time at different motion speeds, thereby uniformly capturing motion events in the time domain, the number of sub-windows can optionally be greater than or equal to a first threshold and less than or equal to a second threshold. Preferably, the number of sub-windows 'd' is 8 or 16.

[0114] Optionally, the length of the sub-window is in milliseconds. For example, if the sliding window is 0.5s and each sliding window needs to sample 1024 points, and the number of sub-windows is 8, then the length of each sub-window is 62.5ms, and each sub-window randomly samples 128 points, with each point including 3D information (x′). m y′ m ,t′ m ).

[0115] With the second-order sampling method of the present invention, the sampling becomes more uniform over time, and even slow motion can be represented.

[0116] Figure 7This is a schematic diagram of the hierarchical structure of the present invention. The hierarchical structure extracts features for each sliding window. The hierarchical structure includes p-level feature extractors, where p is a positive integer. Each level includes a local feature extraction module and a global feature extraction module. Preferably, p is 4.

[0117] The local feature extraction module extracts local features from the point cloud data sampled within each sliding window. Then, the global feature extraction module integrates the extracted local features to establish a global relationship.

[0118] The local feature extraction module groups the point cloud data within the sliding window, such as into S groups. Optionally, S key points are selected based on the Farthest Point Sampling (FPS) method, and N points around each key point are selected as the same group. For example, the K-nearest neighbor method can be used to identify at least one neighboring point around any key point, or a sphere can be drawn in the spatial domain around the key point, and N points are uniformly or randomly selected within the sphere around the key point as a group.

[0119] By grouping points, the point cloud dataset within the sliding window is mapped from a global point cloud distribution to a local point cloud distribution. For each group of points, residual local feature extraction and pooling are performed.

[0120] The process involves dimensionality enhancement through a residual local feature extraction unit. For example, the dimension (or number of dimensions) of each point within the group is increased from D to D'. Specifically, for the residual local feature extraction unit in the first level, the dimension (or number of dimensions) is increased from 3 to 32; for other levels, D' = 2D. Subsequently, pooling is used to obtain the features corresponding to the group. The pooling methods include max pooling, average pooling, and summation pooling.

[0121] Subsequently, the global feature unit synthesizes the relevant local features to obtain a tensor representing the global feature dimension corresponding to that level. After global feature extraction, the represented features are of a higher level.

[0122] Similarly, the local feature extraction module in the next-level feature extractor groups its input data, performs residual feature extraction and pooling for each group of points, and then merges the relevant local features through the global feature unit to obtain a tensor representing the global feature dimension corresponding to that level.

[0123] The first local feature extraction module in the second-level feature extractor groups its input data into S', where S' = S / 2. After residual feature extraction and pooling, it is processed by the second global feature extraction module, and so on.

[0124] For example, the hierarchical structure includes four levels. The first-level feature extractor includes a first local feature extraction module and a first global feature extraction module. A sliding window contains tens of thousands of point cloud data points. The first local feature extraction module divides the point cloud data within the sliding window into S groups, and randomly selects N points from each group. At this point, the dimension D of each point is 3, i.e., (x′... m y′ m ,t′ m Let N1 be 24. The input of the first local feature extraction module can be represented as [N1,S,D] = [24,512,3]. After dimensionality increase and pooling by the first local feature extraction module, its output is [N1,S,D'] = [1,512,32]. The first global feature unit merges related local features, and its output represents higher-level information, but its input and output dimensions remain the same, so the output is still [1,512,32]. Subsequently, the output of the second local feature extraction module in the second stage is [N2,256,64], and the output of the second deglobal feature extraction module in the second stage is [1,256,64]. And so on, the output of the third stage is [1,128,128], the output of the third stage is [1,64,256]...

[0125] The hierarchical structure of this invention, which uses a multi-level stacking method, can effectively reduce the reduction of model parameters without compromising accuracy.

[0126] Optionally, the local feature extraction module includes a grouping unit, a residual local feature extraction unit, and a pooling unit. The grouping unit is used to group the input of the feature extractor; the residual local feature extraction unit is coupled to the grouping unit, and the pooling unit is coupled to the residual local feature extraction unit. By performing residual feature extraction and pooling on each group of data respectively, local features corresponding to each group are obtained.

[0127] The residual local feature extraction unit includes an upscaling unit and a residual feature extraction unit.

[0128] Optionally, the global feature extraction module is a residual global feature extraction module, which performs global feature extraction through residual connections.

[0129] Optionally, the residual local feature extraction unit and the residual global feature extraction module use the same residual feature extraction unit, wherein the input and output dimensions of the residual feature extraction unit are the same and remain unchanged.

[0130] In one embodiment, the residual feature extraction unit includes several neuron clusters. The input of the residual feature extraction unit is weighted by a first weight matrix and then projected onto a first neuron cluster. The output of the first neuron cluster is weighted by a second weight matrix and then projected onto a second neuron cluster, and so on. The output of the (M-1)th (second-to-last) neuron cluster is weighted by the Mth weight and then added (or summed) to the input of the residual feature extraction unit. The sum is then projected onto the Mth neuron cluster, and the output of the Mth neuron cluster is the output of the residual feature extraction unit. Here, the Mth neuron cluster is the last neuron cluster, and M is a positive integer greater than or equal to 2.

[0131] Optionally, neurons in the plurality of neuron clusters are activated based on the ReLU function.

[0132] Preferably, the Tensor Train Decomposition method is used to compress the weight parameter matrices of the local feature extraction module and the global feature extraction module.

[0133] Optionally, the weight parameter matrix, after being decomposed into tensor sequences and normalized, is input into the ReLU neuron for activation.

[0134] Figure 8 This is a schematic diagram of tensor sequence decomposition in a certain embodiment of the present invention. For an n-dimensional weight... It can be approximated as several tensor cores The product of:

[0135]

[0136] in It is composed of indices l1, l2, ..., l n The specified element, r m Tensor Core rank It is a three-dimensional tensor The matrix slice, where R represents the real number field.

[0137] The rank r of tensor sequence decomposition m Choosing an appropriate rank is crucial, as it influences the size of the compression model and its performance on the dataset. m The larger it is, the closer it gets. But with r m Increasing the value of r results in a significant increase in the number of parameters, a decrease in the compression ratio, and an increase in computational complexity and cost. Alternatively, r... m Greater than or equal to the third threshold, less than or equal to the fourth threshold, preferably, r m It can be 4 or 8.

[0138] Use double-indexed vector om p m Represent each integer l m For example, l m =o m ×p m , where o m p m This represents the width of a two-layer neural network, specifically the width of the layers before and after the synaptic weights, such as o. m p represents the width of the input layer neural network before the synaptic weights. m This represents the width of the output layer neural network after describing the synaptic weights. Thus, each tensor core is updated to... Furthermore, the weight matrix is ​​reshaped (or reconstructed) into a tensor.

[0139] Optionally, the network of the present invention is a multilayer perceptron (MLP), which is a fully connected network.

[0140] Figure 9 This is a schematic diagram of tensor sequence decomposition in a preferred embodiment of the present invention, such as l m =o m ×p m =256×64=16384, after tensor sequence decomposition, the original weight parameters are reshaped, for example, l m It is reshaped into a 4th-order (or 4-dimensional) tensor, denoted as l. m = [16, 16, 8, 8], that is, 16 × 16 × 8 × 8 = 16384. If r m Taking 4, the number of parameters after tensor sequence decomposition and compression is: 1×16×4 + 4×16×4 + 4×8×4 + 4×8×1 = 464. The compressed number of parameters is less than 30% of the original number of parameters 16384. Similarly, the double index vector o between the two layers of the network... m =256 and pm=64 were reshaped into o m =[4,4,4,4], pm=[4,4,2,2].

[0141] Alternatively, l can be m Decompose into higher-order tensors, such as l m =[16, 8, 8, 4, 4]. l m The lower the order of the decomposition, the larger the tensor kernel, resulting in poor compression. m More isn't necessarily better. While increasing the number of decompositions reduces the tensor kernel size and compresses the number of parameters, the additional multiply-accumulate calculations consume more resources, increasing the computational load and impacting processing speed. Therefore, l m The order of the decomposition is greater than or equal to the fifth threshold and less than or equal to the sixth threshold. Preferably, lm The order of decomposition is 4 or 5.

[0142] Matrix-vector multiplication in neural networks in These represent the input and output dimension values, respectively. As weight, As a bias. Reconstructed in the same way as above, using an n-dimensional tensor approximation, such that... Therefore, the output is also an n-dimensional tensor. Matrix-vector multiplication can be reformulated as:

[0143]

[0144] The tensor decomposition strategy of this invention greatly reduces neural network parameters and computational complexity, while also improving computational speed.

[0145] Furthermore, the tensor point cloud network of this invention exhibits excellent versatility, achieving good results on data of varying sizes without requiring specific network structure design. Tests were conducted using a single tensor point cloud network on small, medium, and large datasets, demonstrating that the network can effectively perform action recognition on datasets of different sizes. Without compromising accuracy, it effectively reduces the number of parameters and computational load, lowering the hardware resource and performance requirements of the devices used. This provides an effective solution for point cloud processing tasks requiring low computational complexity, high accuracy, and generalized action recognition.

[0146] Although the invention has been described with reference to specific features and embodiments, various modifications, combinations, and substitutions can be made therein without departing from the invention. The scope of protection of this invention is not limited to the specific embodiments of processes, machines, manufactures, material compositions, apparatuses, methods, and steps described in the specification, and these methods and modules may also be implemented in one or more related, interdependent, cooperative, or upstream / downstream products or methods.

[0147] Therefore, the specification and drawings should be simply regarded as a description of some embodiments of the technical solutions defined by the appended claims, and thus the appended claims should be interpreted in accordance with the principle of the greatest reasonable interpretation, and are intended to cover as much as possible all modifications, variations, combinations or equivalents within the scope of the invention, while avoiding unreasonable interpretations.

[0148] To achieve better technical effects or for the needs of certain applications, those skilled in the art may make further improvements to the technical solution based on this invention. However, even if such improvements / designs are inventive and / or progressive, as long as they rely on the technical concept of this invention and cover the technical features defined in the claims, the technical solution should also fall within the protection scope of this invention.

[0149] The technical features mentioned in the appended claims may have alternative technical features, or the order of certain technical processes or material organization may be rearranged. Those skilled in the art, upon learning of this invention, will readily conceive of these alternative means, or alter the order of the technical processes or material organization, and then employ substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the claims explicitly define the aforementioned means and / or order, these modifications, alterations, and substitutions should all fall within the scope of protection of the claims based on the principle of equivalents.

[0150] The method steps or modules described in the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application or design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered outside the scope of protection claimed by this invention.

Claims

1. A point cloud processing network, characterized in that, The point cloud processing network includes: The spatiotemporal preprocessing module is used to sample the received event cloud or event stream to obtain preprocessed point cloud data; The hierarchical structure, whose input is coupled to the output of the spatiotemporal preprocessing module, includes a feature extractor for feature extraction. The feature extractor is applied to the neural network and includes a weight compression module for compressing the weight parameters of the neural network using a tensor sequence decomposition method. The tensor sequence decomposition method is used to compress the original n-dimensional weight parameters. Reconstructed into n tensor cores The product of n, where n is a positive integer. For index, m∈[1,n]; where, the tensor core It is a three-dimensional tensor , Tensor Core rank; The classification module, whose input is coupled to the output of the hierarchical structure, uses the extracted point cloud features for training or inference to obtain the classification result; The spatiotemporal preprocessing module segments the spatiotemporal event stream using sliding windows and normalizes the sampled data within each sliding window. Each sliding window is evenly divided into at least two sub-windows, and the same number of data points are randomly sampled within each sub-window to obtain point cloud data corresponding to the sliding window. The hierarchical structure performs feature extraction based on the point cloud data corresponding to each sliding window, and then the classification module obtains the initial classification results corresponding to each sliding window. The classification module further uses a voting mechanism based on the multiple initial classification results corresponding to multiple sliding windows to obtain the final classification result.

2. The point cloud processing network according to claim 1, characterized in that, The feature extractor includes: At least one level of mutually coupled local feature extraction module and global feature extraction module; The local feature extraction module is used to group the input of the feature extractor to achieve local feature extraction. Then, the global feature extraction module integrates the extracted local features to establish global relationships.

3. The point cloud processing network according to claim 2, characterized in that, The local feature extraction module includes a grouping unit, a residual local feature extraction unit, and a pooling unit; The grouping unit is used to group the input of the feature extractor; The residual local feature extraction unit is coupled to the grouping unit, and the pooling unit is coupled to the residual local feature extraction unit. By performing residual feature extraction and pooling on each group of data respectively, local features corresponding to each group are obtained. The global feature extraction module is used to merge the local features of each group to obtain a tensor representing the global feature dimension.

4. The point cloud processing network according to claim 3, characterized in that, The residual local feature extraction unit includes an upscaling unit and a residual feature extraction unit; The dimension-up unit is used to increase the dimension; The residual feature extraction unit is coupled to the dimensionality-upgrading unit, and their input and output dimensions are the same.

5. The point cloud processing network according to claim 4, characterized in that, The residual feature extraction unit and / or the global feature extraction module include: Several clusters of neurons; After being weighted by the first weight matrix, the input of the residual feature extraction unit or the global feature extraction module is projected onto the first neuron cluster; After being weighted by the second weight matrix, the output of the first neuron cluster is projected onto the second neuron cluster, and so on. The output of the (M-1)th neuron cluster is weighted by the second weight matrix and summed with the input of the residual feature extraction unit or the global feature extraction module. The summation result is then projected onto the Mth neuron cluster, and the output of the Mth neuron cluster is used as the output of the residual feature extraction unit or the global feature extraction module, where M is a positive integer greater than or equal to 2.

6. A chip, characterized in that, include: The point cloud processing network as described in any one of claims 1-5.

Images