Method and device for processing three-dimensional data

Abstract

The present application discloses a method for processing three-dimensional data, which can be applied to the field of intelligent driving. The method includes: for the target convolution operator in the convolutional neural network, the corresponding intermediate result can be first calculated through one or more background values ​​of the three-dimensional data. Then, when processing the first data, it is only necessary to find the corresponding intermediate result according to the background identifier in the sparse distribution map, and then obtain the voxel value of the second data according to the intermediate result, without repeating the convolution processing process multiple times, thereby achieving convolution acceleration. In addition, the solution provided in the embodiment of the present application is particularly suitable for generalized sparse data with multiple background values.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for processing three-dimensional data, and a method and device for constructing a convolutional neural network. Background Art

[0002] Convolutional neural networks (CNNs) have been widely used in computer vision in recent years, performing tasks such as classification, detection, and segmentation. These tasks are typically performed on images using two-dimensional (2D) convolutions. 2D convolutions only extract spatial features from a single static image. For scenarios with 3D data features, such as video data or 3D images, 3D convolutions are required.

[0003] Deep convolutional neural networks (such as ResNet and ForkNet) have broad application prospects in the 3D field. However, the biggest challenge is the very high computational complexity of three-dimensional convolution, which typically requires more than 10 times more computing power than a two-dimensional convolutional neural network of the same size. This is especially true on the inference side, which is very sensitive to latency. Slow network inference speeds can cause lags, directly impacting the user experience on the end side. In the field of autonomous driving, where latency requirements are even higher, large amounts of three-dimensional data are well suited for processing using three-dimensional convolutional neural networks. However, because the performance of three-dimensional convolutional neural networks does not meet real-time requirements, they cannot be applied.

[0004] At present, accelerating the processing of three-dimensional data by three-dimensional convolutional neural networks has become an urgent problem to be solved. Summary of the Invention

[0005] The present invention provides a method for processing three-dimensional data to improve the computational speed of three-dimensional convolution. The present invention also provides a method, device, system, computer-readable storage medium, and computer program product for constructing a convolutional neural network.

[0006] The first aspect of the present application provides a method for processing three-dimensional data, which can be applied to the field of intelligent driving. The method includes: inputting first data, at least one first background value of the first data, and a first sparse distribution map of the first data into a target convolution operator of a convolutional neural network, wherein the convolutional neural network includes at least one first convolution operator, the target convolution operator is any one of the at least one first convolution operator, the first data includes a plurality of first voxels, at least one first background value indicates at least one background in the first data, at least one first background value corresponds one-to-one to at least one background, the first sparse distribution map describes the sparse distribution of the plurality of first voxels through background identifiers and non-background identifiers, the background identifier indicates that the first voxel corresponding to the background identifier in the first data is the background, and the non-background identifier indicates that the first data The first voxel corresponding to the non-background identifier in the first data is non-background; according to the background identifier of the background point corresponding to the first voxel indicating the background in the first data on the first sparse distribution map, the intermediate result corresponding to the background identifier of the background point is found from the corresponding relationship, the corresponding relationship indicates the relationship between the intermediate result of each of the at least one first background value and the background identifier corresponding to each of the at least one first background value, the intermediate result of one of the at least one first background values ​​is calculated by the convolution kernel of the first background value and the target convolution operator, and the intermediate result corresponding to the background identifier of the background point is used to determine the value of the second voxel in the second data; the target convolution operator is controlled to output the second data, and the second data includes multiple second voxels.

[0007] In this first aspect, the convolutional neural network refers to a three-dimensional convolutional neural network, such as ResNet and ForkNet, which includes a neural network with multiple three-dimensional convolution operators and is used to process three-dimensional data. Three-dimensional data refers to data with three dimensions and is generally sparse data. Three-dimensional data can be video data, voice data, or text data. Sparse data is data with a large number of repeated elements, which are called the background of the data.

[0008] In this application, the first data is a multidimensional array (tensor) of video data, voice data, or text data. There is at least one first background value, that is, one or more. "Multiple" in this application includes two or more, and can also be described as at least two. Generally speaking, a "background value" refers to a numerical value used to describe the background of a repeating element, which can be 0, 1, 2, ... 101, 102, 103, or any other arbitrary numerical value, for example, a decimal, a negative number, or a complex number can all be used as a background value. In a three-dimensional data, one or more backgrounds can be included, and a background value can be set for each background. For example, when air is used as the background, the background value of the air background is set to 101; when the surface of an object is used as the background, the background value of the physical surface background is set to 102; when the background is an area not detected by the depth camera, the background value of the background of the undetected area is set to 102. Sparse distribution refers to the background distribution of three-dimensional data. The background distribution can be represented by the background identifier of each voxel. The background identifier can be a numerical value or character that numbers the background, such as: 1, 2, 3 or A, B, C, etc. The background identifier is associated with the background value, for example, background identifier 1 is associated with background value 101 , background identifier 2 is associated with background value 102 , and background identifier 3 is associated with background value 103 .

[0009] In this application, three-dimensional data includes voxels. Voxel is the abbreviation of volume element, which indicates the smallest unit of spatial segmentation of three-dimensional data, similar to the pixel in two-dimensional data. The first voxel is the voxel in the first data. The second voxel is the voxel in the second data.

[0010] In this application, the first sparse distribution map refers to the sparse distribution map corresponding to the first data. A unit in the first sparse distribution map can be called a "point". Among them, the point corresponding to the first voxel indicating the background can be called a "background point", and the point corresponding to the first voxel indicating the non-background can be called a "non-background point". Non-background points can also be called "dense points". Background points are marked with background identifiers, such as: 1, 2, 3, and non-background points are marked with a unified non-background identifier.

[0011] In this application, background identifiers correspond to first background values. There are three first background values, 101, 102, and 103, and background identifiers, 1, 2, and 3. The correspondence can be as follows: 101->1, 102->2, and 103->3. The correspondence between first background values ​​and background identifiers is typically implicit, based on their order, but can also be displayed via a table or mapping. Non-background identifiers are uniformly denoted by M, which is the abbreviation for Max and represents a non-background point.

[0012] In the present application, before searching for the intermediate result from the first correspondence, the intermediate result between each first background value and the convolution kernel is calculated, such as: 101 and the intermediate result 1 of the convolution kernel, 102 and the intermediate result 2 of the convolution kernel, and 103 and the intermediate result 3 of the convolution kernel. Because the first background value corresponds to the background identifier, a correspondence between the background identifier and the convolution result can be established, such as: 1->intermediate result 1, 2->intermediate result 2, and 3->intermediate result 3. Usually, the first data includes many first voxels. In the prior art, the process of convolution processing the first data is to calculate the values ​​of these first voxels one by one with the convolution kernel. Among these first voxels, the voxels indicating the background usually account for the vast majority, and the values ​​of these voxels indicating the background are repeated in large quantities. Calculating them one by one with the convolution kernel is very slow. In this first aspect, calculations are performed for each first background value prior to the convolution kernel. After obtaining an intermediate result, the correspondence between the intermediate result and the background identifier is stored. When convolving the first data, the intermediate result can be directly found by searching the corresponding relationship for the first voxel in the first data indicating background. This intermediate result can then be used to obtain the value of the second voxel in the second data. This eliminates the need for extensive calculations between the convolution kernel and the value of each first voxel, thereby reducing computational overhead and accelerating convolution. For example, if there are 1000 first voxels in the first data, 950 of which indicate background, and only three of these 950 background-indicating first voxels indicate first background values ​​of different background types, then in this first aspect, only the three first background values ​​need to be calculated with the convolution kernel, and then a lookup operation performed based on the corresponding relationship is performed. This eliminates the need to calculate all 950 values ​​with the convolution kernel. This makes the lookup much faster than the calculation with the convolution kernel, significantly reducing overhead and significantly improving the speed of the convolution process.

[0013] In a possible implementation of the first aspect, a convolutional neural network includes: an operation subnet, a background derivation subnet, and a sparse distribution derivation subnet, wherein the network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, and the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value comes from an operator that inputs at least one first background value to a third convolution operator, and the position of the third convolution operator in the background derivation subnet corresponds to the position of the target convolution operator in the operation subnet.

[0014] In this possible implementation, the operation subnet, the background derivation subnet, and the sparse distribution derivation subnet can cooperate to input at least one first background value and a first sparse distribution map into the target convolution operator, thereby ensuring that the target convolution operator in the operation subnet can achieve convolution acceleration when processing three-dimensional data.

[0015] In a possible implementation of the first aspect, a convolutional neural network includes: an operation subnet and a sparse distribution derivation subnet, the network structure of the sparse distribution derivation subnet is the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, and the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; and at least one first background value is pre-configured for the target convolution operator.

[0016] In this possible implementation, the first background value can be pre-configured, so that at least one first background value and a first sparse distribution map can be input into the target convolution operator through the cooperation of the operation subnet and the sparse distribution deduction subnet, thereby ensuring that the target convolution operator in the operation subnet can achieve convolution acceleration when processing three-dimensional data. Moreover, there is no need to calculate the first background value through the background deduction subnet, which can save computing resources of the inference side device.

[0017] In a possible implementation of the first aspect, the intermediate result in the above-mentioned correspondence is a result obtained by calculating the first background value at different positions of the convolution kernel; the above-mentioned step: according to the background identifier of the background point corresponding to the first voxel indicating the background on the first data on the first sparse distribution map, searching for the intermediate result corresponding to the background identifier of the background point from the correspondence, including: for each first voxel indicating the background in the sliding window of the first data, according to the background identifier corresponding to each first voxel indicating the background in the first sparse distribution map, and the position of each first voxel indicating the background in the sliding window, searching for the intermediate result corresponding to the background identifier at the corresponding position from the correspondence, and the sliding window of the first data is the same size as the convolution kernel of the target convolution operator.

[0018] In this possible implementation, in a three-dimensional convolution operator, the convolution kernel is usually in the form of a multidimensional array, and the size of the convolution kernel is usually pre-set, for example: 3*3*1*L, where the first 3 represents width, the second 3 represents height, 1 represents depth, and L represents the number of channel dimensions. In the process of convolution processing of three-dimensional video data, the channel dimension usually represents color. Of course, the channel dimension can also have other definitions. In one channel dimension, the convolution kernel is like a sliding window of three-dimensional data in that one channel dimension. If the convolution kernel has multiple channel dimensions, when calculating the intermediate result of the background value, for any of the 9 positions of 3*3*1, the background value must be multiplied by the vector of the corresponding position in the L channel dimensions, and then the products of the L channel dimensions at this position are accumulated to obtain the intermediate result of this position. A sliding window in the data refers to a window that can slide across multiple voxels in the 3D data. This window can cover multiple voxels at a time. For example, if the sliding window is 3*3, then one sliding window can cover 9 voxels, which can also be referred to as the sliding window covering 9 voxels. This way, when searching for intermediate results, the corresponding intermediate result can be found based on the position of the first voxel in the sliding window and the corresponding background identifier, which speeds up the search and, in turn, the processing of 3D data.

[0019] In a possible implementation of the first aspect, the method further includes: accumulating intermediate results of first voxels indicating the background in the sliding window of the first data to obtain a value of a second voxel, and all first voxels in the sliding window of the first data indicate the background.

[0020] In this possible implementation, the value of the second voxel can be obtained by simply accumulating the intermediate results. Compared with the situation where the value of the second voxel needs to be calculated with the convolution kernel each time, the speed of processing three-dimensional data is improved.

[0021] In a possible implementation of the first aspect, the method further includes: accumulating the intermediate results of the first voxel indicating the background in the sliding window of the first data, and adding the intermediate results of the first voxel indicating the non-background to obtain the value of a second voxel, and the intermediate result of the first voxel indicating the non-background is obtained by calculating the value of the first voxel indicating the non-background and the convolution kernel.

[0022] In this possible implementation, it is only necessary to calculate the convolution kernel for each individual first voxel indicating non-background, and then accumulate the intermediate result with the first voxel indicating background to obtain the value of the second voxel. Compared with the method of requiring calculation with the convolution kernel every time to obtain the value of the second voxel, the speed of processing three-dimensional data is improved.

[0023] In a possible implementation of the first aspect, the target convolution operator is the first convolution operator in the operation subnet, and the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background, and for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

[0024] In a possible implementation manner of the first aspect, at least one first background value of the first data is represented in the form of a one-dimensional array.

[0025] In a possible implementation of the first aspect, the target convolution operator is not the first convolution operator in the operation subnetwork, and the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data.

[0026] In a possible implementation manner of the first aspect, the first background values ​​of the sub-data in different channel dimensions are different.

[0027] In this possible implementation, the first background values ​​are different, and the three-dimensional data after convolution processing has a better effect.

[0028] In a possible implementation manner of the first aspect, at least one first background value of the first data is represented in the form of a two-dimensional array.

[0029] In a possible implementation manner of the first aspect, the first data is a multidimensional array of video data, voice data, or text data.

[0030] The second aspect of the present application provides a method for processing three-dimensional data, comprising: inputting first data into a target convolution operator of a convolutional neural network, the convolutional neural network comprising at least one first convolution operator, the target convolution operator being any one of the at least one first convolution operator, the first data comprising a plurality of first voxels, the target convolution operator further inputting: at least one second background value obtained by convolution processing at least one background value of the first data, and a second sparse distribution map obtained by performing a row identification check operation on the first sparse distribution map, the identification check operation being to check whether the identifications of the points covered by the sliding window of the first sparse distribution map are all the same or not all the same, the identifications comprising background identifications and non-background identifications, and at least one first background value obtained by performing a row identification check operation on the first sparse distribution map. A background value indicates at least one background in the first data, at least one first background value corresponds one-to-one to at least one background, a first sparse distribution map describes the sparse distribution of multiple first voxels through background identifiers and non-background identifiers, the background identifier indicates that the first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicates that the first voxel corresponding to the non-background identifier in the first data is non-background; according to the background identifier representing the background point on the second sparse distribution map, the second background value corresponding to the background identifier of the background point is found, and the second background value corresponding to the background identifier of the background point is the value of the second voxel corresponding to the background point in the second data; the target convolution operator is controlled to output second data, and the second data includes multiple second voxels.

[0031] The contents of the second aspect that are repeated in the first aspect will not be repeated. The process of obtaining the second background value from the first background value can be obtained by convolving the first background value with the convolution kernel of the second convolution operator. In the process of obtaining the second sparse distribution map according to the first sparse distribution map, it is only necessary to check whether the identifiers in the sliding window are exactly the same to obtain the identifier of the second sparse distribution map, which can save computing resources. In the second aspect, after the target convolution operator receives the first data, the second background value and the second sparse distribution map, it can directly search for the corresponding second background value through the background identifier in the second sparse distribution map. The second background value found is the value of the second voxel at the position of the background identifier. It can be seen that the solution provided in the embodiment of the present application can directly find the corresponding value for the second voxel indicated by the background identifier in the second sparse distribution map, without the need for the convolution process, thereby achieving convolution acceleration.

[0032] In a possible implementation of the second aspect, the method further includes: for a non-background point on the second sparse distribution map, according to the background identifier of the target point on the first sparse distribution map, searching from the correspondence relationship for an intermediate result corresponding to the background identifier of the target point, the target point being the non-background point on the second sparse distribution map positioned to the corresponding background-indicating point on the first sparse distribution map, the correspondence indicating the relationship between the intermediate results of at least one first background value and the background identifier corresponding to at least one first background value, the intermediate result of one of the at least one first background values ​​being calculated by the convolution kernel of the first background value and the target convolution operator, the intermediate result in the correspondence being the result obtained by calculating the first background value at different positions of the convolution kernel, and the intermediate result corresponding to the background identifier of the target point being used to determine the value of the second voxel corresponding to the non-background point on the second sparse distribution map.

[0033] In this possible implementation, the value of the second voxel in the second data can be obtained by searching twice, which can further improve the speed of processing three-dimensional data.

[0034] In a possible implementation of the second aspect, the convolutional neural network includes: an operation subnet, a background derivation subnet, and a sparse distribution derivation subnet, and the network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identification verification operation on the first sparse distribution map, and the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value comes from an operator that inputs at least one first background value to a third convolution operator, at least one second background value is obtained by the third convolution operator performing convolution processing on the at least one first background value, and the position of the third convolution operator in the background derivation subnet corresponds to the position of the target convolution operator in the operation subnet.

[0035] In a possible implementation of the second aspect, the convolutional neural network includes: an operation subnet and a sparse distribution derivation subnet, the network structure of the sparse distribution derivation subnet is the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identification verification operation on the first sparse distribution map, the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value and at least one second background value are pre-configured for the target convolution operator.

[0036] In a possible implementation of the second aspect, the method further includes: accumulating the intermediate results corresponding to each target point found to obtain the value of the second voxel corresponding to the non-background point, and the points corresponding to the non-background points located on the first sparse distribution map are all points indicating the background.

[0037] In a possible implementation of the second aspect, the method further includes: accumulating the intermediate results corresponding to each target point found, and adding the intermediate results corresponding to the dense points corresponding to the non-background points located on the first sparse distribution map, to obtain the value of the second voxel corresponding to the non-background point, the dense point is the point indicating the non-background in the first sparse distribution map, and the intermediate result corresponding to the dense point is calculated by the value of the first voxel indicating the non-background corresponding to the dense point in the first data and the convolution kernel.

[0038] In a possible implementation of the second aspect, the target convolution operator is the first convolution operator in the operation subnet, and the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background, and for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

[0039] In a possible implementation manner of the second aspect, at least one first background value of the first data is represented in the form of a one-dimensional array, and at least one second background value is represented in the form of a two-dimensional array.

[0040] In a possible implementation of the second aspect, the target convolution operator is not the first convolution operator in the operation subnetwork, and the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data.

[0041] In a possible implementation manner of the second aspect, the first background values ​​of the sub-data in different channel dimensions are different.

[0042] In a possible implementation manner of the second aspect, the at least one first background value of the first data is represented in the form of a two-dimensional array.

[0043] In a possible implementation manner of the second aspect, the first data is a multidimensional array of video data, voice data, or text data.

[0044] The solutions described in the above-mentioned second aspect and any possible implementation of the second aspect will not be repeated in the second aspect for solutions similar to those described in the first aspect and any possible implementation of the first aspect. You can refer to the relevant introduction in the similar solutions described in the first aspect and any possible implementation of the first aspect for understanding.

[0045] In a third aspect, the present application provides a method for constructing a convolutional neural network, including: obtaining an initial convolutional neural network, the initial convolutional neural network includes an operation subnet, and the operation subnet includes at least one first convolution operator; according to the operation subnet, copying a sparse distribution inference subnet; constructing at least one of a first input relationship and a second input relationship, the first input relationship being the input relationship from the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position, and the second input relationship being the input relationship from the operator that inputs data to the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position, so as to obtain a target convolutional neural network.

[0046] In the third aspect, the operation subnet can be the entire initial convolutional neural network or a part of the initial convolutional neural network. The first convolution operator is a three-dimensional convolution operator. These three-dimensional convolution operators and other operators are arranged according to a certain network structure to perform processing on three-dimensional data. The network structure of the sparse distribution deduction subnet is the same as the network structure of the operation subnet. The sparse distribution deduction subnet is used to derive a sparse distribution map of three-dimensional data. In the third aspect, based on the initial convolutional neural network used only to process three-dimensional data, a sparse distribution deduction subnet of the same structure is added to the operation subnet. In this way, it is ensured that in the process of convolution processing of three-dimensional data by convolution operators at different positions in the operation subnet, the sparse distribution map at the corresponding position can be obtained from the sparse distribution deduction subnet, and the corresponding background value can be obtained from the pre-configured background value, thereby ensuring that the target convolutional neural network constructed by this application can improve the speed of convolution processing.

[0047] In a possible implementation of the third aspect, before obtaining the target convolutional neural network as described above, the method further includes: copying a background derivation subnetwork according to the operation subnetwork; constructing at least one of a third input relationship and a fourth input relationship, the third input relationship being an input relationship from a third convolution operator in the background derivation subnetwork to a first convolution operator at a corresponding position, and the fourth input relationship being an input relationship from an operator that inputs data to the third convolution operator in the background derivation subnetwork to the first convolution operator at a corresponding position, the first input relationship corresponding to the third input relationship, and the second input relationship corresponding to the fourth input relationship.

[0048] In this possible implementation, the background derivation subnet has the same network structure as the computation subnet. The background derivation subnet is used to derive background values ​​for three-dimensional data. The addition of the background derivation subnet allows for obtaining background values ​​when processing three-dimensional data, thereby ensuring that the target convolutional neural network constructed in this application can increase the speed of convolution processing.

[0049] In a possible implementation of the third aspect, the method further includes: determining an operation subnetwork from the initial convolutional neural network, wherein the input data of each operator in the operation subnetwork is sparsely distributed in the same manner in the channel dimension, and the output data of each operator is sparsely distributed in the same manner in the channel dimension, and the channel dimension represents a dimension different from the three-dimensional space.

[0050] In this possible implementation, after the operation subnet is determined, it can be copied directly according to the determined operation subnet, or the operation subnet can be cut out from the initial convolutional neural network and then copied. After the sparse distribution inference subnet and the background inference subnet are obtained by copying, and the input relationship between the operators in the three subnets is established, the remaining network in the initial convolutional neural network except the operation subnet can be spliced ​​to the position before separation from the operation subnet.

[0051] In a fourth aspect, the present application provides a device for processing three-dimensional data, the device having the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, for example: a first processing unit, a second processing unit, and a third processing unit. These three processing units can also be implemented by one or two processing units.

[0052] In a fifth aspect, the present application provides a device for processing three-dimensional data, which has the function of implementing the method of the second aspect or any possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, for example: a first processing unit, a second processing unit, and a third processing unit. These three processing units can also be implemented by one or two processing units.

[0053] In a sixth aspect, the present application provides a device for constructing a convolutional neural network, which has the function of implementing the method of the third aspect or any possible implementation of the third aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, for example: an acquisition unit, a first processing unit, and a second processing unit. These two processing units can also be implemented by one processing unit.

[0054] In a seventh aspect, the present application provides a computer device comprising at least one processor, a memory, an input / output (I / O) interface, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes the method according to the first aspect or any possible implementation of the first aspect.

[0055] In an eighth aspect, the present application provides a computer device comprising at least one processor, a memory, an input / output (I / O) interface, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes the method according to the second aspect or any possible implementation of the second aspect.

[0056] In a ninth aspect, the present application provides a computer device comprising at least one processor, a memory, an input / output (I / O) interface, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes the method according to the third aspect or any possible implementation of the third aspect.

[0057] The tenth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0058] In the eleventh aspect of the present application, a computer-readable storage medium is provided for storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.

[0059] The twelfth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the third aspect or any possible implementation of the third aspect.

[0060] The thirteenth aspect of the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0061] In the fourteenth aspect, the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the second aspect or any possible implementation of the second aspect.

[0062] In a fifteenth aspect, the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the third aspect or any possible implementation of the third aspect.

[0063] In a sixteenth aspect, the present application provides a chip system, comprising at least one processor configured to implement the functions of the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may further include a memory configured to store program instructions and data necessary for the device for processing three-dimensional data. The chip system may be composed of a chip or may include a chip and other discrete components.

[0064] In aspect 17 of the present application, a chip system is provided, comprising at least one processor configured to implement the functions of aspect 2 or any one of the possible implementations of aspect 2. In one possible design, the chip system may further include a memory configured to store program instructions and data necessary for the device for processing three-dimensional data. The chip system may be composed of a chip or may include a chip and other discrete components.

[0065] In aspect 18 of the present application, a chip system is provided, which includes at least one processor, and the at least one processor is used to implement the functions of the third aspect or any possible implementation of the third aspect. In one possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the device to construct a convolutional neural network. The chip system can be composed of a chip or can include a chip and other discrete devices.

[0066] The nineteenth aspect of the present application provides a data structure applied to the method for processing three-dimensional data described in the first or second aspect above, wherein the method for processing three-dimensional data applies at least one first background value of the first data, and the at least one background value is represented in the form of a one-dimensional array or a multi-dimensional array.

[0067] The twentieth aspect of the present application provides a data structure applied to the method for processing three-dimensional data described in the first or second aspect above, wherein the method for processing three-dimensional data is applied to a first sparse distribution map of the first data, and the first sparse distribution map is represented in the form of a multidimensional array.

[0068] In the twenty-first aspect of the present application, a data structure is provided for use in the method for processing three-dimensional data described in the first or second aspect above. The method for processing three-dimensional data is applied to first data, and the first data is sparsely distributed and identically distributed in different channel dimensions, and the serial numbers of the identically distributed channel dimensions are represented in the form of a one-dimensional array.

[0069] The solution provided in the embodiment of the present application calculates the intermediate results of each first background value and the convolution kernel before searching for the intermediate results from the first correspondence, such as: 101 and the intermediate result 1 of the convolution kernel, 102 and the intermediate result 2 of the convolution kernel, and 103 and the intermediate result 3 of the convolution kernel. Because the first background value corresponds to the background identifier, a correspondence between the background identifier and the convolution result can be established, such as: 1->intermediate result 1, 2->intermediate result 2 and 3->intermediate result 3. Usually, the first data includes many first voxels. The process of convolution processing the first data in the prior art is the process of calculating the values ​​of these first voxels one by one with the convolution kernel. Among these first voxels, the voxels indicating the background usually account for the vast majority, and the values ​​of these voxels indicating the background are repeated in large quantities. Calculating them one by one with the convolution kernel is very slow. In this application, a calculation is performed for each first background value before the convolution kernel is applied. After obtaining an intermediate result, the correspondence between the intermediate result and the background identifier is stored. When the first data is convolved, the intermediate result can be directly found by looking up the correspondence for the first voxel indicating the background in the first data. The intermediate result is then used to obtain the value of the second voxel in the second data. This eliminates the need for extensive calculations for each first voxel value and the convolution kernel, thereby saving computational overhead and achieving convolution acceleration. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1A This is a schematic diagram of an embodiment of an artificial intelligence system provided by an embodiment of the present application;

[0071] Figure 1B This is a schematic diagram of an embodiment of an artificial intelligence system provided by an embodiment of the present application;

[0072] Figure 2 This is a schematic diagram of the structure of the server provided in an embodiment of the present application;

[0073] Figure 3 1 is a schematic diagram of an embodiment of a method for constructing a convolutional neural network provided in an embodiment of the present application;

[0074] Figures 4A to 4G Schematic diagram of examples of scenarios at different stages in the process of building a convolutional neural network provided by an embodiment of the present application;

[0075] Figure 5 1 is a schematic diagram of an embodiment of a method for processing three-dimensional data provided by an embodiment of the present application;

[0076] Figure 6 This is an example schematic diagram provided by an embodiment of the present application;

[0077] Figure 7 is another exemplary schematic diagram provided in an embodiment of the present application;

[0078] Figure 8 is a schematic diagram of another embodiment of the method for processing three-dimensional data provided in an embodiment of the present application;

[0079] Figure 9 is a schematic diagram of another embodiment of the method for processing three-dimensional data provided in an embodiment of the present application;

[0080] Figure 10 is a schematic diagram of another embodiment of the method for processing three-dimensional data provided in an embodiment of the present application;

[0081] Figure 11 is another exemplary schematic diagram provided in an embodiment of the present application;

[0082] Figure 12 is another exemplary schematic diagram provided in an embodiment of the present application;

[0083] Figure 13 1 is a schematic diagram of an embodiment of a device for processing three-dimensional data provided in an embodiment of the present application;

[0084] Figure 14 1 is a schematic diagram of an embodiment of a device for processing three-dimensional data provided in an embodiment of the present application;

[0085] Figure 15 1 is a schematic diagram of an embodiment of a device for constructing a convolutional neural network provided in an embodiment of the present application;

[0086] Figure 16 This is a schematic diagram of an embodiment of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0088] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0089] The present invention provides a method for processing three-dimensional data to improve the computational speed of three-dimensional convolution. The present invention also provides a corresponding method for constructing a convolutional neural network, a corresponding device, a system, a computer-readable storage medium, and a computer program product. These are described in detail below.

[0090] The embodiments of the present application involve constructing a convolutional neural network and using the constructed convolutional neural network to process three-dimensional data. For ease of understanding, some vocabulary related to the present application is introduced below.

[0091] The convolutional neural network in this application refers to a three-dimensional convolutional neural network, such as ResNet, ForkNet, etc., which includes a neural network of multiple three-dimensional convolution operators for processing three-dimensional data.

[0092] Three-dimensional data refers to data with three dimensions. It is usually sparse data and can be video data, voice data, or text data.

[0093] Sparse data is data with a large number of repeated elements, which are called the background of the data.

[0094] Sparse data can be categorized into broadly defined sparse data and narrowly defined sparse data. Broadly defined sparse data refers to 3D data with multiple background values. Narrowly defined sparse data refers to 3D data with only one background value. The solution provided in this application is applicable to both broadly defined and narrowly defined sparse data.

[0095] The background value is a numerical value used to describe the background of a repeating element. It can be 0, 1, 2, ... 101, 102, 103, or any other arbitrary value. For example, decimals, negative numbers, or complex numbers can all be used as background values. A 3D data set can include one or more backgrounds, and a background value can be set for each background. For example, when air is used as the background, the background value of the air background is set to 101; when the surface of an object is used as the background, the background value of the physical surface background is set to 102; when the background is an area not detected by the depth camera, the background value of the background of the undetected area is set to 102.

[0096] Sparse distribution refers to the background distribution of 3D data. Background distribution can be represented by the background identifier of each voxel. The background identifier can be a numerical value or character that numbers the background, such as 1, 2, 3 or A, B, C, etc. The background identifier is associated with the background value, for example, background identifier 1 is associated with background value 101, background identifier 2 is associated with background value 102, and background identifier 3 is associated with background value 103.

[0097] Three-dimensional data includes voxels, which is the abbreviation of volume element and indicates the smallest unit of three-dimensional data in spatial segmentation, similar to pixels in two-dimensional data.

[0098] Convolution involves multiplying a value by each position of the convolution kernel of the convolution operator, and then accumulating the results to obtain the convolution result of that value. This value can be a background value or a non-background value. If only the product is multiplied without accumulating, multiple intermediate results will be obtained. The number of intermediate results corresponds to the size of the convolution kernel.

[0099] In a 3D convolution operator, the convolution kernel typically takes the form of a multidimensional array. The kernel size is typically predefined, for example, 3*3*1*L, where the first 3 represents width, the second 3 represents height, 1 represents depth, and L represents the number of channel dimensions. When performing convolution on 3D video data, the channel dimension typically represents color, though other definitions are possible. The convolution kernel acts like a sliding window over the 3D data along a channel dimension.

[0100] If the convolution kernel has multiple channel dimensions, when calculating the intermediate background value, for each of the nine positions in the 3*3*1 matrix, the background value is multiplied by the vector at the corresponding position in the L channel dimensions. These products at that position in the L channel dimensions are then accumulated to obtain the intermediate result at that position. If these nine intermediate results are accumulated again, the convolution result is obtained, which is usually a vector.

[0101] A sliding window of three-dimensional data refers to a window that can slide across multiple voxels of three-dimensional data. The window can cover multiple voxels at a time. For example, if the size of the sliding window is a 3*3 window, then the sliding window can cover 9 voxels in one sliding, which can also be called the sliding window covering 9 voxels.

[0102] The artificial intelligence (AI) system provided in the embodiment of the present application includes a scheme for constructing a convolutional neural network and a scheme for processing three-dimensional data, which can be implemented in different devices. The scheme for constructing a convolutional neural network can be performed offline. After obtaining an applicable target convolutional neural network, the target convolutional neural network can be sent to a terminal device or other server in the form of a file, and the terminal device or other server uses the target convolutional neural network to execute the process of processing three-dimensional data. Figure 1A and Figure 1B The architectural diagram of the artificial system is shown for introduction.

[0103] like Figure 1A As shown, the first server obtains an initial convolutional neural network and then executes the network construction process to obtain the target convolutional neural network. This initial convolutional neural network can be pre-configured by the developer based on the application scenario. For example, the initial convolutional neural network configured for autonomous driving scenarios may differ from the initial convolutional neural networks configured for other scenarios, but the construction principles are the same. The process from the initial convolutional neural network to the target convolutional neural network will be described later.

[0104] After the first server constructs the target convolutional neural network, it can send the target convolutional neural network to the corresponding terminal device, which uses the target convolutional neural network to process the three-dimensional data to be processed by the terminal device. The three-dimensional data can be directly collected by the terminal device, or the terminal device can collect two-dimensional data and then perform depth expansion on the two-dimensional data.

[0105] It can also be Figure 1B As shown, after the first server builds the target convolutional neural network, it can send the target convolutional neural network to the second server. The terminal device sends the three-dimensional data to be processed to the second server, and the second server processes the three-dimensional data sent by the terminal device. After the processing is completed, the second server returns the processing results to the terminal device.

[0106] The solution for processing three-dimensional data provided in the embodiments of the present application can be applied to multiple fields of artificial intelligence (AI), such as: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0107] The first server and the second server may be physical machines, or virtual machines (VMs) or containers. VMs or containers may be virtualized devices that are partitioned from the hardware resources of a physical machine using a virtualized approach.

[0108] A terminal device (also known as user equipment (UE)) is a device with wireless transceiver capabilities that can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; on water (such as ships); or in the air (for example, on airplanes, balloons, and satellites). The terminal can be a mobile phone, a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal used in industrial control, a wireless terminal used in self-driving cars, a wireless terminal used in remote medical care, a wireless terminal used in smart grids, a wireless terminal used in transportation safety, a wireless terminal used in smart cities, or a wireless terminal used in smart homes.

[0109] In this application, if Figure 2 As shown, the above-mentioned first server may include a hardware layer and a software layer. The hardware layer may include a memory and one or more processors. These processors may include a central processing unit (CPU), a graphics processing unit (GPU) or a dedicated chip. The dedicated chip refers to a chip specifically used to build a network. The dedicated chip may be an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), etc.

[0110] The software layer may include source code and an AI compiler. The source code includes an initial convolutional neural network and a description of the three-dimensional data in the scenario in which the target convolutional neural network constructed by the initial convolutional neural network is applied. For example, in an autonomous driving scenario, the size and type of the three-dimensional image data or video data input to the target convolutional neural network are defined. The source code also includes background values ​​included in the three-dimensional data input to the target convolutional neural network, such as the three background values ​​101, 102, and 103.

[0111] The AI ​​compiler includes background parameter parsing logic and sparse compilation optimization logic. The AI ​​compiler will execute the parameter parsing logic and sparse compilation optimization logic according to the content in the source code to obtain the target convolutional neural network.

[0112] Figure 2 The structure of the first server for constructing a convolutional neural network is introduced. The method for constructing a convolutional neural network provided in an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0113] Figure 3 Schematic diagram of an embodiment of a method for constructing a convolutional neural network in an embodiment of the present application.

[0114] like Figure 3 As shown, an embodiment of the method for constructing a convolutional neural network provided in an embodiment of the present application includes:

[0115] 101. Obtain an initial convolutional neural network.

[0116] The initial convolutional neural network includes an operation subnetwork, which includes at least one first convolution operator.

[0117] The operation subnetwork can be the entire initial convolutional neural network or a part of the initial convolutional neural network.

[0118] The first convolution operator is a three-dimensional convolution operator. These three-dimensional convolution operators and other operators are arranged according to a certain network structure to perform processing on three-dimensional data.

[0119] After executing step 101, you can execute step 102 or step 103.

[0120] 102. According to the operation subnet, copy the sparse distribution inference subnet.

[0121] The network structure of the sparse distribution inference subnet is the same as that of the computation subnet. The sparse distribution inference subnet is used to infer a sparse distribution graph of three-dimensional data.

[0122] After step 102 , step 104 is executed.

[0123] 103. According to the operation subnet, copy the background derivation subnet and the sparse distribution derivation subnet.

[0124] The network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as those of the computation subnet. The background derivation subnet is used to derive the background value of the 3D data, while the sparse distribution derivation subnet is used to derive the sparse distribution map of the 3D data.

[0125] After step 103 , step 105 is executed.

[0126] 104. Construct at least one of the first input relationship and the second input relationship to obtain a target convolutional neural network.

[0127] The first input relationship is the input relationship from the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position, and the second input relationship is the input relationship from the operator that inputs data to the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position, so as to obtain the target convolutional neural network.

[0128] 105. Construct at least one of the first input relationship and the second input relationship, and construct at least one of the third input relationship and the fourth input relationship to obtain a target convolutional neural network.

[0129] The first input relationship is the input relationship from the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position, and the second input relationship is the input relationship from the operator that inputs data to the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position.

[0130] Construct at least one of a third input relationship and a fourth input relationship, where the third input relationship is an input relationship from the third convolution operator in the background deduction subnet to the first convolution operator at the corresponding position, and the fourth input relationship is an input relationship from the operator that inputs data to the third convolution operator in the background deduction subnet to the first convolution operator at the corresponding position, the first input relationship corresponds to the third input relationship, and the second input relationship corresponds to the fourth input relationship.

[0131] In this application, the three-dimensional convolution operator in the operation subnet is called the first convolution operator, the three-dimensional convolution operator in the sparse distribution inference subnet is called the second convolution operator, and the three-dimensional convolution operator in the background inference subnet is called the third convolution operator.

[0132] The target convolutional neural network obtained by combining the above steps 101, 102 and 104 includes an operation subnet and a sparse distribution inference subnet, but does not include a background inference subnet. In this case, the first server also needs to copy a background inference subnet based on the operation subnet, and then input the background value of the three-dimensional data into the background inference subnet for operation. The first background value and the second background value required by the first convolution operator in the operation subnet can be obtained, and then connected to the operation subnet in the form of a data file.

[0133] The target convolutional neural network obtained by combining the above steps 101, 103 and 105 includes an operation subnet, a sparse distribution inference subnet and a background inference subnet.

[0134] The solution provided in the embodiment of the present application adds a background deduction subnet and a sparse distribution deduction subnet of the same structure to the operation subnet on the basis of the initial convolutional neural network used only for processing three-dimensional data. In this way, it can be ensured that in the process of convolution processing of three-dimensional data by convolution operators at different positions in the operation subnet, the background value at the corresponding position can be obtained from the background deduction subnet, and the sparse distribution map at the corresponding position can be obtained from the sparse distribution deduction subnet, thereby ensuring that the target convolutional neural network constructed by the present application can improve the speed of convolution processing.

[0135] Optionally, the above Figure 3 After step 102 of the introduced embodiment, it also includes: determining an operation subnetwork from the initial convolutional neural network, the input data of each operator in the operation subnetwork is sparsely distributed in the same channel dimension, and the output data of each operator is sparsely distributed in the same channel dimension, and the channel dimension represents a dimension different from the three-dimensional space.

[0136] After determining the operation subnet, it can be copied directly according to the determined operation subnet, or the operation subnet can be cut out from the initial convolutional neural network and then copied. After the sparse distribution inference subnet and the background inference subnet are obtained by copying, and the input relationship of the above step 104 or 105 is established between the operators in the three subnets, the remaining network in the initial convolutional neural network except the operation subnet can be spliced ​​to the position before separation from the operation subnet.

[0137] In an embodiment of the present application, after obtaining the initial convolutional neural network, it is necessary to detect the initial convolutional neural network and detect the operation subnet therefrom. The process of detecting the operation subnet can be to start from the first operator of the initial convolutional neural network and detect each operator in the convolutional neural network one by one. If the operators detected continuously are operators on the whitelist and the convolution operator is also on the whitelist, the detection can be continued downward. If an operator is detected that is not an operator on the whitelist, the detection is terminated, and a subnet from the first operator to the last convolution operator before the operator where the detection is terminated is determined, or it can be cut out to form the operation subnet. Among them, the whitelist is pre-configured. For each operator in the whitelist, if the input data is sparsely distributed in the same way in each channel dimension, it can be ensured that the output data of the operator is sparsely distributed in the same way in each channel dimension. The channel dimension can be a color dimension or a dimension representing other features. The same sparse distribution can also be called "same distribution".

[0138] The process of cutting the computational subnet can be found in Figure 4A and Figure 4B For example, Figure 4A As shown in the figure, the initial convolutional neural network includes a placeholder operator (placeholder), a three-dimensional convolution operator (Conv3D) 1, a constant addition operator (BiasAdd) 1, a three-dimensional convolution operator 2, a constant addition operator 2, an accumulation operator (Add), a three-dimensional convolution operator 3, a constant addition operator 3, a reduction summation operator (ReduceSum), and a three-dimensional convolution operator 4. Starting from the placeholder operator, operators are detected one by one. If the operator is detected to be a three-dimensional convolution operator or an operator on the whitelist, the next operator is detected. Figure 4A In the scenario shown, when the reduction summation operator is detected, it is determined that the reduction summation operator is not on the whitelist, and the detection is terminated. Then, the related convolution operators starting from the placeholder operator to the three-dimensional convolution operator 3 are divided into the following Figure 4B In the operator subnetwork shown, the initial convolutional neural network in the operator subnetwork can then be cut out. In this scenario, the reduction and summation operator cannot maintain the same sparse distribution of different channel dimensions of the output data.

[0139] After cutting out the operation subnet from the initial convolutional neural network, copy the background derivation subnet and the sparse distribution derivation subnet according to the operation subnet. After copying the background derivation subnet and the sparse distribution derivation subnet, you can adjust the names of the operators of the background derivation subnet and the sparse distribution derivation subnet. Figure 4CAs shown, the first operator in both the background derivation subnet and the sparse distribution derivation subnet is adjusted to the initialization operator. The 3D convolution operator in the operation subnet is described as "LN_Conv3D", the 3D convolution operator in the background derivation subnet is described as "Conc3D Background", and the 3D convolution operator in the sparse distribution derivation subnet is described as "Conv3D Sparse Distribution". In addition, because the additive constant operator (BiasAdd) in the sparse distribution derivation subnet does not affect the sparse distribution, it can be adjusted to a null operator, which can save some computational effort during 3D data processing.

[0140] Then, input relationships between the operators corresponding to at least one first convolution operator in the background derivation subnet and the sparse distribution derivation subnet and the corresponding first convolution operator are established, respectively. This is the first input relationship, second input relationship, third input relationship, and fourth input relationship in steps 104 and 105 above.

[0141] like Figure 4D As shown, the input relationship between the operator that inputs data to "Conc3D background" in the background derivation subnet and the "LN_Conv3D" corresponding to the "Conc3D background" can be established, which is the fourth input relationship. The connection between the operator that inputs data to "Conv3D sparse distribution" in the sparse distribution derivation subnet and the "LN_Conv3D" corresponding to the "Conv3D sparse distribution" can be established, which is the second input relationship. Figure 4D As shown, the initialization operator in the background derivation subnet is established as an input to "LN_Conv3D1", and the initialization operator in the sparse distribution derivation subnet is established as an input to "LN_Conv3D1". Similarly, the BaisAdd1 in the background derivation subnet is established as an input to "LN_Conv3D2", and the first empty operator in the sparse distribution derivation subnet is established as an input to "LN_Conv3D2". The Add in the background derivation subnet is established as an input to "LN_Conv3D3", and the Add in the sparse distribution derivation subnet is established as an input to "LN_Conv3D3", so as to obtain the following: Figure 4D The target convolutional neural network shown.

[0142] It can also be, such as Figure 4E As shown, the connection between "Conc3D background" and "Conv3D sparse distribution" and the corresponding "LN_Conv3D" is established, that is, the third input relationship and the first input relationship. Figure 4EIn the background derivation subnet, the “Conc3D background 1” in the background derivation subnet is established as an input to “LN_Conv3D1”, and the “Conv3D sparse distribution 1” in the sparse distribution derivation subnet is established as an input to “LN_Conv3D1”. Similarly, the “Conc3D background 2” in the background derivation subnet is established as an input to “LN_Conv3D2”, and the “Conv3D sparse distribution 2” in the sparse distribution derivation subnet is established as an input to “LN_Conv3D2”. The “Conc3D background 3” in the background derivation subnet is established as an input to “LN_Conv3D3”, and the “Conv3D sparse distribution 3” in the sparse distribution derivation subnet is established as an input to “LN_Conv3D3”, so as to obtain the following: Figure 4E The target convolutional neural network shown.

[0143] In this application, you can also Figure 4E On the basis of , the second correspondence and the fourth correspondence are established, and the following can be obtained: Figure 4F The target convolutional neural network shown.

[0144] In addition, the above Figures 4A to 4F The target convolutional neural network described above includes three subnets. The target convolutional neural network obtained by executing steps 101, 102 and 104 does not include the background derivation subnet. Figure 4D 、 Figure 4E and 4F By removing the background derivation subnet but retaining at least one of the third input relation and the fourth input relation, we can obtain the corresponding three scene diagrams of the target convolutional neural network that only includes the computation subnet and the sparse background derivation subnet. Figure 4F Taking the corresponding scenario as an example, we can get the following Figure 4G The target convolutional neural network is shown. The background values ​​for the third and fourth input relationships can be input to the corresponding "LN_Conv3D" via a file. The background values ​​for the third and fourth input relationships can be stored in a single memory block or in different registers. When different "LN_Conv3D"s need background values, they can be read from the memory or registers.

[0145] After the target convolutional neural network is constructed, it can be converted into a binary code file suitable for the terminal device or the second server and then sent to the corresponding terminal device or the second server. Alternatively, it can be sent after the corresponding terminal device or the second server sends a load request. Alternatively, the target convolutional neural network can be sent directly without binary code conversion.

[0146] The above embodiments introduce the method of constructing a convolutional neural network provided by the embodiments of the present application. The following describes the method of processing three-dimensional data provided by the embodiments of the present application in conjunction with the accompanying drawings.

[0147] The method for processing three-dimensional data provided in an embodiment of the present application can be implemented through three schemes, namely: 1. inputting first data, at least one first background value, and a first sparse distribution map into the target operator of a convolutional neural network; 2. inputting first data, at least one second background value, and a second sparse distribution map into the target operator of a convolutional neural network; 3. inputting first data, at least one first background value, at least one second background value, a first sparse distribution map, and a second sparse distribution map into the target operator of a convolutional neural network. Regardless of which scheme is adopted, the target convolution operator can output the second data, but there is a slight difference in speed, which is introduced below.

[0148] In addition, it should be noted that a "convolutional neural network" is used in the embodiment of the method for processing three-dimensional data. The "convolutional neural network" is the target convolutional neural network constructed in the above embodiment.

[0149] 1. Input first data, at least one first background value, and a first sparse distribution map into a target operator of a convolutional neural network.

[0150] like Figure 5 As shown, an embodiment of the method for processing three-dimensional data provided by the embodiment of the present application includes:

[0151] 201. Input first data, at least one first background value of the first data, and a first sparse distribution map of the first data into a target convolution operator of a convolutional neural network.

[0152] In which, the convolutional neural network includes at least one first convolution operator, the target convolution operator is any one of the at least one first convolution operator, the first data includes multiple first voxels, at least one first background value indicates at least one background in the first data, at least one first background value corresponds one-to-one to at least one background, and the first sparse distribution map describes the sparse distribution of multiple first voxels through background identifiers and non-background identifiers, the background identifier indicates that the first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicates that the first voxel corresponding to the non-background identifier in the first data is non-background.

[0153] In the present application, the first data is a multi-dimensional array (tensor) of video data, voice data or text data.

[0154] There is at least one first background value, that is, one or more. "Multiple" in this application includes two or more, and can also be described as at least two. The first background value represents the background value in the first data, such as: 0, 1, 2, 101, 102, or 103, etc., and is not limited to positive, negative, decimal, or complex numbers. The first background value can be randomly selected, and this is not limited in this application.

[0155] The voxels in the first data are called first voxels. Each first voxel will have a value, and the value of the first voxel can be a vector. If the value of the first voxel indicates the background, it means that the first voxel is a voxel indicating the background. If the value of the first voxel indicates the non-background, it means that the first voxel is a voxel indicating the non-background. There will be many repeated background values ​​of the first data, such as: the first data includes 1000 first voxels, 950 of these 1000 are voxels indicating the background, and these 950 first voxels may have only three background values, such as: 101, 102 and 103. Of course, the three background values ​​101, 102 and 103 are just examples, and the first data may also involve more background values ​​or fewer background values. The first background value of the first data in this application refers to 101, 102, and 103 in this example, indicating that there are three types of background, and each type of background corresponds to a background value.

[0156] The first sparse distribution graph structure corresponds to the structure of the first data. Figure 6 In the example shown, if the structure of the first data is a 5*5*1 structure, then the first sparse distribution map is a 5*5 sparse distribution map. A unit in the first data is a first voxel, and a unit in the first sparse distribution map can be called a "point". Among them, the point corresponding to the first voxel indicating the background can be called a "background point", and the point corresponding to the first voxel indicating the non-background can be called a "non-background point". Non-background points can also be called "dense points". Background points are marked with background identifiers, such as: 1, 2, 3, and non-background points are marked with a unified non-background identifier.

[0157] The background identifier corresponds to the first background value, for example: Figure 6 In the example shown, there are three first background values, 101, 102, and 103, and background identifiers, 1, 2, and 3. The mapping can be as follows: 101->1, 102->2, and 103->3. The mapping between first background values ​​and background identifiers is typically implicit based on their order, but can also be displayed via a table or mapping. Non-background identifiers are uniformly denoted by M, which is the abbreviation for Max and indicates a non-background point.

[0158] 202. According to the background identifier of the background point corresponding to the first voxel indicating the background in the first data on the first sparse distribution map, search for an intermediate result corresponding to the background identifier of the background point from the corresponding relationship.

[0159] The correspondence indicates the relationship between the intermediate results of at least one first background value and the background identifier corresponding to at least one first background value. The intermediate result of one of the at least one first background values ​​is calculated by the convolution kernel of the first background value and the target convolution operator, and the intermediate result corresponding to the background identifier of the background point is used to determine the value of the second voxel in the second data.

[0160] Before using this correspondence to search for intermediate results, it is usually necessary to first establish and store this correspondence. The meaning of the convolution kernel can be understood by referring to the previous introduction.

[0161] 203. Control the target convolution operator to output second data, where the second data includes a plurality of second voxels.

[0162] The solution provided in the embodiment of the present application calculates the intermediate results of each first background value and the convolution kernel before searching for the intermediate results from the first correspondence, such as: 101 and the intermediate result 1 of the convolution kernel, 102 and the intermediate result 2 of the convolution kernel, and 103 and the intermediate result 3 of the convolution kernel. Because the first background value corresponds to the background identifier, a correspondence between the background identifier and the convolution result can be established, such as: 1->intermediate result 1, 2->intermediate result 2 and 3->intermediate result 3. Usually, the first data includes many first voxels. The process of convolution processing the first data in the prior art is the process of calculating the values ​​of these first voxels one by one with the convolution kernel. Among these first voxels, the voxels indicating the background usually account for the vast majority, and the values ​​of these voxels indicating the background are repeated in large quantities. Calculating them one by one with the convolution kernel is very slow. In this application, a calculation is performed for each first background value before the convolution kernel is applied. After obtaining an intermediate result, the correspondence between the intermediate result and the background identifier is stored. When the first data is convolved, the intermediate result can be directly found by looking up the correspondence for the first voxel indicating the background in the first data. The intermediate result is then used to obtain the value of the second voxel in the second data. This eliminates the need for extensive calculations for each first voxel value and the convolution kernel, thereby saving computational overhead and achieving convolution acceleration.

[0163] Optionally, the intermediate results in the above correspondence are results obtained by calculating the first background value at different positions of the convolution kernel. Thus, the above step 102 includes: for each first voxel indicating the background in the sliding window of the first data, based on the background identifier corresponding to each first voxel indicating the background in the first sparse distribution map and the position of each first voxel indicating the background in the sliding window, searching from the correspondence for the intermediate result corresponding to the background identifier at the corresponding position, wherein the sliding window of the first data is the same size as the convolution kernel of the target convolution operator.

[0164] In the embodiment of the present application, after inputting at least one first background value into the target convolution operator, each first background value can be used to perform calculations with the convolution kernel. Taking a 3*3*1*4 convolution kernel as an example, it means that the convolution kernel has 4 channel dimensions, 9 positions in one channel dimension, and the positions can be shared by other channel dimensions. The vectors of these 9 positions in the first channel dimension are represented as shown in Table 1 below:

[0165] Table 1: Vectors of the 9 positions on the first channel dimension of the convolution kernel

[0166] A1 B1 C1 D1 E1 F1 G1 H1 I1

[0167] If the first background value is 101, then the product with the vector of 9 positions in the first channel dimension can be obtained as shown in Table 2:

[0168] Table 2: Vectors of the 9 positions on the first channel dimension of the convolution kernel

[0169] 101A1 101B1 101C1 101D1 101E1 101F1 101G1 101H1 101I1

[0170] If the vectors of the 9 positions in the second channel dimension are represented by A2, B2, C2, D2, E2, F2, G2, H2 and I2, then the first background value is 101 multiplied by the vectors of the 9 positions to obtain 101A2, 101B2, 101C2, 101D2, 101E2, 101F2, 101G2, 101H2 and 101I2.

[0171] Similarly, the vectors at the 9 positions in the third channel dimension are represented by A3, B3, C3, D3, E3, F3, G3, H3 and I3. The first background value is the product of 101 and the vectors at the 9 positions, which can be obtained as 101A3, 101B3, 101C3, 101D3, 101E3, 101F3, 101G3, 101H3 and 101I3.

[0172] The vectors at the 9 positions in the third channel dimension are represented by A4, B4, C4, D4, E4, F4, G4, H4 and I4. The first background value is 101 and the product of the vectors at the 9 positions to obtain 101A4, 101B4, 101C4, 101D4, 101E4, 101F4, 101G4, 101H4 and 101I4.

[0173] Among them, A1, A2, A3 and A4 are vectors of different dimensions at the same position. Then the intermediate result at the position where A1 is located is 101A1+101A2+101A3+104A4. Similarly, by accumulating the product of the first background value and the vector at other positions, we can obtain the intermediate results at different positions with the first background value of 101 as shown in Table 3 below.

[0174] Table 3: Intermediate results calculated with the first background value 101 and the convolution kernel at different positions

[0175] 101A1+101A2+101A3+101A4 101B1+101B2+101B3+101B4 101C1+101C2+101C3+101C4 101D1+101D2+101D3+101D4 101E1+101E2+101E3+101E4 101F1+101F2+101F3+101F4 101G1+101G2+101G3+101G4 101H1+101H2+101H3+101H4 101I1+101I2+101I3+101I4

[0176] Similarly, if the first background value is 102, 103 or other values, the intermediate results corresponding to the value at different positions of the convolution kernel can be obtained in the above manner.

[0177] The background identifier corresponding to the first background value 101 is represented by “1”, so the corresponding relationship is the corresponding relationship between “1” and the above Table 3.

[0178] In the process of obtaining the value of the second voxel by sliding the window on the first data, because the sliding window of the first data is the same size as the convolution kernel, the sliding window is usually two-dimensional or three-dimensional. Since the convolution kernel is 3*3*1*4, the sliding window can be expressed as 3*3 or 3*3*1. The sliding window of the first data will cover 9 first voxels in one sliding. If the 9 first voxels in the sliding window are all background values, as shown in Table 4:

[0179] Table 4: First background values ​​of the 9 first voxels in the sliding window of the first data

[0180] 101 102 103 101 102 102 101 101 101

[0181] Among them, 101 appears in 5 positions: the first row and the first column, the second row and the first column, the third row and the first column, the third row and the second column, and the third row and the third column. Then we need to get the intermediate results of these positions from the correspondence between 1 and Table 3, which are: 101A1+101A2+101A3+101A4, 101D1+101D2+101D3+101D4, 101G1+101G2+101G3+101G4, 101H1+101H2+101H3+101H4 and 101I1+101 Similarly, from the table of intermediate results between 2 and the first background value 102, three intermediate results can be obtained: 102B1+102B2+102B3+102B4, 102E1+102E2+102E3+102E4, and 102F1+102F2+102F3+102F4. From the table of intermediate results between 3 and the first background value 103, one intermediate result: 103C1+103C2+103C3+103C4 can be obtained. Then, by accumulating these nine intermediate results, the value of one second voxel corresponding to each of the nine first voxels in Table 4 can be obtained.

[0182] The nine first voxels in the sliding window of the first data described above are all background voxels. If the sliding window of the first data includes a first voxel that indicates non-background, the calculation process can be understood by referring to the following process. As shown in Table 5, eight first voxels indicate background and one first voxel indicates non-background.

[0183] Table 5: First background values ​​of the 9 first voxels in the sliding window of the first data

[0184] 101 102 103 101 102 500 101 101 101

[0185] In this case, for the 8 first voxels indicating the background, you can refer to the scheme for the above 9 first voxels, and obtain the intermediate results corresponding to the background identification of these 8 first voxels respectively. Then, for the first voxels indicating the non-background on the second row and third column, you can use 500 to multiply the vectors on the four channel dimensions of the 9 positions of the convolution kernel, and then add the results of the same position but different channel dimensions to obtain an intermediate result similar to Table 3. From it, find the intermediate results on the second row and third column, such as: 500F1+500F2+500F3+500F4, and then add the other 8 intermediate results with 500F1+500F2+500F3+500F4 to obtain the value of a second voxel corresponding to the 9 first voxels in Table 5.

[0186] The process of determining the corresponding intermediate result based on the first background value, and the correspondence between the background identifier and the intermediate result described above can also be called the process of building a table in the target convolution operator. For the 9 voxels in the sliding window of the first data, the process of looking up the table based on the background identifier in the above embodiment only lists a scenario of one channel dimension. The first data is usually multi-channel dimensional. The first background on each channel dimension may be different, but no matter how many channel dimensions there are, the sparse distribution of the sub-data on each channel dimension is the same, so these channel dimensions share a first sparse distribution map. However, for different channel dimensions, the background values ​​mapped by the background identifier on the first sparse distribution map are different.

[0187] like Figure 7 As shown, the first data of the four channel dimensions described above has three first background values ​​in the first channel dimension, namely 101, 102, and 103; three first background values ​​in the second channel dimension, namely 201, 202, and 203; three first background values ​​in the third channel dimension, namely 301, 302, and 303; and three first background values ​​in the fourth channel dimension, namely 401, 402, and 403. These four channel dimensions share a first sparse distribution map. Thus, in different channel dimensions, the same background identifier corresponds to intermediate results of different first background values. For example, background identifier 1 corresponds to intermediate results of 101, 201, 301, and 401; background identifier 2 corresponds to intermediate results of 102, 202, 302, and 402; and background identifier 3 corresponds to intermediate results of 103, 203, 303, and 403. In this way, when calculating the value of the second voxel of sub-data of different channel dimensions, the above calculation process for the first channel dimension can be used to find the corresponding intermediate result by looking up the table, and then calculate the value of the second voxel. There is no need to use the value of the 9 voxels in the sliding window and the convolution kernel for multiplication each time, and then accumulate them, thereby achieving convolution acceleration in each channel dimension.

[0188] The embodiment of constructing a convolutional neural network as described above, and Figures 4A to 4GFrom the example, it can be seen that in the scheme of inputting at least one first background value and a first sparse distribution map, the convolutional neural network includes: an operation subnet, a background deduction subnet and a sparse distribution deduction subnet. The network structures of the background deduction subnet and the sparse distribution deduction subnet are the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from the operator that inputs the first sparse distribution map to the second convolution operator, and the position of the second convolution operator in the sparse distribution deduction subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value comes from the operator that inputs at least one first background value to the third convolution operator, and the position of the third convolution operator in the background deduction subnet corresponds to the position of the target convolution operator in the operation subnet. The structure including three subnets can be referred to. Figure 4D Understand.

[0189] In addition, it can also be: the convolutional neural network includes an operation subnet and a sparse distribution derivation subnet, the network structure of the sparse distribution derivation subnet is the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from the operator that inputs the first sparse distribution map to the second convolution operator, and the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value is pre-configured for the target convolution operator. Although this structure is not shown in the figure, you can refer to Figure 4D ,exist Figure 4D Based on this, we delete the background derivation subnetwork to understand it.

[0190] The relationship between the operation subnet, background inference subnet and sparse distribution inference subnet can be understood by referring to the above process of building a convolutional neural network. Figure 8 Introduce the process of using this convolutional neural network to process three-dimensional data.

[0191] like Figure 8As shown, the initial data is input into the first operator (placeholder operator) of the operation subnet, at least one background value of the initial data is input into the first operator (initialization operator) of the background derivation subnet, and the initial data and at least one background value of the initial data are input into the first operator (initialization operator) of the sparse distribution derivation subnet. The initial data is three-dimensional data. For the first convolution operator in the operation subnet as the target convolution operator, the initial data is the first data of the first convolution operator, and the at least one background value of the initial data is the at least one first background value of the first data. In this case, the first data is three-dimensional data of one channel dimension. At least one first background value is represented in the form of a one-dimensional array, such as: {101, 102, 103}. If the target convolution operator is not the first convolution operator in the operation subnet, the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data. This process can be referred to. Figure 7 In this case, at least one first background value of the first data is represented in the form of a two-dimensional array. Figure 7 The first background value of the four channel dimensions in the example can be expressed as:

[0192]

[0193] After the initialization operator in the background derivation subnet receives at least one background value of the initial data in the form of a one-dimensional array, it does not process the at least one background value of the initial data, but directly inputs the at least one background value of the initial data as the at least one first background value of the first convolution operator in the operation subnet to Conv3D background 1 in the background derivation subnet and LN_Conv3D1 in the operation subnet.

[0194] After the initialization operator in the sparse distribution deduction subnet receives the initial data equivalent to the first data and at least one first background value equivalent to at least one background value of the initial data, each first voxel in the first data is scanned one by one, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background; for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

[0195] The process can be found in the above Figure 6 To understand, as mentioned above Figure 6As shown, taking a 5*5 three-dimensional data as an example, the first data includes 25 first voxels, each of which has its corresponding background value. In this example, the first background value in the form of a one-dimensional array can be expressed as {101, 102, 103}. By scanning each first voxel, the background value 101 can be marked with 1 in the sparse distribution map, the background value 102 can be marked with 2 in the sparse distribution map, and the background value 103 can be marked with 3 in the sparse distribution map. In this way, according to the positions of the 25 first voxels, Figure 6 Each "grid" in the target sparse distribution map is called a point, corresponding to a first voxel, and the background identifier in the grid corresponds to the first background value of the first voxel at the corresponding position.

[0196] The initialization operator in the sparse distribution derivation subnet can output the obtained first sparse distribution map to LN_Conv3D1 in the operation subnet, and can also pass the target sparse distribution map as the first sparse distribution map to Conv3D Sparse Distribution 1 in the sparse distribution derivation subnet.

[0197] like Figure 8 As shown in , for different convolution operators (LN_Conv3D1, LN_Conv3D2, and LN_Conv3D3) in the operation subnetwork, the input data is called the first data, the input background value is called the first background value, and the input sparse distribution map is called the first sparse distribution map, but these inputs are different for different convolution operators.

[0198] After receiving the first data, at least one first background value, and the first sparse distribution map, LN_Conv3D1, LN_Conv3D2, or LN_Conv3D3 processes the first data according to the table building and table lookup procedures described in the above embodiment, thereby obtaining the value of the second voxel of the second data.

[0199] The above describes a solution for inputting first data, at least one first background value of the first data, and a first sparse distribution map into a target convolution operator. The above solution can achieve convolution acceleration by table lookup during the processing of three-dimensional data.

[0200] 2. Input the first data, at least one second background value, and the second sparse distribution map into the target operator of the convolutional neural network.

[0201] like Figure 9 As shown, another embodiment of the method for processing three-dimensional data provided by the embodiment of the present application includes:

[0202] 301. Input the first data into a target convolution operator of a convolutional neural network.

[0203] A convolutional neural network includes at least one first convolution operator, a target convolution operator is any one of the at least one first convolution operator, the first data includes multiple first voxels, and the target convolution operator further inputs: at least one second background value obtained by convolution processing on at least one background value of the first data, and a second sparse distribution map obtained by performing a row identification check operation on the first sparse distribution map, the identification check operation is to check whether the identifications of points covered by the sliding window of the first sparse distribution map are all the same or not all the same, the identification includes a background identification and a non-background identification, at least one first background value indicates at least one background in the first data, and at least one first background value corresponds one-to-one to at least one background, the first sparse distribution map describes the sparse distribution of the multiple first voxels through the background identification and the non-background identification, the background identification indicates that the first voxel corresponding to the background identification in the first data is background, and the non-background identification indicates that the first voxel corresponding to the non-background identification in the first data is non-background.

[0204] This embodiment can be combined with Figure 10 The process of obtaining the second background value from the first background value can be obtained by convolving the first background value with the convolution kernel of the second convolution operator. The above process can refer to the process of obtaining the intermediate result from the first background value and the convolution kernel to understand the first half. Then, you only need to accumulate the 9 intermediate results to obtain the second background value.

[0205] For the process from the first sparse distribution map to the second sparse distribution map, please refer to Figure 11 To understand. The first sparse distribution map input to Conv3D sparse distribution 1 can be convolved through a 3*3 sliding window, that is, the 3*3 sliding window slides from left to right and from top to bottom on the first sparse distribution map according to a certain step size, such as sliding grid by grid. For the grids on the left, right, top and bottom, virtual rows and virtual columns can be added in a virtual extension (pending) manner. The values ​​of the background identifiers in the virtual rows and virtual columns can be marked with identifier 1. In this way, the background identifiers in the 9 grids in the sliding window are checked each time. If they are all the same, such as: all 1, then the background identifier of the point where the sliding window is located this time is 1. If the background identifiers in the 9 grids in the sliding window are not all the same, then the identifier of the point where the sliding window is located this time is set to M, indicating that the position is a non-background point. Sliding in sequence, the second sparse distribution map can be obtained.

[0206] 302. According to the background identifier representing the background point on the second sparse distribution map, search for a second background value corresponding to the background identifier of the background point.

[0207] The second background value corresponding to the background identifier of the background point is the value of the second voxel corresponding to the background point in the second data.

[0208] The second sparse distribution map and the second background value belong to the second data to be output by the target convolution operator. Therefore, when the second sparse distribution map and the second background value are known, the corresponding second background value can be directly found according to the background identifier in the second sparse distribution map.

[0209] Regarding the search scheme: If the operator input by the second data of the target convolution operator is a unary operator, that is, the operator has only one input from the target convolution operator, then you can also skip searching for the second background value, and it is assumed that the second background value has been stored in the corresponding position.

[0210] If the operator input by the second data of the target convolution operator is a binary operator, that is, the binary operator has inputs from other operators in addition to the input from the target convolution operator, in this case, it is necessary to find the second background value.

[0211] 303. Control the target convolution operator to output second data, where the second data includes a plurality of second voxels.

[0212] In the solution provided by the embodiment of the present application, after the target convolution operator receives the first data, the second background value, and the second sparse distribution map, it can directly search for the corresponding second background value using the background identifier in the second sparse distribution map. The second background value found is the value of the second voxel at the position corresponding to the background identifier. It can be seen that the solution provided by the embodiment of the present application can directly search for the corresponding value for the second voxel indicated by the background identifier in the second sparse distribution map, eliminating the need for further convolution processing, thereby achieving convolution acceleration.

[0213] In the above Figure 9 On the basis of the corresponding embodiment, at least one first background value and a first sparse distribution map may be added to the target convolution operator as input, which is the following solution 3.

[0214] 3. Input the first data, at least one first background value, at least one second background value, the first sparse distribution map, and the second sparse distribution map into the target operator of the convolutional neural network.

[0215] This solution is equivalent to combining the above-mentioned solution 1 and solution 2. The difference is that in terms of the search order, the second background value corresponding to the background identifier of the background point is first searched based on the background identifier representing the background point on the second sparse distribution map. For the non-background point on the second sparse distribution map, the intermediate result corresponding to the background identifier of the target point is searched from the correspondence relationship based on the background identifier of the target point on the first sparse distribution map. The target point is the non-background point on the second sparse distribution map positioned to the corresponding background-indicating point on the first sparse distribution map. The correspondence relationship indicates the relationship between the intermediate result of at least one first background value and the background identifier corresponding to at least one first background value. The intermediate result of one of the at least one first background values ​​is calculated by the convolution kernel of the first background value and the target convolution operator. The intermediate result in the corresponding relationship is the result obtained by calculating the first background value at different positions of the convolution kernel. The intermediate result corresponding to the background identifier of the target point is used to determine the value of the second voxel corresponding to the non-background point on the second sparse distribution map.

[0216] That is, for a non-background point on the second sparse distribution map, locate a sliding window on the first sparse distribution map. The 9 points in the sliding window can be called target points. Then, the 9 points in the sliding window are used to obtain the value of the second voxel of a non-background point on the second sparse distribution map. Figure 12 As shown, for the M in the second row and second column of the second sparse distribution map, a sliding window can be corresponding to it on the first sparse distribution map. The sliding window includes 9 points, all of which are background points. The background identifiers are 7 1s and 2 2s. Then, the table lookup process described in the above solution 1 can be performed based on these 9 background identifiers and their positions in the sliding window. The intermediate results found can be accumulated to obtain the value of the second voxel corresponding to the non-background point on the second sparse distribution map. If the sliding window of the first sparse distribution map includes non-background identifiers, the intermediate results can be found and accumulated for the background identifier portion. For the points with non-background identifiers, the value of the first voxel corresponding to the point with non-background identifiers can be calculated by referring to the convolution kernel of the target convolution operator described in the above solution 1. After obtaining the intermediate result, it can be accumulated with the intermediate result of the background identifier point on the first sparse distribution map, thereby obtaining the value of the second voxel corresponding to the non-background point on the second sparse distribution map.

[0217] The convolutional neural network of solution 3 may include: an operation subnet, a background derivation subnet, and a sparse distribution derivation subnet. The network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to the second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identification check operation on the first sparse distribution map, and the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value comes from an operator that inputs at least one first background value to the third convolution operator, and at least one second background value is obtained by the third convolution operator performing convolution processing on at least one first background value, and the position of the third convolution operator in the background derivation subnet corresponds to the position of the target convolution operator in the operation subnet. This structure can be referred to. Figure 4F Understand.

[0218] The convolutional neural network of solution 3 can also be: the convolutional neural network includes: an operation subnet and a sparse distribution deduction subnet, the network structure of the sparse distribution deduction subnet is the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from the operator that inputs the first sparse distribution map to the second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identification check operation on the first sparse distribution map, the position of the second convolution operator in the sparse distribution deduction subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value and at least one second background value are pre-configured for the target convolution operator. This structure can be referred to Figure 4G Understand.

[0219] In combination Figure 4F and Figure 4G In terms of structure, if the target convolution operator is the first convolution operator in the operation subnet, the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written in the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background, and for the first voxel indicating the non-background, the non-background identifier is written in the point in the blank bitmap corresponding to the first voxel indicating the non-background.

[0220] This process is the same as the corresponding process in Scheme 1. In addition, the forms of at least one first background value, at least one second background value, first data and second data can be understood by referring to the introduction of Scheme 1 and Scheme 2 above, and will not be repeated here.

[0221] In the solution described in Solution 3, compared with Solutions 1 and 3, two searches can be performed, and the acceleration effect of convolution is further improved.

[0222] In the solution provided above in the embodiment of the present application, the operation subnet, the background derivation subnet and the sparse distribution derivation subnet cooperate with each other, and different functions can be completed by different subnets, such as: the background derivation subnet is used to complete the derivation of background values ​​at different stages, and the sparse distribution derivation subnet is used to complete the derivation of background sparse distribution at different stages. Combined with the operation subnet, the background value of the corresponding stage can be combined to complete the preliminary convolution processing at different stages of processing three-dimensional data. When processing the second voxel, the background identifier of the corresponding stage can be combined to find the corresponding convolution result. It can be seen that the cooperation of the three subnets can efficiently perform convolution processing, thereby improving the speed of convolution processing.

[0223] To illustrate the effect of this application, the following is a set of experimental data. The single-core performance data of the ForkNet convolutional network tested on an ArmV8.2 architecture server in this application embodiment is shown in Table 6.

[0224] ForkNet is a convolutional network for 3D semantic segmentation and 3D reconstruction. Its input data is a three-dimensional array of voxel values ​​in a storage space. Each point in the three-dimensional array has three values, corresponding to air (background 0), the surface of the object (background 1), and the area not detected by the depth camera (background 2). The 12 convolution operators in ForkNet can be sparsely accelerated, and the acceleration effects are listed row by row in Table 3. After each layer of convolution, the background and sparse distribution will change, but the number of backgrounds is always 3. Table 3 lists the proportion of the three backgrounds in the input data of each convolution, and "density" refers to the proportion of non-background points. The "Performance" column in the table is a statistic of the performance data, where the "This Application" column is the time consumed after using the solution provided in the embodiment of the present application, the "Dense" column is the time consumed when using the dense Conv3D operator, and the "Time Ratio" column is the "This Application" column divided by the "Dense" column.

[0225] Table 6: Single-core performance statistics of convolution operators in ForkNet convolutional networks

[0226]

[0227] As shown in Table 6, in the ForkNet network, the generalized sparsity of the input data is effectively utilized, reducing the total time consumed by the 12 convolution operators to 70.56% of the time consumed by the dense Conv3D operator. The sparse background of the input data is not equal to 0 and is not unique, indicating that the present invention fully utilizes generalized sparsity.

[0228] This application makes full use of multi-background data features to accelerate multiple convolution operators, thereby covering generalized sparse scenarios that cannot be covered by narrow sparse optimization methods, and achieving the effect of accelerating the entire network.

[0229] The above introduces the method for constructing a convolutional neural network and the method for processing three-dimensional data provided in the embodiments of the present application. The following introduces the device provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0230] like Figure 13 As shown, an embodiment of the apparatus 40 for processing three-dimensional data provided in the embodiment of the present application includes:

[0231] The first processing unit 401 is used to input the first data, at least one first background value of the first data, and the first sparse distribution map of the first data into a target convolution operator of a convolutional neural network, wherein the convolutional neural network includes at least one first convolution operator, the target convolution operator is any one of the at least one first convolution operator, the first data includes multiple first voxels, the at least one first background value indicates at least one background in the first data, the at least one first background value corresponds one-to-one to the at least one background, and the first sparse distribution map describes the sparse distribution of the multiple first voxels through background identifiers and non-background identifiers, the background identifier indicates that the first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicates that the first voxel corresponding to the non-background identifier in the first data is non-background.

[0232] The second processing unit 402 searches for an intermediate result corresponding to the background identifier of the background point from a correspondence relationship based on the background identifier of the background point on the first sparse distribution map corresponding to the first voxel indicating the background in the first data input by the first processing unit 401, wherein the correspondence relationship indicates a relationship between the intermediate result of at least one first background value and the background identifier corresponding to at least one first background value, wherein the intermediate result of one of the at least one first background values ​​is calculated by the convolution kernel of the first background value and the target convolution operator, and the intermediate result corresponding to the background identifier of the background point is used to determine the value of the second voxel in the second data; and controls the target convolution operator to output the second data.

[0233] The third processing unit 403 controls the target convolution operator to output second data, where the second data includes a plurality of second voxels.

[0234] The solution provided in the embodiment of the present application calculates the intermediate results of each first background value and the convolution kernel before searching for the intermediate results from the first correspondence, such as: 101 and the intermediate result 1 of the convolution kernel, 102 and the intermediate result 2 of the convolution kernel, and 103 and the intermediate result 3 of the convolution kernel. Because the first background value corresponds to the background identifier, a correspondence between the background identifier and the convolution result can be established, such as: 1->intermediate result 1, 2->intermediate result 2 and 3->intermediate result 3. Usually, the first data includes many first voxels. The process of convolution processing the first data in the prior art is the process of calculating the values ​​of these first voxels one by one with the convolution kernel. Among these first voxels, the voxels indicating the background usually account for the vast majority, and the values ​​of these voxels indicating the background are repeated in large quantities. Calculating them one by one with the convolution kernel is very slow. In this application, a calculation is performed for each first background value before the convolution kernel is applied. After obtaining an intermediate result, the correspondence between the intermediate result and the background identifier is stored. When the first data is convolved, the intermediate result can be directly found by looking up the correspondence for the first voxel indicating the background in the first data. The intermediate result is then used to obtain the value of the second voxel in the second data. This eliminates the need for extensive calculations for each first voxel value and the convolution kernel, thereby saving computational overhead and achieving convolution acceleration.

[0235] Optionally, the convolutional neural network includes: an operation subnet, a background deduction subnet and a sparse distribution deduction subnet, and the network structures of the background deduction subnet and the sparse distribution deduction subnet are the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to the second convolution operator, and the position of the second convolution operator in the sparse distribution deduction subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value comes from an operator that inputs at least one first background value to the third convolution operator, and the position of the third convolution operator in the background deduction subnet corresponds to the position of the target convolution operator in the operation subnet.

[0236] Optionally, the convolutional neural network includes: an operation subnet and a sparse distribution deduction subnet, the network structure of the sparse distribution deduction subnet is the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, and the position of the second convolution operator in the sparse distribution deduction subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value is pre-configured for the target convolution operator.

[0237] Optionally, the intermediate result in the corresponding relationship is the result obtained by calculating the first background value at different positions of the convolution kernel; the second processing unit 402 is used to search the corresponding relationship for the intermediate result corresponding to the background identifier at the corresponding position for each first voxel indicating the background in the sliding window of the first data, according to the background identifier corresponding to each first voxel indicating the background in the first sparse distribution map, and the position of each first voxel indicating the background in the sliding window, and the sliding window of the first data is the same size as the convolution kernel of the target convolution operator.

[0238] Optionally, the second processing unit 402 is further configured to accumulate intermediate results of first voxels indicating background in the sliding window of the first data to obtain a value of a second voxel, wherein all first voxels in the sliding window of the first data indicate background.

[0239] Optionally, the second processing unit 402 is further used to accumulate the intermediate results of the first voxel indicating the background in the sliding window of the first data, and add the intermediate results of the first voxel indicating the non-background to obtain the value of a second voxel, and the intermediate result of the first voxel indicating the non-background is obtained by calculating the value of the first voxel indicating the non-background and the convolution kernel.

[0240] Optionally, the target convolution operator is the first convolution operator in the operation subnet, and the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background, and for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

[0241] Optionally, the at least one first background value of the first data is represented in the form of a one-dimensional array.

[0242] Optionally, the target convolution operator is not the first convolution operator in the operation subnetwork, and the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data.

[0243] Optionally, the first background values ​​of the sub-data in different channel dimensions are different.

[0244] Optionally, the at least one first background value of the first data is represented in the form of a two-dimensional array.

[0245] Optionally, the first data is a multidimensional array of video data, voice data or text data.

[0246] The corresponding functions of the above-mentioned device 40 for processing three-dimensional data can also be understood by referring to the relevant contents of the above-mentioned method for processing three-dimensional data, and will not be repeated here.

[0247] like Figure 14 As shown, an embodiment of the apparatus 50 for processing three-dimensional data provided in the embodiment of the present application includes:

[0248] A first processing unit 501 is configured to input first data into a target convolution operator of a convolutional neural network. The convolutional neural network includes at least one first convolution operator, and the target convolution operator is any one of the at least one first convolution operator. The first data includes a plurality of first voxels. The target convolution operator further inputs: at least one second background value obtained by convolving at least one background value of the first data, and a second sparse distribution map obtained by performing a row identifier check operation on the first sparse distribution map. The identifier check operation is to check whether identifiers of points covered by a sliding window of the first sparse distribution map are all the same or not all the same. The identifiers include background identifiers and non-background identifiers. The at least one first background value indicates at least one background value in the first data, and the at least one first background value has a one-to-one correspondence with the at least one background value. The first sparse distribution map describes the sparse distribution of the plurality of first voxels using the background identifier and the non-background identifier. The background identifier indicates that a first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicates that a first voxel corresponding to the non-background identifier in the first data is non-background.

[0249] The second processing unit 502 is configured to search for a second background value corresponding to the background identifier of the background point according to the background identifier representing the background point on the second sparse distribution map, where the second background value corresponding to the background identifier of the background point is the value of a second voxel corresponding to the background point in the second data.

[0250] The third processing unit 503 is configured to control the target convolution operator to output second data, where the second data includes a plurality of second voxels.

[0251] In the solution provided by the embodiment of the present application, after the target convolution operator receives the first data, the second background value, and the second sparse distribution map, it can directly search for the corresponding second background value using the background identifier in the second sparse distribution map. The second background value found is the value of the second voxel at the position corresponding to the background identifier. It can be seen that the solution provided by the embodiment of the present application can directly search for the corresponding value for the second voxel indicated by the background identifier in the second sparse distribution map, eliminating the need for further convolution processing, thereby achieving convolution acceleration.

[0252] Optionally, the second processing unit 502 is further used to search for an intermediate result corresponding to the background identifier of the target point from the correspondence relationship for a non-background point on the second sparse distribution map, based on the background identifier of the target point on the first sparse distribution map. The target point is a non-background point on the second sparse distribution map positioned to the corresponding background-indicating point on the first sparse distribution map. The correspondence relationship indicates a relationship between the intermediate result of at least one first background value and the background identifier corresponding to at least one first background value. The intermediate result of one of the at least one first background values ​​is calculated by the convolution kernel of the first background value and the target convolution operator. The intermediate result in the correspondence relationship is a result obtained by calculating the first background value at different positions of the convolution kernel. The intermediate result corresponding to the background identifier of the target point is used to determine the value of the second voxel corresponding to the non-background point on the second sparse distribution map.

[0253] Optionally, the convolutional neural network includes: an operation subnet, a background derivation subnet and a sparse distribution derivation subnet, and the network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to the second convolution operator, and the second sparse distribution map is obtained after the second convolution operator performs a row identification verification operation on the first sparse distribution map, and the position of the second convolution operator in the sparse distribution derivation subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value comes from an operator that inputs at least one first background value to the third convolution operator, and at least one second background value is obtained by the third convolution operator performing convolution processing on the at least one first background value, and the position of the third convolution operator in the background derivation subnet corresponds to the position of the target convolution operator in the operation subnet.

[0254] Optionally, the convolutional neural network includes: an operation subnet and a sparse distribution deduction subnet, the network structure of the sparse distribution deduction subnet is the same as the network structure of the operation subnet; the operation subnet includes at least one first convolution operator; the first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identification verification operation on the first sparse distribution map, the position of the second convolution operator in the sparse distribution deduction subnet corresponds to the position of the target convolution operator in the operation subnet; at least one first background value and at least one second background value are pre-configured for the target convolution operator.

[0255] Optionally, the second processing unit 502 is further configured to accumulate the intermediate results corresponding to each target point found to obtain the value of the second voxel corresponding to the non-background point, and the points corresponding to the non-background points located on the first sparse distribution map are all points indicating the background.

[0256] Optionally, the second processing unit 502 is further configured to accumulate the intermediate results corresponding to each target point found to obtain the value of the second voxel corresponding to the non-background point, and the points corresponding to the non-background points located on the first sparse distribution map are all points indicating the background.

[0257] Optionally, the second processing unit 502 is further used to accumulate the intermediate results corresponding to each target point found, and add the intermediate results corresponding to the dense points corresponding to the non-background points located on the first sparse distribution map, to obtain the value of the second voxel corresponding to the non-background point. The dense point is the point indicating the non-background in the first sparse distribution map, and the intermediate result corresponding to the dense point is calculated by the value of the first voxel indicating the non-background corresponding to the dense point in the first data and the convolution kernel.

[0258] Optionally, the target convolution operator is the first convolution operator in the operation subnet, and the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background, and for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

[0259] Optionally, at least one first background value of the first data is represented in the form of a one-dimensional array, and at least one second background value is represented in the form of a two-dimensional array.

[0260] Optionally, the target convolution operator is not the first convolution operator in the operation subnetwork, and the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data.

[0261] Optionally, the first background values ​​of the sub-data in different channel dimensions are different.

[0262] Optionally, the at least one first background value of the first data is represented in the form of a two-dimensional array.

[0263] Optionally, the first data is a multidimensional array of video data, voice data or text data.

[0264] The corresponding functions of the above-mentioned device 50 for processing three-dimensional data can also be understood by referring to the relevant contents of the above-mentioned method for processing three-dimensional data, and will not be repeated here.

[0265] like Figure 15 As shown, an embodiment of the apparatus 60 for constructing a convolutional neural network provided in an embodiment of the present application includes:

[0266] The acquisition unit 601 is used to acquire an initial convolutional neural network, where the initial convolutional neural network includes an operation subnet, and the operation subnet includes at least one first convolution operator.

[0267] The first processing unit 602 is configured to copy a sparse distribution derivation subnet according to the operation subnet.

[0268] The second processing unit 603 is used to construct at least one of a first input relationship and a second input relationship, where the first input relationship is an input relationship from the second convolution operator in the sparse distribution inference subnet to the first convolution operator at a corresponding position, and the second input relationship is an input relationship from the operator that inputs data to the second convolution operator in the sparse distribution inference subnet to the first convolution operator at a corresponding position, so as to obtain a target convolutional neural network.

[0269] The solution provided in the embodiment of the present application adds a background deduction subnet and a sparse distribution deduction subnet of the same structure to the operation subnet on the basis of the initial convolutional neural network used only for processing three-dimensional data. In this way, it can be ensured that in the process of convolution processing of three-dimensional data by convolution operators at different positions in the operation subnet, the background value at the corresponding position can be obtained from the background deduction subnet, and the sparse distribution map at the corresponding position can be obtained from the sparse distribution deduction subnet, thereby ensuring that the target convolutional neural network constructed by the present application can improve the speed of convolution processing.

[0270] Optionally, before obtaining the target convolutional neural network, the method also includes: copying the background derivation subnetwork according to the operation subnetwork; the second processing unit 603 is also used to construct at least one of a third input relationship and a fourth input relationship, the third input relationship is the input relationship from the third convolution operator in the background derivation subnetwork to the first convolution operator at the corresponding position, and the fourth input relationship is the input relationship from the operator that inputs data to the third convolution operator in the background derivation subnetwork to the first convolution operator at the corresponding position, the first input relationship corresponds to the third input relationship, and the second input relationship corresponds to the fourth input relationship.

[0271] Optionally, the first processing unit 602 is also used to determine an operation subnetwork from the initial convolutional neural network, wherein the input data of each operator in the operation subnetwork is sparsely distributed in the same channel dimension, and the output data of each operator is sparsely distributed in the same channel dimension, and the channel dimension represents a dimension different from the three-dimensional space.

[0272] The corresponding functions of the above-mentioned device 60 for constructing a convolutional neural network can also be understood by referring to the relevant content of the above-mentioned method for constructing a convolutional neural network, and will not be repeated here.

[0273] Figure 16As shown, it is a possible logical structure diagram of the computer device 70 provided in the embodiment of the present application. The computer device 70 can be a device 40 for processing three-dimensional data, a device 50 for processing three-dimensional data, or a device 60 for constructing a convolutional neural network. The computer device 70 includes: a processor 701, a communication interface 702, a memory 703 and a bus 704. The processor 701, the communication interface 702 and the memory 703 are interconnected via the bus 704. In the embodiment of the present application, the processor 701 is used to control and manage the actions of the computer device 70, for example, the processor 701 is used to execute Figures 3 to 12 In the process of obtaining and processing the method embodiment, the communication interface 702 is used to support the computer device 70 to communicate. The memory 703 is used to store the program code and data of the computer device 70.

[0274] Among them, the processor 701 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 701 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 16 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0275] In another embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the computer is prepared to execute the above-mentioned Figures 3 to 4G The method of building a convolutional neural network in , or perform the above Figures 5 to 12 Methods for processing three-dimensional data.

[0276] In another embodiment of the present application, a computer program product is further provided. The computer program product includes computer-executable instructions stored in a computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the computer is prepared to execute the above-mentioned Figures 3 to 4G The method of building a convolutional neural network in , or perform the above Figures 5 to 12Methods for processing three-dimensional data.

[0277] In another embodiment of the present application, a chip system is further provided, the chip system including a processor, the processor being configured to implement the above Figures 3 to 4G The method of building a convolutional neural network in , or perform the above Figures 5 to 12 In one possible design, the chip system may further include a memory for storing program instructions and data necessary for inter-process communication. The chip system may consist of a chip alone or include a chip and other discrete components.

[0278] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0279] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0280] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0281] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0282] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0283] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0284] The above is only a specific implementation of the embodiment of the present application, but the scope of protection of the embodiment of the present application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the embodiment of the present application, and they should be included in the scope of protection of the embodiment of the present application. Therefore, the scope of protection of the embodiment of the present application should be based on the scope of protection of the claims.

Claims

1. A method for processing three-dimensional data, characterized in that: include: Inputting first data, at least one first background value of the first data, and a first sparse distribution map of the first data into a target convolution operator of a convolutional neural network, wherein the convolutional neural network includes at least one first convolution operator, the target convolution operator is any one of the at least one first convolution operator, the first data includes a plurality of first voxels, the at least one first background value indicates at least one background in the first data, the at least one first background value has a one-to-one correspondence with the at least one background, and the first sparse distribution map describes the sparse distribution of the plurality of first voxels through a background identifier and a non-background identifier, the background identifier indicates that a first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicates that a first voxel corresponding to the non-background identifier in the first data is non-background; searching, based on a background identifier of a background point corresponding to a first voxel indicating background in the first data on the first sparse distribution map, for an intermediate result corresponding to the background identifier of the background point from a corresponding relationship, wherein the corresponding relationship indicates a relationship between the intermediate result of each of the at least one first background values ​​and the background identifier corresponding to each of the at least one first background values, the intermediate result of one of the at least one first background values ​​being calculated by a convolution kernel of the first background value and the target convolution operator, and the intermediate result corresponding to the background identifier of the background point being used to determine a value of a second voxel in the second data; The target convolution operator is controlled to output the second data, where the second data includes a plurality of second voxels.

2. The method according to claim 1, characterized in that The convolutional neural network includes: an operation subnet, a background derivation subnet and a sparse distribution derivation subnet, and the network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as the network structure of the operation subnet; The operation subnetwork includes the at least one first convolution operator; The first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, and a position of the second convolution operator in the sparse distribution derivation subnetwork corresponds to a position of the target convolution operator in the operation subnetwork; The at least one first background value comes from an operator that inputs the at least one first background value to a third convolution operator, and a position of the third convolution operator in the background derivation subnetwork corresponds to a position of the target convolution operator in the operation subnetwork.

3. The method according to claim 1, characterized in that The convolutional neural network includes: a computing subnet and a sparse distribution inference subnet, wherein the network structure of the sparse distribution inference subnet is the same as that of the computing subnet; The operation subnetwork includes the at least one first convolution operator; The first sparse distribution map comes from an operator that inputs the first sparse distribution map to a second convolution operator, and a position of the second convolution operator in the sparse distribution derivation subnetwork corresponds to a position of the target convolution operator in the operation subnetwork; The at least one first background value is pre-configured to the target convolution operator.

4. The method according to any one of claims 1 to 3, characterized in that The intermediate results in the corresponding relationship are the results obtained by calculating the first background value at different positions of the convolution kernel; The step of searching, from a corresponding relationship based on a background identifier of a background point corresponding to a first voxel indicating background in the first data on the first sparse distribution map, for an intermediate result corresponding to the background identifier of the background point includes: For each first voxel indicating the background in the sliding window of the first data, based on the background identifier corresponding to each first voxel indicating the background in the first sparse distribution map, and the position of each first voxel indicating the background in the sliding window, the intermediate result corresponding to the background identifier at the corresponding position is searched from the corresponding relationship, and the sliding window of the first data is the same size as the convolution kernel of the target convolution operator.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The intermediate results of the first voxels indicating the background in the sliding window of the first data are accumulated to obtain a value of a second voxel, and the first voxels in the sliding window of the first data all indicate the background.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The intermediate results of the first voxel indicating the background in the sliding window of the first data are accumulated and added to the intermediate result of the first voxel indicating the non-background to obtain the value of a second voxel, where the intermediate result of the first voxel indicating the non-background is obtained by calculating the value of the first voxel indicating the non-background and the convolution kernel.

7. The method according to claim 2 or 3, characterized in that The target convolution operator is the first convolution operator in the operation subnet, and the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background; for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

8. The method according to claim 7, characterized in that The at least one first background value of the first data is represented in the form of a one-dimensional array.

9. The method according to claim 2 or 3, characterized in that If the target convolution operator is not the first convolution operator in the operation subnetwork, the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data.

10. The method according to claim 9, characterized in that The first background values ​​of the sub-data in different channel dimensions are different.

11. The method according to claim 10, characterized in that The at least one first background value of the first data is represented in the form of a two-dimensional array.

12. The method according to any one of claims 1 to 3, characterized in that The first data is a multi-dimensional array of video data, voice data or text data.

13. A method for processing three-dimensional data, characterized in that: include: Inputting first data into a target convolution operator of a convolutional neural network, the convolutional neural network including at least one first convolution operator, the target convolution operator being any one of the at least one first convolution operator, the first data including a plurality of first voxels, the target convolution operator further inputting: at least one second background value obtained by performing convolution processing on at least one first background value of the first data, and a second sparse distribution map obtained by performing a row identifier check operation on a first sparse distribution map, the identifier check operation being to check whether identifiers of points covered by a sliding window of the first sparse distribution map are all the same or not all the same, the identifiers including background identifiers and non-background identifiers, the at least one first background value indicating at least one background in the first data, the at least one first background value corresponding one-to-one to the at least one background, the first sparse distribution map describing the sparse distribution of the plurality of first voxels using the background identifiers and the non-background identifiers, the background identifier indicating that a first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicating that a first voxel corresponding to the non-background identifier in the first data is non-background; searching, according to the background identifier representing the background point on the second sparse distribution map, for a second background value corresponding to the background identifier of the background point, the second background value corresponding to the background identifier of the background point being the value of a second voxel corresponding to the background point in the second data; For a non-background point on the second sparse distribution map, based on the background identifier of the target point on the first sparse distribution map, searching from the correspondence for an intermediate result corresponding to the background identifier of the target point, the target point being the non-background point on the second sparse distribution map positioned at the corresponding background-indicating point on the first sparse distribution map, the correspondence indicating a relationship between the intermediate result of each of the at least one first background values ​​and the background identifier corresponding to each of the at least one first background values, the intermediate result of one of the at least one first background values ​​being calculated by a convolution kernel of the first background value and the target convolution operator, the intermediate results in the correspondence being results calculated by applying the first background value at different positions of the convolution kernel, and the intermediate result corresponding to the background identifier of the target point being used to determine a value of a second voxel corresponding to the non-background point on the second sparse distribution map; The target convolution operator is controlled to output the second data, where the second data includes a plurality of second voxels.

14. The method according to claim 13, characterized in that The convolutional neural network includes: an operation subnet, a background derivation subnet and a sparse distribution derivation subnet, and the network structures of the background derivation subnet and the sparse distribution derivation subnet are the same as the network structure of the operation subnet; The operation subnetwork includes the at least one first convolution operator; The first sparse distribution map comes from an operator that inputs the first sparse distribution map into a second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identifier verification operation on the first sparse distribution map, and a position of the second convolution operator in the sparse distribution derivation subnet corresponds to a position of the target convolution operator in the operation subnet; The at least one first background value comes from an operator that inputs the at least one first background value into a third convolution operator, and the at least one second background value is obtained by convolution processing of the at least one first background value by the third convolution operator. The position of the third convolution operator in the background derivation subnetwork corresponds to the position of the target convolution operator in the operation subnetwork.

15. The method according to claim 13, characterized in that The convolutional neural network includes: a computing subnet and a sparse distribution inference subnet, wherein the network structure of the sparse distribution inference subnet is the same as that of the computing subnet; The operation subnetwork includes the at least one first convolution operator; The first sparse distribution map comes from an operator that inputs the first sparse distribution map into a second convolution operator, the second sparse distribution map is obtained after the second convolution operator performs a row identifier verification operation on the first sparse distribution map, and a position of the second convolution operator in the sparse distribution derivation subnet corresponds to a position of the target convolution operator in the operation subnet; The at least one first background value and the at least one second background value are pre-configured to the target convolution operator.

16. The method according to claim 13, characterized in that The method further comprises: The intermediate results corresponding to each target point found are accumulated to obtain the value of the second voxel corresponding to the non-background point. The points corresponding to the non-background points located on the first sparse distribution map are all points indicating the background.

17. The method according to claim 13, wherein The method further comprises: The intermediate results corresponding to each target point found are accumulated, and the intermediate results corresponding to the dense points corresponding to the non-background points on the first sparse distribution map are added to obtain the value of the second voxel corresponding to the non-background point. The dense points are points indicating non-background in the first sparse distribution map, and the intermediate results corresponding to the dense points are calculated by adding the value of the first voxel indicating non-background corresponding to the dense points in the first data and the convolution kernel.

18. The method according to claim 14 or 15, characterized in that The target convolution operator is the first convolution operator in the operation subnet, and the first sparse distribution map is obtained by scanning each first voxel in the first data one by one by the initialization operator in the sparse distribution deduction subnet, wherein, for the first voxel indicating the background, the corresponding background identifier is written at the corresponding point in the blank bitmap corresponding to the first data according to the first background value corresponding to the first voxel indicating the background, and the corresponding point is the point in the blank bitmap corresponding to the data indicating the background; for the first voxel indicating the non-background, the non-background identifier is written at the point in the blank bitmap corresponding to the first voxel indicating the non-background.

19. The method according to claim 18, characterized in that At least one first background value of the first data is represented in the form of a one-dimensional array, and at least one second background value is represented in the form of a two-dimensional array.

20. The method according to claim 14 or 15, characterized in that If the target convolution operator is not the first convolution operator in the operation subnetwork, the first data has multiple channel dimensions, wherein the sparse distribution map of the sub-data on each channel dimension is the same as the first sparse distribution map of the first data.

21. The method according to claim 20, characterized in that The first background values ​​of the sub-data in different channel dimensions are different.

22. The method according to claim 21, characterized in that The at least one first background value of the first data is represented in the form of a two-dimensional array.

23. The method according to any one of claims 13 to 17, characterized in that The first data is a multi-dimensional array of video data, voice data or text data.

24. A method for constructing a convolutional neural network, characterized in that include: Acquire an initial convolutional neural network, where the initial convolutional neural network includes an operation subnet, and the operation subnet includes at least one first convolution operator; According to the operation subnet, copy the sparse distribution derivation subnet and the background derivation subnet; Constructing at least one of a first input relationship and a second input relationship, and constructing at least one of a third input relationship and a fourth input relationship, to obtain a target convolutional neural network; wherein the first input relationship is an input relationship from the second convolution operator in the sparse distribution inference subnetwork to the first convolution operator at the corresponding position, and the second input relationship is an input relationship from an operator in the sparse distribution inference subnetwork that inputs data to the second convolution operator to the first convolution operator at the corresponding position; The third input relationship is the input relationship from the third convolution operator in the background derivation subnet to the first convolution operator at the corresponding position, and the fourth input relationship is the input relationship from the operator that inputs data to the third convolution operator in the background derivation subnet to the first convolution operator at the corresponding position. The first input relationship corresponds to the third input relationship, and the second input relationship corresponds to the fourth input relationship. The target convolutional neural network is applied to the method described in any one of claims 1 to 12 above, or to the method described in any one of claims 13 to 23 above.

25. The method according to claim 24, characterized in that The method further comprises: The operation subnetwork is determined from the initial convolutional neural network, wherein the input data of each operator in the operation subnetwork is sparsely distributed in the same channel dimension, and the output data of each operator is sparsely distributed in the same channel dimension, where the channel dimension represents a dimension different from the three-dimensional space.

26. A device for processing three-dimensional data, characterized in that: include: a first processing unit, configured to input first data, at least one first background value of the first data, and a first sparse distribution map of the first data into a target convolution operator of a convolutional neural network, wherein the convolutional neural network includes at least one first convolution operator, the target convolution operator is any one of the at least one first convolution operator, the first data includes a plurality of first voxels, the at least one first background value indicates at least one background in the first data, the at least one first background value corresponds one-to-one with the at least one background, and the first sparse distribution map describes the sparse distribution of the plurality of first voxels through a background identifier and a non-background identifier, the background identifier indicates that a first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicates that a first voxel corresponding to the non-background identifier in the first data is non-background; a second processing unit configured to search, from a correspondence relationship, for an intermediate result corresponding to the background identifier of a background point on the first sparse distribution map corresponding to a first voxel indicating background in the first data input by the first processing unit, wherein the correspondence relationship indicates a relationship between the intermediate result of each of the at least one first background values ​​and the background identifier corresponding to each of the at least one first background values, the intermediate result of one of the at least one first background values ​​being calculated by a convolution kernel of the first background value and the target convolution operator, and the intermediate result corresponding to the background identifier of the background point being used to determine a value of a second voxel in the second data; The third processing unit is configured to control the target convolution operator to output the second data obtained by the second processing unit, where the second data includes a plurality of second voxels.

27. A device for processing three-dimensional data, characterized in that: include: a first processing unit, configured to input first data into a target convolution operator of a convolutional neural network, the convolutional neural network comprising at least one first convolution operator, the target convolution operator being any one of the at least one first convolution operator, the first data comprising a plurality of first voxels, the target convolution operator further inputting: at least one second background value obtained by performing convolution processing on at least one first background value of the first data, and a second sparse distribution map obtained by performing a row identifier check operation on a first sparse distribution map, the identifier check operation being to check whether identifiers of points covered by a sliding window of the first sparse distribution map are all the same or not all the same, the identifiers comprising background identifiers and non-background identifiers, the at least one first background value indicating at least one type of background in the first data, the at least one first background value corresponding one-to-one to the at least one background, the first sparse distribution map describing the sparse distribution of the plurality of first voxels via the background identifiers and the non-background identifiers, the background identifier indicating that a first voxel corresponding to the background identifier in the first data is background, and the non-background identifier indicating that a first voxel corresponding to the non-background identifier in the first data is non-background; a second processing unit configured to search, based on the background identifier representing the background point on the second sparse distribution map input by the first processing unit, for a second background value corresponding to the background identifier of the background point, the second background value corresponding to the background identifier of the background point being a value of a second voxel corresponding to the background point in the second data; The second processing unit is further configured to, for a non-background point on the second sparse distribution map, search, based on the background identifier of the target point on the first sparse distribution map, from a correspondence relationship for an intermediate result corresponding to the background identifier of the target point, the target point being a point indicating the background corresponding to the non-background point on the second sparse distribution map and located on the first sparse distribution map, the correspondence relationship indicating a relationship between the intermediate result of each of the at least one first background values ​​and the background identifier corresponding to each of the at least one first background values, the intermediate result of one of the at least one first background values ​​being calculated by a convolution kernel of the first background value and the target convolution operator, the intermediate results in the correspondence relationship being results obtained by calculating the first background value at different positions of the convolution kernel, and the intermediate result corresponding to the background identifier of the target point being used to determine a value of a second voxel corresponding to the non-background point on the second sparse distribution map; The third processing unit is configured to control the target convolution operator to output the second data obtained by the second processing unit, where the second data includes a plurality of second voxels.

28. A device for constructing a convolutional neural network, characterized in that include: An acquisition unit, configured to acquire an initial convolutional neural network, wherein the initial convolutional neural network includes an operation subnet, and the operation subnet includes at least one first convolution operator; A first processing unit is configured to copy a sparse distribution derivation subnet and a background derivation subnet according to the operation subnet in the initial convolutional neural network obtained by the obtaining unit; A second processing unit is used to construct at least one of a first input relationship and a second input relationship, and to construct at least one of a third input relationship and a fourth input relationship to obtain a target convolutional neural network; wherein, the first input relationship is the input relationship from the second convolution operator in the sparse distribution inference subnet copied from the first processing unit to the first convolution operator at the corresponding position, the second input relationship is the input relationship from the operator that inputs data to the second convolution operator in the sparse distribution inference subnet to the first convolution operator at the corresponding position, the third input relationship is the input relationship from the third convolution operator in the background inference subnet to the first convolution operator at the corresponding position, and the fourth input relationship is the input relationship from the operator that inputs data to the third convolution operator in the background inference subnet to the first convolution operator at the corresponding position, the first input relationship corresponds to the third input relationship, and the second input relationship corresponds to the fourth input relationship. The target convolutional neural network is applied to the method described in any one of claims 1 to 12 above, or to the method described in any one of claims 13 to 23 above.

29. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented, or the method according to any one of claims 13 to 23 is implemented, or the method according to any one of claims 24 to 25 is implemented.

30. A computing device, characterized in that comprising a processor and a computer-readable storage medium storing a computer program; The processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the computer program implements the method according to any one of claims 1 to 12, or the method according to any one of claims 13 to 23, or the method according to any one of claims 24 to 25.

31. A chip system, characterized in that: The method comprises a processor, wherein the processor is called to execute the method according to any one of claims 1 to 12, or implement the method according to any one of claims 13 to 23, or implement the method according to any one of claims 24 to 25.

Images