Sparse neural network decision structure and training method thereof, data processing method and device, chip and electronic equipment
By adopting sparse neural network decision structure and low bit operations on the edge computing platform, the contradiction between computing power consumption and efficiency of convolutional neural networks in a resource-scarce environment is solved, and the balance of accuracy and speed is achieved.
Patent Information
- Application Number
- CN202310662762.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
Convolutional neural network models are difficult to deploy on edge computing platforms with scarce memory and computing resources, and there is a contradiction between computing power consumption and operating efficiency. How to balance accuracy performance and operating speed has become an urgent problem for technicians to solve.
The data processing method based on the decision structure of the sparse neural network is adopted to save computational amount and improve operation speed while maintaining processing accuracy through the sparse convolution kernel and the multiplication and addition of low bits.
While ensuring the accuracy of convolution processing, it significantly improves the running speed, solves the resource consumption problem of convolutional neural networks on the edge computing platform, and achieves a good balance of accuracy and speed.
Smart Images

Figure CN119089944A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of neural network technology, and in particular to a data processing method based on a sparse neural network decision structure, a sparse neural network decision structure, a data processing device based on a sparse neural network decision structure, a training method for a sparse neural network decision structure, a chip, an electronic device and a computer storage medium. Background Art
[0002] With the continuous development of deep learning technology, in the field of images, more and more deep learning algorithms based on convolutional neural networks (CNN) have achieved excellent processing results in image classification, target detection, depth estimation, image segmentation and super-resolution. However, due to the huge number of parameters and computing resource overhead, convolutional neural network models are difficult to deploy on edge computing platforms with scarce memory and computing resources. In practical applications, in addition to the demand for high-precision algorithms, a large number of application fields also have strict requirements on the running speed of the algorithms. Under the condition of certain computing resources, there is a contradiction between the computing power consumption and running efficiency of the convolutional neural network model. How to balance the accuracy performance and running speed of the convolutional neural network has become a problem that technicians in this field need to solve urgently. Summary of the invention
[0003] Embodiments of the present application provide a data processing method based on a sparse neural network decision structure, a sparse neural network decision structure, a data processing device based on a sparse neural network decision structure, a training method for a sparse neural network decision structure, a chip, an electronic device, and a computer storage medium.
[0004] In a first aspect, an embodiment of the present application provides a data processing method based on a sparse neural network decision structure, the data processing method comprising: performing a first processing on an input feature to obtain first feature data, the first processing comprising a first convolution processing using a first convolution kernel, the first convolution kernel being a sparse convolution kernel having a first sparse coefficient, and the dimension of the first feature data being higher than the dimension of the input feature; performing a second processing on the first feature data to obtain second feature data, the second processing comprising a second convolution processing using a second convolution kernel, the second convolution kernel being a sparse convolution kernel having a second sparse coefficient; and performing a third processing on the second feature data to obtain third feature data, the third processing comprising a third convolution processing using a third convolution kernel, the third convolution kernel being a sparse convolution kernel having a third sparse coefficient, the dimension of the third feature data being lower than the dimension of the second feature data, and the sparse coefficient being the proportion of the number of non-zero weights in the convolution kernel to the total number of weights.
[0005] In the second aspect, the embodiment of the present application provides a sparse neural network decision structure, which includes a first convolution layer module, a second convolution layer module and a third convolution layer module. The first convolution layer module is used to perform a first process on the input feature to obtain the first feature data, the first process includes a first convolution process using a first convolution kernel, the first convolution kernel is a sparse convolution kernel with a first sparse coefficient, and the dimension of the first feature data is higher than the dimension of the input feature. The second convolution layer module is used to perform a second process on the first feature data to obtain the second feature data, the second process includes a second convolution process using a second convolution kernel, the second convolution kernel is a sparse convolution kernel with a second sparse coefficient. The third convolution layer module is used to perform a third process on the second feature data to obtain the third feature data, the third process includes a third convolution process using the third convolution kernel, the third convolution kernel is a sparse convolution kernel with a third sparse coefficient, the dimension of the third feature data is lower than the dimension of the second feature data, and the sparse coefficient is the proportion of the number of non-zero weights in the convolution kernel to the total number of weights.
[0006] In a third aspect, an embodiment of the present application provides a data processing device based on a sparse neural network decision structure, wherein the data processing device includes a sparse neural network decision structure. The sparse neural network decision structure includes a first convolution layer module, a second convolution layer module, and a third convolution layer module. The first convolution layer module is used to perform a first processing on the input feature to obtain first feature data, wherein the first processing includes a first convolution processing using a first convolution kernel, wherein the first convolution kernel is a sparse convolution kernel having a first sparse coefficient, and the dimension of the first feature data is higher than the dimension of the input feature. The second convolution layer module is used to perform a second processing on the first feature data to obtain second feature data, wherein the second processing includes a second convolution processing using a second convolution kernel, wherein the second convolution kernel is a sparse convolution kernel having a second sparse coefficient. The third convolution layer module is used to perform a third processing on the second feature data to obtain third feature data, the third processing includes a third convolution processing using the third convolution kernel, the third convolution kernel is a sparse convolution kernel with a third sparse coefficient, the dimension of the third feature data is lower than the dimension of the second feature data, and the sparse coefficient is the proportion of the number of non-zero weights in the convolution kernel to the total number of weights.
[0007] In a fourth aspect, an embodiment of the present application provides a training method for a sparse neural network decision structure, the training method comprising:
[0008] Train an initial sparse convolutional network, the initial sparse convolutional network comprising a first convolutional module, a second convolutional module and a third convolutional module, the first convolutional module comprising a first convolutional layer, a first batch of normalization layers and a first activation layer, the convolution kernel of the first convolutional layer is a sparse convolution kernel with a first sparse coefficient, the second convolutional module comprising a second convolutional layer, a second batch of normalization layers and a second activation layer, the convolution kernel of the second convolutional layer is a sparse convolution kernel with a second sparse coefficient, the third convolutional module comprising a third convolutional layer, a third batch of normalization layers and a third activation layer, the convolution kernel of the third convolutional layer is a sparse convolution kernel with a third sparse coefficient;
[0009] The weight values of the convolution kernels in the first convolution layer, the second convolution layer, and the third convolution layer are all approximated to 2 by incremental network quantization. n ; and merging the first batch of normalization and the first activation layer into a low-bit multiplication-addition operation, merging the second batch of normalization and the second activation layer into a low-bit multiplication-addition operation, and merging the third batch of normalization and the third activation layer into a low-bit multiplication-addition operation to obtain the sparse neural network decision structure.
[0010] In a fifth aspect, an embodiment of the present application provides a chip comprising a memory and a processor; wherein the memory is used to store a computer program that can be run on the processor; and the processor is used to enable an electronic device equipped with the chip to execute the above-mentioned data processing method, or to run the above-mentioned data processing device, when running the computer program.
[0011] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; wherein the memory is used to store a computer program that can be run on the processor; and the processor is used to execute the data processing method described in the claim above, or to run the data processing device described above, when running the computer program.
[0012] In a seventh aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the computer program implements the above-mentioned data processing method, or runs the above-mentioned data processing device.
[0013] The data processing method based on a sparse neural network decision structure, the sparse neural network decision structure, the data processing device based on a sparse neural network decision structure, the training method of a sparse neural network decision structure, the chip, the electronic device and the computer storage medium of the present application respectively use a sparse first convolution kernel, a sparse second convolution kernel and a sparse third convolution kernel in the first processing, the second processing and the third processing, and the weights in these sparse convolution kernels are sparsely arranged and have reasonable sparsity coefficients, which can save the amount of calculation for performing multiplication and addition calculations with input features or feature data in each convolution processing process and improve the running speed, while ensuring the processing accuracy of the convolution processing, and well balancing the accuracy performance and running speed of the convolution neural network.
[0014] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0016] Figure 1 is a flow chart of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0017] Figure 2 It is a schematic diagram of the structure of a sparse neural network decision structure in some implementation modes of the present application;
[0018] Figure 3 is a schematic diagram of a data processing method based on a sparse neural network decision structure according to an embodiment of the present application;
[0019] Figure 4 is a schematic diagram of a data processing method based on a sparse neural network decision structure according to an embodiment of the present application;
[0020] Figure 5 yes Figure 4 Schematic diagram of the convolution kernel in the data processing method shown;
[0021] Figures 6 to 9 is a flow chart of a data processing method based on a sparse neural network decision structure in certain embodiments of the present application;
[0022] Fig.10 It is a structural schematic diagram of a data processing device based on a sparse neural network decision structure in some embodiments of the present application;
[0023] Fig.11 is a flow chart of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0024] Fig.12 is a schematic diagram of a data processing method based on a sparse neural network decision structure according to an embodiment of the present application;
[0025] Fig.13 is a flow chart of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0026] Fig.14 It is a schematic diagram of four interpolation strategies for interpolating bad pixel pixels in input features of the present application;
[0027] Fig.15 is a schematic diagram of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0028] Fig.16 is a flow chart of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0029] Fig.17 It is a structural schematic diagram of a data processing device based on a sparse neural network decision structure in some embodiments of the present application;
[0030] Fig.18 is a schematic diagram of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0031] Fig.19 is a schematic diagram of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0032] Fig. 20 It is a structural schematic diagram of a data processing device based on a sparse neural network decision structure in some embodiments of the present application;
[0033] Fig.21 is a schematic diagram of a data processing method based on a sparse neural network decision structure in some embodiments of the present application;
[0034] Fig. 22 It is a schematic diagram of the structure of the chip of some implementation methods of the present application;
[0035] Fig.23 is a schematic diagram of the structure of an electronic device in some embodiments of the present application;
[0036] Fig.24 is a schematic diagram of the connection between a computer storage medium and a processor in some embodiments of the present application;
[0037] Fig.25 It is a schematic diagram of the principle of a training method for a sparse neural network decision structure in some embodiments of the present application;
[0038] Fig.26 It is a schematic diagram of the structure of an initial sparse convolutional network trained by the training method of the sparse neural network decision structure in some embodiments of the present application. DETAILED DESCRIPTION
[0039] The following is a further description of the embodiments of the present application in conjunction with the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements or elements with the same or similar functions from beginning to end. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the present application.
[0040] In recent years, with the development of computer technology, machine learning using neural networks has become a research hotspot, especially deep learning (DL), which is a new research direction in the field of machine learning. It is introduced into machine learning to make it closer to its original goal - artificial intelligence (AI). Among them, convolutional neural network (CNN) is a type of neural network. It is a type of feedforward neural network (FNN) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural network has the ability to represent learning and can perform shift invariant classification on input information according to its hierarchical structure. Therefore, it is also called "Shift Invariant Artificial Neural Network (SIANN)".
[0041] DL algorithms represented by neural networks (NN) such as CNN have been widely used in tasks such as image classification, image recognition, target detection, depth estimation, image segmentation and super-resolution in the fields of machine vision, autonomous driving, and image processing. However, due to its complex structure, the convolutional neural network model will lead to a huge amount of parameters and computing resource overhead, and it is difficult to deploy on edge computing platforms where memory and computing resources are scarce. In practical applications, in addition to the demand for high-precision algorithms, a large number of application fields also have strict requirements on the running speed of the algorithms. Under the condition of certain computing resources, there is a contradiction between the computing power consumption and running efficiency of the convolutional neural network model. How to balance the accuracy performance and running speed of the convolutional neural network has become a problem that technicians in this field need to solve urgently. In order to solve this problem, the present application provides a data processing method based on a sparse neural network decision structure ( Figure 1), sparse neural network decision structure ( Figure 2 ), a data processing device based on a sparse neural network decision structure ( Fig.10 , Fig.17 and Fig. 20 ), training method of sparse neural network decision structure ( Fig.25 Display), chip ( Fig. 22 Display), electronic equipment ( Fig.23 Display) and computer storage media ( Fig.24 Show).
[0042] See also Figure 1 , a data processing method based on a sparse neural network decision structure provided in this application includes:
[0043] 01: Perform a first processing on the input feature to obtain first feature data;
[0044] 02: Performing a second process on the first feature data to obtain second feature data; and
[0045] 03: Perform a third processing on the second feature data to obtain third feature data.
[0046] Please combine Figure 2 and Figure 3 The present application provides a sparse neural network decision structure, and the above data processing method can be applied to the sparse neural network decision structure. Specifically, the sparse neural network decision structure includes a first convolution layer module, a second convolution layer module and a third convolution layer module. The first convolution layer module is used to perform a first processing on the input feature to obtain the first feature data, the first processing includes a first convolution processing using a first convolution kernel, the first convolution kernel is a sparse convolution kernel with a first sparse coefficient, and the dimension of the first feature data is higher than the dimension of the input feature. The second convolution layer module is used to perform a second processing on the first feature data to obtain the second feature data, the second processing includes a second convolution processing using a second convolution kernel, the second convolution kernel is a sparse convolution kernel with a second sparse coefficient. The third convolution layer module is used to perform a third processing on the second feature data to obtain the third feature data, the third processing includes a third convolution processing using the first convolution kernel, the third convolution kernel is a sparse convolution kernel with a third sparse coefficient, and the dimension of the third feature data is lower than the dimension of the second feature data. The sparse coefficient is the proportion of the number of non-zero weights in the convolution kernel to the total number of weights.
[0047] Among them, "input features" can be image data, such as images to be detected with bad pixels to be detected, image data to be de-mosaiced, optical flow images or binocular vision images that need to be matched for image blocks, etc. The input feature can be a tensor of [H, W, C], where the length of the input feature is H, the width is W, and the number of channels is C. Figure 4In the illustrated embodiment, the number of channels is 3, but it is understandable that the number of channels of the input features may be other numbers.
[0048] The first processing includes a first convolution processing using a first convolution kernel. Figure 5 ,for Figure 4 Schematic diagram of the convolution kernel in the data processing method shown. Among them, the first convolution kernel is a sparse convolution kernel with a first sparse coefficient S1, and the sparse coefficient is the proportion of the number of non-zero weights in the convolution kernel to the total number of weights. Specifically, in an embodiment of the present application, the first convolution kernel is a 1×1 sparse convolution kernel, and the number of first convolution kernels is multiple, for example, 5. At this time, the expansion coefficient E=5, and the 5 first convolution kernels are the first convolution kernel K11, the first convolution kernel K12, the first convolution kernel K13, the first convolution kernel K14 and the first convolution kernel K15. In each first convolution kernel, there are most zero-valued weights, and there are a few non-zero weights, such as Figure 5 As shown, the weight values shown by the shadows are non-zero weights, and the weight values shown by the other non-shaded ones are zero weights. Compared with the situation where the weight values in the traditional convolution kernel are basically non-zero weights, the weight arrangement of the first convolution kernel of the present application is sparse, and has a reasonable first sparse coefficient S1, which can not only save the amount of calculation for performing multiplication and addition calculations with the input features during the first convolution processing, improve the running speed, but also ensure the processing accuracy of the first convolution processing. In addition, the first sparse coefficients S1 of each first convolution kernel can be the same or different. For example, the first sparse coefficient S1 of the first convolution kernel K11, the first sparse coefficient S1 of the first convolution kernel K12, the first sparse coefficient S1 of the first convolution kernel K14, and the first sparse coefficient S1 of the first convolution kernel K15 are all 1 / 3, and the first sparse coefficient S1 of the first convolution kernel K13 is 2 / 3.
[0049] The second process includes a second convolution process using a second convolution kernel. Figure 4 and Figure 5 , the second convolution kernel is a sparse convolution kernel with a second sparse coefficient. Specifically, in the embodiment of the present application, the second convolution kernel is a 3×3 sparse convolution kernel, and the number of the second convolution kernels is multiple, for example, 5, and the 5 second convolution kernels are second convolution kernel K21, second convolution kernel K22, second convolution kernel K23, second convolution kernel K24 and second convolution kernel K25. In each second convolution kernel, there are most zero-valued weights, and there are a few non-zero weights, such as Figure 5As shown, the weight values shown by the shadows are non-zero weights, and the weight values shown by the other non-shaded ones are zero weights. Compared with the situation where the weight values in the traditional convolution kernel are basically non-zero weights, the weight arrangement of the second convolution kernel of the present application is sparse, and has a reasonable second sparse coefficient S2, which can not only save the amount of calculation for performing multiplication and addition calculations with the first feature data during the second convolution processing, improve the running speed, but also ensure the processing accuracy of the second convolution processing. In addition, the second sparse coefficients S2 of each second convolution kernel can be the same or different. For example, the second sparse coefficient S2 of the second convolution kernel K21 and the second sparse coefficient S2 of the second convolution kernel K25 are both 1 / 3, the second sparse coefficient S2 of the second convolution kernel K22 and the second sparse coefficient S2 of the second convolution kernel K24 are both 2 / 9, and the second sparse coefficient S2 of the third convolution kernel K23 is 4 / 9.
[0050] The third process includes a third convolution process using a third convolution kernel. Figure 4 and Figure 5 , the third convolution kernel is a sparse convolution kernel with a third sparse coefficient. Specifically, in the implementation of the present application, the third convolution kernel is a 1×1 sparse convolution kernel, and the number of the third convolution kernels is multiple, for example, 3, and the three third convolution kernels are respectively a third convolution kernel K31, a third convolution kernel K32, and a third convolution kernel K33. In each third convolution kernel, there are most zero-valued weights, and there are a few non-zero weights, such as Figure 5 As shown, the weight values shown by the shadows are non-zero weights, and the weight values shown by the other non-shaded ones are zero weights. Compared with the situation where the weight values in the traditional convolution kernel are basically non-zero weights, the weight arrangement of the third convolution kernel of the present application is sparse, and has a reasonable third sparse coefficient S3, which can not only save the amount of calculation for performing multiplication and addition calculations with the second feature data during the third convolution processing, improve the running speed, but also ensure the processing accuracy of the third convolution processing. In addition, the third sparse coefficient S3 of each third convolution kernel can be the same or different. For example, the third sparse coefficient S3 of the third convolution kernel K31 and the third sparse coefficient S3 of the third convolution kernel K32 are both 2 / 5, and the third sparse coefficient S3 of the third convolution kernel K33 is 1 / 5.
[0051] In addition, it should be noted that the sparse coefficient of each convolution kernel can be related to the number of parameters in each convolution layer (one convolution layer performs the first processing, one convolution layer performs the second processing, and one convolution layer performs the third processing). The convolution layer with a larger number of parameters is given a greater sparsity, that is, the smaller the sparse coefficient.
[0052] The dimension of the first feature data (5 dimensions) is higher than the dimension of the input feature (3 dimensions), that is, the first processing is equivalent to dimensionality increase processing. The dimension of the third feature data (3 dimensions) is lower than the dimension of the second feature data (5 dimensions), that is, the second processing is equivalent to dimensionality reduction processing. The data processing method based on the sparse neural network decision structure and the sparse neural network decision structure in the implementation mode of the present application respectively use a sparse first convolution kernel, a sparse second convolution kernel and a sparse third convolution kernel in the first processing, the second processing and the third processing. The weights in these sparse convolution kernels are arranged sparsely and have reasonable sparse coefficients, which can save the amount of calculation of multiplication and addition calculations performed with the input features or feature data in each convolution processing process, improve the running speed, and at the same time ensure the processing accuracy of the convolution processing, and well balance the accuracy performance and running speed of the convolution neural network.
[0053] See also Figure 6 In some embodiments, 01: performing a first processing on the input feature to obtain first feature data, including:
[0054] 011: Perform the first convolution process on the input features;
[0055] 013: Performing a first batch of normalization processing on the data after the first convolution processing; and
[0056] 015: Perform a first activation process on the first batch of normalized data to output first feature data.
[0057] See also Figure 2 The first convolution layer module includes a first convolution unit, a first normalization unit and a first activation unit. The first convolution unit is used to perform a first convolution process on the input feature, and the first convolution process uses a shift approximate multiplication method to perform convolution. The first normalization unit is used to perform a first normalization process on the data after the first convolution process. The first activation unit is used to perform a first activation process on the data after the first normalization process to output the first feature data.
[0058] See also Figure 3, the first convolution processing is a 1×1 sparse convolution processing, which is a Pointwise convolution. Among them, the first convolution processing uses the shift approximate multiplication method to perform convolution. The first batch of normalization processing is to normalize the data after the first convolution processing, so that the data becomes a distribution with a mean of 0 and a standard deviation of 1 or a distribution ranging from 0 to 1. In the case where the data after the first convolution processing is not normalized, the neural network may learn slowly or even be difficult to learn due to the scattered distribution of sample features. The present application performs batch normalization processing on the data after the first convolution processing, which can accelerate model training and even improve model accuracy. The first activation processing uses a nonlinear activation function. Among them, the nonlinear activation function can be a ReLU6 activation function. If the ReLU activation function is used, the output range of the output value is 0 to positive infinity, then the low-precision float16 / int8 cannot accurately describe such a large range of values, resulting in a loss of accuracy. The first activation processing of the present application uses the ReLU6 activation function to limit the maximum output value to 6, which can ensure good numerical resolution and avoid precision loss even when the mobile end has low precision of float16 / int8.
[0059] See also Figure 7 In some embodiments, 02: performing a second process on the first feature data to obtain second feature data, comprising:
[0060] 021: Perform a second convolution process on the first feature data;
[0061] 023: performing a second batch of normalization processing on the data after the second convolution processing; and
[0062] 025: Perform a second activation process on the second batch of normalized data to output second feature data.
[0063] See also Figure 2 The second convolution layer module includes a second convolution unit, a second batch normalization unit and a second activation unit. The second convolution unit is used to perform a second convolution process on the first feature data, and the second convolution process uses a shift approximate multiplication method to perform convolution. The second batch normalization unit is used to perform a second batch normalization process on the data after the second convolution process. The second activation unit is used to perform a second activation process on the data after the second batch normalization process to output the second feature data.
[0064] See also Figure 3, the second convolution processing is 3×3 sparse convolution processing, which is Depthwise convolution. Among them, the second convolution processing uses shift approximate multiplication to perform convolution. The second batch of normalization processing is to normalize the data after the second convolution processing, so that the data becomes a distribution with a mean of 0 and a standard deviation of 1 or a distribution ranging from 0 to 1. Once the network is trained, the parameters must be updated. In addition to the data in the input layer (the data in the input features), the input data distribution of each layer of the subsequent network is constantly changing, because during training, the update of the training parameters of the previous layer will lead to changes in the input data distribution of the subsequent layers. Take the second layer of the network as an example: the second layer input of the network is calculated by the parameters and input features of the first layer, and the parameters of the first layer are constantly changing throughout the training process, so it will inevitably cause changes in the input data distribution of each subsequent layer. This application not only performs the first batch of normalization processing on the data after the first convolution processing, but also performs the second batch of normalization processing on the data after the second convolution processing, which can accelerate model training and even improve model accuracy. The second activation processing uses a nonlinear activation function. Among them, the nonlinear activation function can be a ReLU6 activation function. As before, the second activation process of the present application uses the ReLU6 activation function to limit the maximum output value to 6, which can ensure that even when the mobile end float16 / int8 has low precision, it can also have good numerical resolution and avoid precision loss.
[0065] See also Figure 8 In some embodiments, 03: performing a third process on the second feature data to obtain third feature data, including:
[0066] 031: Perform a third convolution process on the second feature data;
[0067] 033: performing a third batch of normalization processing on the data after the third convolution processing; and
[0068] 035: Perform a third activation process on the third batch of normalized data to output third feature data.
[0069] See also Figure 2 The third convolution layer module includes a third convolution unit, a third batch normalization unit and a third activation unit. The third convolution unit is used to perform a third convolution process on the second feature data, and the third convolution process uses a shift approximate multiplication method to perform convolution. The third batch normalization unit is used to perform a third batch normalization process on the data after the third convolution process. The third activation unit is used to perform a third activation process on the data after the third batch normalization process to output the third feature data.
[0070] See also Figure 3, the third convolution processing is a 1×1 sparse convolution processing, which is a Pointwise convolution. Among them, the third convolution processing uses shift approximate multiplication to perform convolution. The third batch of normalization processing is to normalize the data after the third convolution processing, so that the data becomes a distribution with a mean of 0 and a standard deviation of 1 or a distribution ranging from 0 to 1. This application not only performs the first batch of normalization processing on the data after the first convolution processing, but also performs the second batch of normalization processing on the data after the second convolution processing, and performs the third batch of normalization processing on the data after the third convolution processing, so as to further accelerate model training and improve model accuracy. The third activation processing uses a linear activation function.
[0071] In addition, for the first convolution process, the second convolution process, and the third convolution process, "convolution is performed by shift approximate multiplication", specifically: add 2 to the weight values in the convolution kernel. n By introducing the shift approximate multiplication method, the convolution multiplication can be replaced by shift. Taking the first convolution process as an example, if the weight values in the first convolution kernel K11 are W111=4, W112=2, and W113=8, and the corresponding input feature data are 3, 1, and 2, respectively, the traditional convolution process is: 4×3+2×1+8×2; this application adopts the shift approximate multiplication method, and first add 2 to the weight values in K11. n Constraints, such as W111=2 2 , W112=2 1 , W113=2 3 , that is, 2 2 ×3+2 1 ×1+2 3 ×2, at this time, the first convolution processing of the present application becomes: 3<<2+1<<1+2<<3, that is, convert 3, 1, and 2 into binary first, and then shift the binary 3 (0011) left by 2 bits to 1100, shift the binary 1 (0001) left by 1 bit to 0010, and shift the binary 2 (0010) left by 3 bits to 10000. In this way, the traditional multiplication calculation can be simplified into a shift operation. Compared with the multiplication calculation, the calculation amount, cost, and power consumption of the shift operation are much smaller. The following Table 1 lists the difference in the amount of calculation of the 1x1 sparse convolution processing and the 3x3 sparse convolution processing of the present application relative to the traditional 1x1 convolution and the traditional 3x3 convolution.
[0072] Table 1
[0073] multiplication addition Shift Traditional 1x1 convolution <![CDATA[EC 2 ]]> <![CDATA[EC 2 ]]> 0 Sparse 1x1 shifted convolution 0 <![CDATA[SEC 2 ]]> <![CDATA[SEC 2 ]]> Traditional DepthWise convolution (3x3) <![CDATA[EK 2 C]]> <![CDATA[EK 2 C]]> 0 Sparse DepthWise Convolution (3x3) 0 <![CDATA[SEK 2 C]]> <![CDATA[SEK 2 C]]>
[0074] Among them, C is the number of channels of the input feature, E is the expansion coefficient, S is the sparse coefficient, and K is the convolution kernel size. From Table 1, we can see that the sparse 1x1 shift convolution reduces the computational complexity by 2(1-S)EC compared to the traditional 1x1 convolution. 2 Compared with the traditional DepthWise convolution, the sparse DepthWise convolution reduces the computational complexity by 2(1-S)EK 2 C. In other words, the first convolution processing and the third convolution processing of the present application reduce the computational complexity by 2(1-S)EC compared to the traditional 1x1 convolution. 2 The second convolution process reduces the computational effort by 2(1-S)EK compared to the traditional DepthWise convolution. 2 C. Therefore, compared with traditional data processing methods and traditional neural network decision structures, the data processing method and sparse neural network decision structure of the present application have much smaller computational complexity. Correspondingly, the running speed is further accelerated, and the accuracy of the output results will not be reduced.
[0075] In addition, it should be noted that the amount of calculation in Table 1 above does not include the multiplication and addition of the batch normalization layer. It can be understood that in some embodiments, when the batch normalization module performs the batch normalization operation, the multiplication used can also adopt the above-mentioned shift approximate multiplication method, thereby reducing the amount of calculation of the batch normalization process in the first processing, the second processing or the third processing, and further improving the running speed of the data processing method and the sparse neural network decision structure of the present application.
[0076] See also Fig. 9 In some embodiments, the data processing method further comprises:
[0077] 04: Perform residual processing on the third feature data and the input feature to obtain the output feature.
[0078] See also Figure 2 ,The sparse neural network decision structure also includes a residual module, the residual module connects the input feature and the third feature data, and the residual module is used to perform residual processing on the third feature data and the input feature to obtain the output feature.
[0079] Residual processing can solve the network degradation problem of the sparse neural network decision structure, and ensure that the performance of the output feature will not be worse than that of the input feature. In addition, the sparse neural network decision structure of the present application may not include a residual module, that is, no residual processing is performed, and the final output is the third feature data.
[0080] See also Fig.10The present application provides a data processing device based on a sparse neural network decision structure, including any of the above-mentioned sparse neural network decision structures. The sparse neural network decision structure can be used as a decision maker in a data processing device and applied to various image data processing tasks, such as the probability of bad pixel pixels in the image to be detected, the weight of the interpolation strategy of bad pixel pixels, the weight of the interpolation strategy of pixels in the image data to be de-mosaiced, and the matching of image blocks in optical flow images or binocular vision images.
[0081] In some embodiments, the sparse neural network decision structure can be used to determine the probability of a bad pixel in the image to be detected. A bad pixel generally refers to a situation where the sensor exhibits abnormal brightness due to circuit abnormalities during the sensor manufacturing process. To detect whether a pixel is a bad pixel, refer to Fig.11 , the data processing method further includes:
[0082] 05: Perform a fourth convolution process on the output feature using the fourth convolution kernel to obtain fourth feature data of category 1; and
[0083] 06: Perform a fourth activation process on the fourth feature data to output the bad pixel probability of each pixel in the input feature.
[0084] Please combine Fig.10 and Fig.12 The data processing device further includes a fourth convolution unit and a fourth activation unit. The fourth convolution unit is used to perform a fourth convolution process on the third feature data or the output feature using a fourth convolution kernel to obtain fourth feature data of type 1, and the fourth convolution kernel is a sparse convolution kernel with a fourth sparse coefficient. The fourth activation unit is used to perform a fourth activation process on the fourth feature data to output the bad pixel probability of each pixel in the input feature.
[0085] The fourth convolution kernel is a sparse convolution kernel with a fourth sparse coefficient. Specifically, the fourth convolution kernel is a 1×1 sparse convolution kernel, the fourth convolution kernel is one, and the fourth sparse coefficient S4 of the fourth convolution kernel can be different from the first sparse coefficient S1 to the third sparse coefficient S3. As before, the weight arrangement of the fourth convolution kernel of the present application is sparse, and has a reasonable fourth sparse coefficient S4, which can save the amount of calculation of performing multiplication and addition calculations with the input features (output features or third feature data) during the fourth convolution process, improve the running speed, and ensure the processing accuracy of the fourth convolution process. The fourth activation process uses a nonlinear activation function. Among them, the nonlinear activation function can be a Sigmoid activation function. After the Sigmoid activation function, the output value is a probability value between 0 and 1. When the probability value is larger, it indicates that the probability that the pixel in the input feature (image to be detected) is a bad pixel is higher. In one example, a probability threshold may be preset. When the probability value is greater than the probability threshold, it indicates that the corresponding pixel is a bad pixel. When the probability value is less than or equal to the probability threshold, it indicates that the corresponding pixel is a non-bad pixel.
[0086] For the input features, in addition to detecting which pixels are bad pixels, if there are bad pixels, the pixel values of the bad pixels need to be compensated to ensure that the input features can be fully expressed. Therefore, in some embodiments, the sparse neural network decision structure can be used to determine the weight of the interpolation strategy for the bad pixels of the image to be detected. In this case, please refer to Fig.13 , the data processing method may further include:
[0087] 07: Performing a fifth convolution process on the output features using the fifth convolution kernel to obtain fifth feature data of class m; and
[0088] 08: Perform a fifth activation process on the fifth feature data to output the weight of the interpolation strategy for each bad pixel in the input feature.
[0089] Please combine Fig.10 and Fig.12 The data processing device further includes a fifth convolution unit and a fifth activation unit. The fifth convolution unit is used to perform a fifth convolution process on the third feature data or the output feature using a fifth convolution kernel to obtain fifth feature data of class m, and the fifth convolution kernel is a sparse convolution kernel with a fifth sparse coefficient. The fifth activation unit is used to perform a fifth activation process on the fifth feature data to output the weight of the interpolation strategy for each bad pixel in the input feature.
[0090] The fifth convolution kernel is a sparse convolution kernel with a fifth sparse coefficient. Specifically, the fifth convolution kernel is a 1×1 sparse convolution kernel, the fifth convolution kernel is m, and the fifth sparse coefficient S5 of the fifth convolution kernel can be different from the first sparse coefficient S1 to the fourth sparse coefficient S4. As before, the weight arrangement of the fifth convolution kernel of the present application is sparse, and has a reasonable fifth sparse coefficient S5, which can not only save the amount of calculation of multiplication and addition calculations performed with the input features (output features or third feature data) during the fifth convolution process, improve the running speed, but also ensure the processing accuracy of the fifth convolution process. The fifth activation process uses a nonlinear activation function. Among them, the nonlinear activation function can be a Softmax activation function. After the Softmax activation function, the output value is a value between 0 and 1, which is the weight of the interpolation strategy of the bad pixel. For example, the interpolation strategy of the bad pixel includes four types: right diagonal interpolation (45°), left diagonal interpolation (145°), vertical interpolation, and horizontal interpolation. like Fig.14 As shown, if the pixel point (1, 1) is a bad pixel, in order to obtain the normal pixel value of the pixel point (1, 1), the pixel point (0, 2) and the pixel point (2, 0) on the right diagonal (45°) line can be used for interpolation, for example, Px = (P3 + P7) / 2; the pixel point (0, 0) and the pixel point (2, 2) on the left diagonal (145°) line can also be used for interpolation, for example, Px = (P1 + P9) / 2; the pixel point (0, 1) and the pixel point (2, 1) in the vertical direction can also be used for interpolation, for example, Px = (P2 + P8) / 2; the pixel point (1, 0) and the pixel point (1, 2) in the horizontal direction can also be used for interpolation, for example, Px = (P4 + P6) / 2. If the output value after the Softmax activation function is [1 / 2, 2 / 3, 9 / 10, 1 / 3], it means that after the decision of the data processing device, the interpolation direction of the bad pixel should be the one with the largest weight, i.e. 9 / 10, that is, interpolation should be performed in the vertical direction.
[0091] For further information, see Fig.15 In some embodiments, the data processing method further comprises:
[0092] When the input feature is an image to be detected, the confidence of the traditional bad pixel detection strategy is obtained;
[0093] When the confidence level is greater than a predetermined threshold, a conventional bad pixel detection strategy is used to perform bad pixel detection and a conventional bad pixel interpolation strategy is used to perform interpolation on the bad pixel to obtain a pixel value corresponding to the bad pixel;
[0094] When the confidence level is less than a predetermined threshold, the data processing device executes the method in steps 01 to 08.
[0095] The data processing device also includes a confidence acquisition unit and a bad pixel interpolation module. The confidence acquisition unit is used to obtain the confidence of the traditional bad pixel detection strategy when the input feature is an image to be detected. When the confidence is greater than a predetermined threshold, the bad pixel interpolation module is used to perform bad pixel detection using a traditional bad pixel detection strategy and perform interpolation on the bad pixel using a traditional bad pixel pixel interpolation strategy to obtain the pixel value of the corresponding bad pixel; when the confidence is less than the predetermined threshold, the data processing device executes the methods in 01 to 08, specifically, the sparse neural network decision structure executes the methods in 01 to 04, and the fourth convolution unit, the fourth activation unit, the fifth convolution unit and the fifth activation unit respectively execute the methods in 05, 06, 07 and 08.
[0096] The accuracy of the output obtained after the sparse neural network decision structure, the fourth convolution unit, the fourth activation unit, the fifth convolution unit and the fifth activation unit execute the methods 01 to 08 is higher than the accuracy of the output obtained after the traditional bad pixel detection strategy executes bad pixel detection and the traditional bad pixel pixel interpolation strategy performs interpolation on the bad pixel. However, in the process of the sparse neural network decision structure, the fourth convolution unit, the fourth activation unit, the fifth convolution unit and the fifth activation unit executing the methods 01 to 08, the amount of calculation is larger than the traditional bad pixel detection method and the traditional interpolation method, and the power consumption is higher. Therefore, the data processing method and data processing device of the implementation mode of the present application can first obtain the confidence of the traditional bad pixel detection strategy. If the confidence of the traditional bad pixel detection strategy is high (higher than the predetermined threshold), it indicates that the traditional bad pixel detection strategy can meet the high-precision output. If the confidence of the traditional bad pixel detection strategy is low (lower than the predetermined threshold), it indicates that the traditional bad pixel detection strategy cannot meet the high-precision output. At this time, the sparse neural network decision structure, the fourth convolution unit, the fourth activation unit, the fifth convolution unit and the fifth activation unit of the present application are used to execute the methods in 01 to 08 to obtain a higher-precision output, thereby satisfying both high-precision output and low calculation and low power consumption, and achieving a better balance between accuracy, computing power and power consumption.
[0097] In some embodiments, the sparse neural network decision structure can be applied to the demosaicing process of the decision mosaic image, see Fig.16 and Fig.17 In some embodiments, the data processing method further comprises:
[0098] 09: Performing a sixth convolution process on the output feature using a sixth convolution kernel to obtain sixth feature data of class m; and 010: Performing a sixth activation process on the sixth feature data to output the weight of the interpolation strategy of the pixel values of the mosaic pixels in the input feature.
[0099] Please combine Fig.18The data processing device further includes a sixth convolution unit and a sixth activation unit. The sixth convolution unit is used to perform a sixth convolution process on the output feature using a sixth convolution kernel to obtain sixth feature data of class m, and the sixth convolution kernel is a sparse convolution kernel with a sixth sparse coefficient; the sixth activation unit is used to perform a sixth activation process on the sixth feature data to output the weight of the interpolation strategy of the pixel value of the mosaic pixel in the input feature.
[0100] The sixth convolution kernel is a sparse convolution kernel with a sixth sparse coefficient. Specifically, the sixth convolution kernel is a 1×1 sparse convolution kernel, and there are m sixth convolution kernels. The sixth sparse coefficient S6 of the sixth convolution kernel may be different from the first sparse coefficient S1 to the fifth sparse coefficient S5. As before, the weight arrangement of the sixth convolution kernel of the present application is sparse, and has a reasonable fifth sparse coefficient S6, which can save the amount of multiplication and addition calculations performed with the input features (output features or third feature data) during the sixth convolution processing, improve the running speed, and at the same time ensure the processing accuracy of the sixth convolution processing. The fifth activation process uses a nonlinear activation function. Among them, the nonlinear activation function can be a Softmax activation function. After the Softmax activation function, the output value is between 0 and 1, which is the weight of the interpolation strategy of the bad pixel. For example, the interpolation strategy of mosaic pixels includes four types: right diagonal interpolation (45°), left diagonal interpolation (145°), vertical interpolation, and horizontal interpolation. Fig.14 As shown, if the pixel point (1, 1) is a mosaic pixel, then in order to obtain the pixel value of the pixel point (1, 1) after de-mosaicing, the pixel point (0, 2) and the pixel point (2, 0) on the right diagonal (45°) line can be used for interpolation, for example, Px = (P3 + P7) / 2; the pixel point (0, 0) and the pixel point (2, 2) on the left diagonal (145°) line can also be used for interpolation, for example, Px = (P1 + P9) / 2; the pixel point (0, 1) and the pixel point (2, 1) in the vertical direction can also be used for interpolation, for example, Px = (P2 + P8) / 2; the pixel point (1, 0) and the pixel point (1, 2) in the horizontal direction can also be used for interpolation, for example, Px = (P4 + P6) / 2. If the output value after the Softmax activation function is [1 / 2, 2 / 3, 9 / 10, 1 / 3], it means that after the decision of the data processing device, the interpolation direction of the mosaic pixel should be the one with the largest weight, i.e. 9 / 10, that is, interpolation should be performed in the vertical direction.
[0101] For further information, see Fig.19 In some embodiments, the data processing method further comprises:
[0102] For the mosaic pixels in the input features, the confidence of the traditional mosaic pixel interpolation strategy is obtained;
[0103] When the confidence level is greater than a preset threshold, a traditional mosaic pixel interpolation strategy is used for the pixel values of the mosaic pixels in the input feature;
[0104] When the confidence level is less than a preset threshold, a sixth convolution process and a sixth activation process are performed on the output feature to obtain the weight of the interpolation strategy for the pixel values of the mosaic pixels in the input feature.
[0105] The data processing device also includes a confidence acquisition unit and a decision unit. The confidence acquisition unit is used to obtain the confidence of the traditional mosaic pixel interpolation strategy for the mosaic pixels in the input features. For the pixel values of the mosaic pixels in the input features, when the confidence is greater than a preset threshold, the decision unit is used to determine to adopt the traditional mosaic pixel interpolation strategy; when the confidence is less than the preset threshold, the decision unit is used to determine that the data processing device executes the methods in 01 to 05, 09, and 010. Specifically, the sparse neural network decision structure executes the methods in 01 to 05, the sixth convolution unit performs the sixth convolution processing on the third feature data or the output feature, and the sixth activation unit performs the sixth activation processing to obtain the weight of the interpolation strategy for the pixel values of the mosaic pixels in the input features.
[0106] The purpose of the demosaicing algorithm is to reconstruct a full-color image from the incomplete color samples output by the photosensitive element covered with a color filter array (CFA). The traditional demosaicing algorithm (traditional mosaic pixel interpolation strategy) requires a lot of logic to judge the texture intensity and direction in local areas. Subsequent interpolation and post-processing (pseudo-color, purple fringing suppression) all rely on the texture direction properties of these local area blocks. When the traditional mosaic pixel interpolation strategy has problems with texture direction prediction, the processed image results may produce abnormalities. Therefore, the data processing method and data processing device of the implementation mode of the present application can first obtain the confidence of the traditional mosaic pixel interpolation strategy. If the confidence of the traditional mosaic pixel interpolation strategy is high (higher than the preset threshold), it indicates that the traditional mosaic pixel interpolation strategy can meet the high-precision output. If the confidence of the traditional mosaic pixel interpolation strategy is low (lower than the preset threshold), it indicates that the traditional mosaic pixel interpolation strategy cannot meet the high-precision output. At this time, the sparse neural network decision structure, the sixth convolution unit and the sixth activation unit of the present application are used to execute the methods in 01 to 05, 09, and 010 to obtain a higher-precision output, thereby satisfying both high-precision output and low calculation and low power consumption, and achieving a better balance between accuracy, computing power and power consumption.
[0107] In some embodiments, the sparse neural network decision structure can be applied to the matching of image patches in the decision optical flow image or binocular vision image. Fig. 20 and Fig.21 , the input features include candidate image blocks and matching image blocks, and the data processing method also includes:
[0108] Performing a seventh convolution process on the output features of the candidate image block and the output features of the matching image block, respectively, to obtain seventh feature data of the candidate image block and seventh feature data of the matching image block, respectively;
[0109] Performing a seventh activation process on the seventh feature data of the candidate image block and the seventh feature data of the matching image block, respectively, to output the eighth feature data of the candidate image block and the eighth feature data of the matching image block;
[0110] Binarizing the eighth feature data of the candidate image block and the eighth feature data of the matching image block to obtain two sets of binary sequences; and
[0111] The distance measurement between the candidate image blocks and the matching image blocks is obtained according to the two sets of binary sequences.
[0112] See also Fig. 20 The data processing device also includes a seventh convolution unit, a seventh activation unit, a binarization unit and a distance measurement acquisition unit. The seventh convolution unit is used to perform a seventh convolution process on the output feature using a seventh convolution kernel to obtain seventh feature data, and the seventh convolution kernel is a sparse convolution kernel with a seventh sparse coefficient. The seventh activation unit is used to perform a seventh activation process on the seventh feature data to output eighth feature data. The binarization unit is used to perform a binarization process on the eighth feature data to obtain a binary sequence. The distance measurement acquisition unit is used to obtain the distance measurement of the candidate image block and the matching image block according to the two sets of binary sequences.
[0113] Among them, the seventh convolution kernel is a sparse convolution kernel with a seventh sparse coefficient. Specifically, the fourth convolution kernel is a 1×1 sparse convolution kernel, and the seventh sparse coefficient S7 of the seventh convolution kernel can be different from the first sparse coefficient S1 to the sixth sparse coefficient S6. As before, the weight arrangement of the seventh convolution kernel of the present application is sparse, and has a reasonable seventh sparse coefficient S7, which can not only save the amount of calculation of multiplication and addition calculations performed with the input features (output features or third feature data) during the seventh convolution processing, improve the running speed, but also ensure the processing accuracy of the seventh convolution processing. The seventh activation process uses a nonlinear activation function. Among them, the nonlinear activation function can be a Softmax activation function. After the Softmax activation function, the output value is a value between 0 and 1. Then, after the eighth feature data is binarized, two sets of binary sequences are obtained, and the distance measurement acquisition unit uses the XOR sum method for the two sets of binary sequences to obtain the distance measurement of the candidate image block and the matching image block. For example, the two sets of binary sequences are:
[0114] [0, 1, 0, 1, 0, 0, 1, 0, 1, 1]; and
[0115] [0,0,0,0,0,0,1,0,1,1]
[0116] If the sum of the XORs is 8, it indicates that the distance measure between the candidate image block and the matching image block is 8. When the distance measure (8) exceeds a preset distance measure threshold (eg, 5), it can be considered that the candidate image block and the matching image block match.
[0117] See also Fig. 22 The present application provides a chip, which includes a memory and a processor. The memory is used to store a computer program that can be run on the processor. In one embodiment, the chip may include a data processing device based on a sparse neural network decision structure as described in any of the above embodiments. In this case, the processor is used to enable an electronic device equipped with the chip to run the data processing device as described in any of the above embodiments when running the computer program. In another embodiment, the processor is used to enable an electronic device equipped with the chip to execute the data processing method as described in any of the above embodiments when running the computer program.
[0118] Furthermore, the memory may be a separate device from the processor, or may be integrated into the processor.
[0119] The chip may also include an input interface. The processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips. The chip may also include an output interface. The processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.
[0120] See also Fig.23 The present application provides an electronic device, and the electronic device includes the above-mentioned chip. Specifically, the electronic device includes a memory and a processor. The memory is used to store a computer program that can be run on the processor. In one embodiment, the electronic device may include the data processing device based on the sparse neural network decision structure described in any of the above embodiments. In this case, the processor is used to enable the electronic device to run the data processing device described in any of the above embodiments when running the computer program. In another embodiment, the processor is used to enable the electronic device to execute the data processing method described in any of the above embodiments when running the computer program.
[0121] The memory may be a separate device independent of the processor, or may be integrated into the processor. Optionally, the electronic device may further include a transceiver, and the processor may control the transceiver to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver may include a transmitter and a receiver. The transceiver may further include an antenna, and the number of antennas may be one or more.
[0122] Optionally, the electronic device may specifically be a smart phone, a tablet computer, a PDA, a laptop computer, a desktop computer or the like, or a device integrating a data processing device based on a sparse neural network decision structure described in any one of the aforementioned embodiments.
[0123] See also Fig.24 The present application provides a computer storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the data processing method described in any of the above embodiments, or runs the data processing device described in any of the above embodiments.
[0124] See also Fig.25 The present application also provides a training method for a sparse neural network decision structure, the training method comprising:
[0125] Train an initial sparse convolutional network; and
[0126] Through incremental network quantization, the weight values of the convolution kernels in the first convolution layer, the weight values of the convolution kernels in the second convolution layer, and the weight values of the convolution kernels in the third convolution layer are all approximated to 2. n ; and merging the first batch of normalization and the first activation layer into a low-bit multiplication-addition operation, merging the second batch of normalization and the second activation layer into a low-bit multiplication-addition operation, and merging the third batch of normalization and the third activation layer into a low-bit multiplication-addition operation to obtain a sparse neural network decision structure.
[0127] Training an initial sparse convolutional network specifically includes: building a network model; adjusting the loss value and starting to read data; substituting vectors; and obtaining the network. Fig.26 , which is a schematic diagram of the structure of the initial sparse convolutional network. Specifically, the initial sparse convolutional network includes a first convolutional module, a second convolutional module and a third convolutional module. The first convolutional module includes a first convolutional layer, a first batch of normalization layers and a first activation layer, and the convolution kernel of the first convolutional layer is a sparse convolution kernel with a first sparse coefficient. The second convolutional module includes a second convolutional layer, a second batch of normalization layers and a second activation layer, and the convolution kernel of the second convolutional layer is a sparse convolution kernel with a second sparse coefficient. The third convolutional module includes a third convolutional layer, a third batch of normalization layers and a third activation layer, and the convolution kernel of the third convolutional layer is a sparse convolution kernel with a third sparse coefficient.
[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0129] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0130] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0131] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0132] A person of ordinary skill in the art will appreciate that all or part of the steps carried by the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.
[0133] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.
Claims
1. A data processing method based on a sparse neural network decision structure, characterized in that: include: Performing a first processing on the input feature to obtain first feature data, the first processing comprising a first convolution processing using a first convolution kernel, the first convolution kernel being a sparse convolution kernel having a first sparse coefficient, and the dimension of the first feature data being higher than the dimension of the input feature; Performing a second processing on the first feature data to obtain second feature data, the second processing comprising a second convolution processing using a second convolution kernel, the second convolution kernel being a sparse convolution kernel having a second sparse coefficient; and Performing a third processing on the second feature data to obtain third feature data, the third processing including a third convolution processing using a third convolution kernel, the third convolution kernel being a sparse convolution kernel with a third sparse coefficient, the dimension of the third feature data being lower than the dimension of the second feature data, and the sparse coefficient being the proportion of the number of non-zero weights in the convolution kernel to the total number of weights.
2. The data processing method according to claim 1, characterized in that: The first processing of the input feature to obtain first feature data includes: Performing the first convolution processing on the input feature, wherein the first convolution processing uses a shift approximate multiplication method to perform convolution; Performing a first batch of normalization processing on the data after the first convolution processing; and A first activation process is performed on the first batch of normalized data to output the first feature data.
3. The data processing method according to claim 1, characterized in that: The performing a second processing on the first feature data to obtain second feature data includes: Performing the second convolution processing on the first feature data, wherein the second convolution processing uses a shift approximate multiplication method to perform convolution; Performing a second batch normalization process on the data after the second convolution process; and A second activation process is performed on the second batch of normalized data to output the second feature data.
4. The data processing method according to claim 1, characterized in that: The performing a third processing on the second feature data to obtain third feature data includes: Performing the third convolution processing on the second feature data, wherein the third convolution processing uses a shift approximate multiplication method to perform convolution; Performing a third batch normalization process on the data after the third convolution process; and A third activation process is performed on the third batch of normalized data to output the third feature data.
5. The data processing method according to claim 1, characterized in that: Also includes: Residual processing is performed on the third feature data and the input feature to obtain an output feature.
6. The data processing method according to claim 5, characterized in that: Also includes: Performing a fourth convolution process on the output feature using a fourth convolution kernel to obtain fourth feature data of category 1, wherein the fourth convolution kernel is a sparse convolution kernel having a fourth sparse coefficient; and A fourth activation process is performed on the fourth feature data to output a bad pixel probability of each pixel in the input feature.
7. The data processing method according to claim 6, characterized in that: Also includes: Performing a fifth convolution process on the output feature using a fifth convolution kernel to obtain fifth feature data of m categories, wherein the fifth convolution kernel is a sparse convolution kernel with a fifth sparse coefficient; and A fifth activation process is performed on the fifth feature data to output the weight of the interpolation strategy for each bad pixel in the input feature.
8. The data processing method according to claim 5, characterized in that: Also includes: Performing a sixth convolution process on the output feature using a sixth convolution kernel to obtain sixth feature data of the m category, wherein the sixth convolution kernel is a sparse convolution kernel having a sixth sparse coefficient; and A sixth activation process is performed on the sixth feature data to output the weight of the interpolation strategy for the pixel values of the mosaic pixels in the input feature.
9. The data processing method according to claim 8, characterized in that: Also includes: For the mosaic pixels in the input features, obtaining the confidence of a traditional mosaic pixel interpolation strategy; When the confidence level is greater than a preset threshold, the traditional mosaic pixel interpolation strategy is adopted for the pixel value of the mosaic pixel in the input feature; When the confidence level is less than the preset threshold, the sixth convolution process and the sixth activation process are performed on the output feature to obtain the weight of the interpolation strategy of the pixel value of the mosaic pixel in the input feature.
10. The data processing method according to claim 5, characterized in that: The input features include candidate image blocks and matching image blocks, and the data processing method further includes: Performing a seventh convolution process on the output features of the candidate image block and the output features of the matching image block, respectively, to obtain seventh feature data of the candidate image block and seventh feature data of the matching image block, respectively, wherein the seventh convolution kernel is a sparse convolution kernel having a seventh sparse coefficient; performing a seventh activation process on the seventh feature data of the candidate image block and the seventh feature data of the matching image block, respectively, to output the eighth feature data of the candidate image block and the eighth feature data of the matching image block; Binarizing the eighth feature data of the candidate image block and the eighth feature data of the matching image block to obtain two sets of binary sequences; and A distance measure between the candidate image block and the matching image block is obtained according to the two groups of binary sequences.
11. A sparse neural network decision structure, characterized in that: include: A first convolutional layer module, the first convolutional layer module is used to perform a first process on the input feature to obtain first feature data, the first process includes a first convolution process using a first convolution kernel, the first convolution kernel is a sparse convolution kernel with a first sparse coefficient, and the dimension of the first feature data is higher than the dimension of the input feature; A second convolutional layer module, the second convolutional layer module is used to perform a second process on the first feature data to obtain second feature data, the second process includes a second convolution process using a second convolution kernel, and the second convolution kernel is a sparse convolution kernel with a second sparse coefficient; and A third convolutional layer module, the third convolutional layer module is used to perform a third processing on the second feature data to obtain third feature data, the third processing includes a third convolution processing using the third convolution kernel, the third convolution kernel is a sparse convolution kernel with a third sparse coefficient, the dimension of the third feature data is lower than the dimension of the second feature data, and the sparse coefficient is the proportion of the number of non-zero weights in the convolution kernel to the total number of weights.
12. The sparse neural network decision structure according to claim 11, characterized in that: The first convolutional layer module includes: A first convolution unit, the first convolution unit is used to perform the first convolution processing on the input feature, the first convolution processing uses a shift approximate multiplication method to perform convolution; a first batch normalization unit, the first batch normalization unit is used to perform a first batch normalization process on the data after the first convolution process; and A first activation unit, wherein the first activation unit is used to perform a first activation process on the first batch of normalized data to output the first feature data.
13. The sparse neural network decision structure according to claim 11, characterized in that: The second convolutional layer module includes: A second convolution unit, the second convolution unit is used to perform the second convolution processing on the first feature data, the second convolution processing uses a shift approximate multiplication method to perform convolution; a second batch normalization unit, configured to perform a second batch normalization process on the data after the second convolution process; and A second activation unit, wherein the second activation unit is used to perform a second activation process on the second batch of normalized data to output the second feature data.
14. The sparse neural network decision structure according to claim 11, characterized in that: The third convolutional layer module includes: A third convolution unit, the third convolution unit is used to perform the third convolution processing on the second feature data, the third convolution processing adopts a shift approximate multiplication method to perform convolution; a third batch normalization unit, configured to perform a third batch normalization process on the data after the third convolution process; and A third activation unit is used to perform a third activation process on the third batch of normalized data to output the third feature data.
15. The sparse neural network decision structure according to claim 11, characterized in that: Also includes: A residual module, wherein the residual module connects the input feature and the third feature data, and the residual module is used to perform residual processing on the third feature data and the input feature to obtain an output feature.
16. A data processing device based on a sparse neural network decision structure, characterized in that: The data processing device comprises: a sparse neural network decision structure as described in any one of claims 11-15.
17. The data processing device according to claim 16, characterized in that: The data processing device further comprises: a fourth convolution unit, the fourth convolution unit being configured to perform a fourth convolution process on the third feature data or the output feature using a fourth convolution kernel to obtain fourth feature data of type 1, the fourth convolution kernel being a sparse convolution kernel having a fourth sparse coefficient; and A fourth activation unit, wherein the fourth activation unit is used to perform a fourth activation process on the fourth feature data to output a bad pixel probability of each pixel in the input feature.
18. The data processing device according to claim 17, characterized in that: The data processing device further comprises: a fifth convolution unit, the fifth convolution unit being configured to perform a fifth convolution process on the third feature data or the output feature using a fifth convolution kernel to obtain fifth feature data of m categories, the fifth convolution kernel being a sparse convolution kernel having a fifth sparse coefficient; and A fifth activation unit, wherein the fifth activation unit is used to perform a fifth activation process on the fifth feature data to output an interpolation strategy for each bad pixel in the input feature.
19. The data processing device according to claim 16, characterized in that: The data processing device further comprises: a sixth convolution unit, the sixth convolution unit being used to perform a sixth convolution process on the output feature using a sixth convolution kernel to obtain sixth feature data of m categories, the sixth convolution kernel being a sparse convolution kernel having a sixth sparse coefficient; and A sixth activation unit, the sixth activation unit is used to perform a sixth activation process on the sixth feature data to output the weight of the interpolation strategy of the pixel value of the mosaic pixel in the input feature.
20. The data processing device according to claim 19, characterized in that The data processing device further comprises: A confidence acquisition unit, the confidence acquisition unit being used to acquire the confidence of a traditional mosaic pixel interpolation strategy for the mosaic pixels in the input feature; and A decision unit, for the pixel value of the mosaic pixel in the input feature, when the confidence level is greater than a preset threshold, the decision unit is used to determine to adopt the traditional mosaic pixel interpolation strategy; when the confidence level is less than the preset threshold, the decision unit is used to determine that the sixth convolution unit performs the sixth convolution processing on the third feature data or the output feature and the sixth activation unit performs the sixth activation processing to obtain the weight of the interpolation strategy for the pixel value of the mosaic pixel in the input feature.
21. The data processing device according to claim 16, characterized in that: The data processing device further comprises: a seventh convolution unit, the seventh convolution unit being used to perform a seventh convolution process on the output features of the candidate image block and the output features of the matching image block using a seventh convolution kernel to obtain seventh feature data, the seventh convolution kernel being a sparse convolution kernel having a seventh sparse coefficient; and a seventh activation unit, configured to perform a seventh activation process on the seventh feature data to output eighth feature data; and A binarization unit, the binarization unit is used to perform binarization processing on the eighth feature data to obtain two sets of binary sequences; and A distance measurement acquisition unit is used to acquire the distance measurement of the candidate image block and the matching image block according to the two groups of binary sequences.
22. A training method for a sparse neural network decision structure, characterized in that: The training method comprises: Train an initial sparse convolutional network, the initial sparse convolutional network comprising a first convolutional module, a second convolutional module and a third convolutional module, the first convolutional module comprising a first convolutional layer, a first batch of normalization layers and a first activation layer, the convolution kernel of the first convolutional layer is a sparse convolution kernel with a first sparse coefficient, the second convolutional module comprising a second convolutional layer, a second batch of normalization layers and a second activation layer, the convolution kernel of the second convolutional layer is a sparse convolution kernel with a second sparse coefficient, the third convolutional module comprising a third convolutional layer, a third batch of normalization layers and a third activation layer, the convolution kernel of the third convolutional layer is a sparse convolution kernel with a third sparse coefficient; The weight values of the convolution kernels in the first convolution layer, the second convolution layer, and the third convolution layer are all approximated to 2 by means of incremental network quantization. n ; and merging the first batch of normalization and the first activation layer into a low-bit multiplication-addition operation, merging the second batch of normalization and the second activation layer into a low-bit multiplication-addition operation, and merging the third batch of normalization and the third activation layer into a low-bit multiplication-addition operation to obtain the sparse neural network decision structure.
23. A chip, characterized in that: The chip includes a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is used to enable the electronic device equipped with the chip to execute the data processing method described in any one of claims 1 to 10, or to run the data processing device described in any one of claims 16 to 21 when running the computer program.
24. An electronic device, characterized in that: The electronic device comprises a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is used to execute the data processing method described in any one of claims 1 to 10, or to run the data processing device described in any one of claims 16 to 21 when running the computer program.
25. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by at least one processor, it implements the data processing method described in any one of claims 1 to 10, or runs the data processing device described in any one of claims 16 to 21.
Citation Information
Patent Citations
Sparse convolutional neural network sorting method, operation method and device and equipment
CN112200295A
Accelerator and acceleration method for sparse convolutional neural network
CN115640833A
Model training method and apparatus
WO2022111617A1
