Method, device and computer-readable storage medium for computing using convolutional neural network

Abstract

The present invention relates to a method, device and computer-readable storage medium for computing using a convolutional neural network. A method for computing using a convolutional neural network is disclosed, comprising: receiving feature data with different attributes input from at least two channels; determining the sparsity of each of the at least two channels based on the attributes of the received feature data; rearranging the at least two channels according to the determined sparsity, so that the sparse channels in the at least two channels are arranged continuously, and the sparsity of the sparse channels is greater than or equal to a preset value; adjusting the parameters of a trained convolutional neural network model based on the rearranged channels, and using the adjusted convolutional neural network model to perform convolution calculations on the feature data of the sparse channels and non-sparse channels separately, and the sparsity of the non-sparse channels is less than the preset value.

Description

Technical Field

[0001] The present invention relates to using convolutional neural networks (CNNs) for computing, and more particularly to methods, devices, and computer-readable storage media for using convolutional neural networks for computing. Background Art

[0002] Convolutional neural networks are widely used for calculation and prediction in the fields of game AI, image recognition, object detection, etc. In game AI, convolutional neural networks can be used to calculate the game situation and simulate player operations. In order to speed up the calculation speed of game AI for player operations to further enhance the user experience, the prior art usually uses sparsification to accelerate the convolution layer in the convolutional neural network. One way to use sparsification to optimize the calculation speed of the convolution layer is to convert the convolution calculation into matrix multiplication and speed up the calculation by using accelerated matrix multiplication. Another way is to adjust the convolutional neural network model structure and retrain: make the model weights sparse or use them with the RELU activation function to iteratively train the model, increase the sparsity, and then use sparse matrix calculations to accelerate the convolution layer calculation speed. However, the above methods either cannot be accelerated when the sparsity is not enough, or reduce the accuracy of the model or require the convolutional neural network model to be retrained. Summary of the invention

[0003] It would be advantageous to provide a solution that could alleviate, mitigate or eliminate the above-mentioned problems.

[0004] According to one aspect of the present invention, there is provided a method for performing calculations using a convolutional neural network, comprising: receiving feature data with different attributes input from at least two channels; determining the sparsity of each of the at least two channels based on the attributes of the received feature data; rearranging the at least two channels according to the determined sparsity so that the sparse channels in the at least two channels are arranged continuously, and the sparsity of the sparse channels is greater than or equal to a preset value; adjusting the parameters of a trained convolutional neural network model based on the rearranged channels, and performing convolution calculations on the feature data of the sparse channels and non-sparse channels separately using the adjusted convolutional neural network model, and the sparsity of the non-sparse channels is less than a preset value.

[0005] In some embodiments, the parameters of the trained convolutional neural network model include at least a structure file for defining the structure of the convolutional neural network and weight parameters for defining parameters when using the convolutional neural network for calculation.

[0006] In some embodiments, the weight parameters include bias parameters for each channel.

[0007] In some embodiments, adjusting the trained convolutional neural network model includes adjusting the structure of the trained convolutional neural network model to describe which of the channels are sparse channels.

[0008] In some embodiments, adjusting the trained convolutional neural network model includes adjusting corresponding weight parameters in the trained convolutional neural network model in the order of the channel rearrangement.

[0009] In some embodiments, the feature data of sparse channels and non-sparse channels are separately convolved using an adjusted convolutional neural network model, and the sparsity of the non-sparse channels is less than a preset value, which also includes: performing sparse channel fast convolution calculation on one or more rearranged adjacent sparse channels based on the parameters in the adjusted convolutional neural network model; performing conventional convolution calculation on one or more rearranged adjacent non-sparse channels based on the parameters in the adjusted convolutional neural network model; and merging the results of the sparse channel fast convolution calculation of the sparse channels and the results of the conventional convolution calculation of the non-sparse channels to obtain a calculation result; wherein, the sparse channel fast convolution calculation refers to calculating only the non-zero items in the sparse channels.

[0010] In some embodiments, merging the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel to obtain the calculation result also includes: merging the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel based on the bias parameter in the weight parameter.

[0011] In some embodiments, the method further includes converting the calculation result into an operation instruction to be executed.

[0012] In some embodiments, the trained convolutional neural network is pre-trained using a set of feature data sets with different attributes and data representations of operation instructions to be performed.

[0013] In some embodiments, the method further includes representing the feature data of each sparse channel using a sparse representation of a key-value pair, wherein the key represents the position of a non-zero item in the sparse matrix, and the value represents the value of the non-zero item.

[0014] In some embodiments, the steps of the method are performed in a convolutional layer of a convolutional neural network.

[0015] According to another aspect of the present invention, there is provided a device for performing calculations using a convolutional neural network, comprising: an input module for receiving feature data with different attributes input from at least two channels; a sparse data representation module for determining the sparsity of each of the at least two channels based on the attributes of the received feature data; and rearranging the at least two channels according to the determined sparsity so that the sparse channels in the at least two channels are arranged continuously and the sparsity of the sparse channels is greater than or equal to a preset value; and a convolutional neural network model adjustment module for adjusting parameters of a trained convolutional neural network model based on the rearranged channels, and a convolutional neural network calculation module for performing convolution calculations on the feature data of the sparse channels and non-sparse channels separately using the adjusted convolutional neural network model, and the sparsity of the non-sparse channels is less than a preset value.

[0016] In some embodiments, the parameters of the trained convolutional neural network model include at least a structure file for defining the structure of the convolutional neural network and weight parameters for defining parameters when using the convolutional neural network for calculation.

[0017] In some embodiments, the weight parameters include bias parameters for each channel.

[0018] In some embodiments, adjusting the trained convolutional neural network model includes adjusting the structure of the trained convolutional neural network model to describe which of the channels are sparse channels.

[0019] In some embodiments, adjusting the trained convolutional neural network model includes adjusting corresponding weight parameters in the trained convolutional neural network model in the order of the channel rearrangement.

[0020] In some embodiments, the convolutional neural network computing module uses the adjusted convolutional neural network model to perform convolution calculations on the feature data of sparse channels and non-sparse channels separately, and the sparsity of the non-sparse channels is less than a preset value, which also includes: performing sparse channel fast convolution calculations on one or more rearranged adjacent sparse channels based on the parameters in the adjusted convolutional neural network model; performing conventional convolution calculations on one or more rearranged adjacent non-sparse channels based on the parameters in the adjusted convolutional neural network model; and merging the results of the sparse channel fast convolution calculations of the sparse channels and the results of the conventional convolution calculations of the non-sparse channels to obtain a calculation result; wherein the sparse channel fast convolution calculation refers to calculating only the non-zero items in the sparse channels.

[0021] In some embodiments, merging the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel to obtain the calculation result also includes: merging the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel based on the bias parameter in the weight parameter.

[0022] In some embodiments, the device is further configured to: convert the calculation results into operation instructions to be executed.

[0023] In some embodiments, the device is further configured such that: the trained convolutional neural network is pre-trained by a set of feature data sets with different attributes and data representations of operation instructions to be executed.

[0024] In some embodiments, the device is further configured to: represent the feature data of each sparse channel using a sparse representation of a key-value pair, where the key represents the position of a non-zero item in the sparse matrix, and the value represents the value of the non-zero item.

[0025] In some embodiments, the device is further configured such that: each step is performed in a convolutional layer of a convolutional neural network.

[0026] According to yet another aspect of the present invention, a non-transitory computer-readable medium of computer program instructions is provided, and when a processor executes the computer program instructions, the processor executes the method as described above.

[0027] According to yet another aspect of the present invention, there is provided a computing device comprising a processor and a memory on which a computer program is stored, the computer program being configured to, when executed on the processor, cause the processor to perform the method as described above.

[0028] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A schematic diagram showing a scenario of calculating a game AI operation according to an embodiment of the present invention;

[0031] Figure 2 A flowchart of calculating game AI operations according to an embodiment of the present invention is shown;

[0032] Figure 3 A schematic diagram of the structure of a convolutional neural network computing module according to an embodiment of the present invention is shown;

[0033] Figure 4 A schematic diagram showing a data flow for performing calculations using a convolutional neural network according to an embodiment of the present invention;

[0034] Figure 5A flowchart of a convolutional neural network calculation method according to an embodiment of the present invention is schematically shown;

[0035] Figure 6 A flowchart of a convolutional neural network calculation method according to an embodiment of the present invention is further shown; and

[0036] Figure 7 An example system is generally illustrated that includes an example computing device that represents one or more systems and / or devices that may implement the various techniques described herein. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the scheme in the embodiment of the present invention will be described in detail and clearly in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0038] Convolutional neural networks are widely used for calculation and prediction in the fields of game AI, image recognition, object detection, etc. In the application scenarios of game AI, convolutional neural network models are usually used to judge the game situation and simulate player operations. The usual method for judging the game situation and simulating player operations is as follows: first, the convolutional neural network model structure and the feature input of multiple channels are designed according to the game rules, and then the convolutional neural network model structure is trained by supervised learning or reinforcement learning methods to obtain the corresponding weights of the convolutional neural network model, and then the convolutional neural network model is arranged in the actual operation scene of the game for calculation. During the game, the features extracted from the actual operation scene are input into the convolutional neural network model, and a series of calculations are performed according to the specific rules such as the convolution layer, pooling layer, fully connected layer and activation function layer defined by the convolutional neural network. The results of the calculation are converted into corresponding operations in the game, so as to simulate the player's operation through the game AI.

[0039] Figure 1 A schematic diagram of a scene 100 to which a game AI according to an embodiment of the present invention can be applied is shown. Figure 1A game battle screen is shown, in which a real player 101 and an AI player 102 are playing a game battle. In addition to the player 101 and the AI ​​player 102, the game battle screen also includes other objects, such as an attack range 103, a tower 104, a stone 105, etc. In the game process, the game situation can be judged and the operation of the AI ​​player 102 can be simulated by extracting various feature data in the game battle scene. The attributes of the feature data may include but are not limited to: game path, the game character itself, other game characters fighting with the game character, teammates of the game character, accessories of the game character (such as pets, props, weapons, etc.), accessories of other game characters fighting with the game character, accessories of teammates of the game character, props in the game (such as reward props, punishment props, obstacles), etc. The sparsity of feature data with the above-mentioned feature data attributes is known, so according to the attributes of the extracted feature data, it can be determined whether the corresponding feature channel is sparse.

[0040] The embodiment of the present invention provides a scheme for using a convolutional neural network to perform calculations using sparse data representation. The input channel includes a sparse channel and a non-sparse channel. Sparse channels and non-sparse channels are relative concepts, and sparsity is used to define the range of the two channels. Sparsity refers to the ratio of the number of zero values ​​in each channel to the number of all values ​​in the channel. Channels with a sparsity greater than or equal to a preset value are defined as sparse channels, while channels with a sparsity less than a preset value are defined as non-sparse channels. The preset value can be, for example, 90%, or it can be adjusted according to the actual scenario and operating platform used. According to one embodiment of the present invention, by rearranging the feature input channels, the sparse channels are arranged adjacently, thereby accelerating the calculation speed of the convolutional neural network, thereby improving the calculation speed. According to an embodiment of the present invention, the feature input channels of the model structure of the first layer (e.g., convolution layer) of the convolutional neural network are divided into sparse channels and non-sparse (dense) channels. The data of the sparse channel can be represented by a sparse format, i.e., a key-value pair (i.e., only non-zero values ​​are saved), where the key represents the position of the non-zero item in the sparse matrix, and the value represents the value of the non-zero item. In addition, a conventional non-sparse format can be used (i.e., an array or matrix is ​​used to save all values ​​of the data channel). The data of the non-sparse channel is represented by the above non-sparse format. Then the sparse channels and non-sparse channels are rearranged so that the sparse channels are arranged adjacently and the non-sparse channels are arranged adjacently. Next, based on the above arrangement, a sparse convolution fast calculation method is used to calculate one or more adjacent sparse channels; a general convolution calculation method is used to calculate one or more adjacent non-sparse channels. Finally, the convolution calculation results of the above sparse channels and non-sparse channels are accumulated to obtain the calculation result of the convolution layer.

[0041] Figure 2 A flowchart 200 for calculating game AI operations according to an embodiment of the present invention is shown. Figure 2 The solid-line box in represents the product processing module, and the dotted-line box 206 represents the calculation-related part.

[0042] In the game battle scenario, real players and AI players typically obtain game status and information from the game battle screen. The AI ​​player predetermines feature information from the game battle screen / map through feature engineering and tuning verification, and then reads the screen / map information for a certain frame or multiple frames in the game battle through the feature extraction module 201, and converts it into feature data. The convolutional neural network calculation module 202 uses the trained AI convolutional neural network model 205 to calculate the extracted feature data to obtain the AI ​​model calculation result. The calculation result is usually one or more floating-point arrays, each of which represents a game operation instruction. The game AI conversion module 203 converts the AI ​​model calculation result into a game operation instruction. Finally, the game operation instruction is sent to the game control module 204 to control the game object to operate and achieve the target effect of the game AI. The AI ​​convolutional neural network model is obtained by pre-training using a set of feature data sets with different attributes and data representations of the operation instructions to be executed. The dotted box 206 is a calculation-related part, which specifically includes a feature extraction module 201, an AI convolutional neural network module 205, and a convolutional neural network calculation module 202.

[0043] It can be understood that the solution according to the embodiment of the present invention is not limited to application in game AI, but is applicable to any scenario that uses convolutional neural networks for calculations.

[0044] Figure 3 The schematic diagram of the structure of the device 300 for calculating using a convolutional neural network according to an embodiment of the present invention is shown. The device 300 for calculating using a convolutional neural network is particularly suitable for the case where there are sparse channels in the feature input, and the device 300 for calculating using a convolutional neural network mainly includes an input module 301, a sparse data representation module 302, a convolutional neural network model adjustment module 303 and a convolutional neural network calculation module 304.

[0045] The input module 301 receives channel-wise input feature data representing different attributes in the game, as well as a trained convolutional neural network model from the AI ​​convolutional neural network module.

[0046] The sparse data representation module 302 determines whether each channel belongs to a sparse channel based on different attributes of the feature data input by each channel, and rearranges the channels according to the criterion of arranging the sparse channels continuously. In some embodiments, it is determined whether the sparse channels determined in all feature input channels are continuous, that is, adjacent. If the determined sparse channels are not continuous, the input channels are rearranged so that the sparse channels are arranged continuously and adjacently and the non-sparse channels are arranged adjacently. Figure 3 In the example, a sparse channel is identified by a non-shaded box 3021, and a non-sparse channel is identified by a shaded box 3022. When it is determined that the sparse channels 3021 are not arranged continuously adjacent to each other, the input channels are rearranged.

[0047] The convolutional neural network model adjustment module 303 adjusts the parameters of the trained convolutional neural network model based on the rearranged channels. In some embodiments, the parameters include a structure file 3031 and a weight parameter 3032. The structure file generally defines the type, dimension, and network topology connection between the specific layers used by the convolutional neural network. The weight parameter file generally stores the relevant parameter information for calculating the feature input, such as the weight parameter of each convolutional layer. In some cases, the weight parameter file also includes a bias parameter (bias). In the structure file of the convolutional neural network, the description of the first layer (e.g., convolutional layer) defines the number of channels and dimensions of the input features. The convolutional neural network model adjustment module 303 adds an additional identifier to the structure file based on the result of whether each channel determined in the sparse data representation module 302 belongs to a sparse channel, which indicates which channels in the feature input channel are sparse channels. The convolutional neural network model adjustment module 303 further adjusts the corresponding parameters in the weight parameter file based on which channels in the feature input channel are sparse channels and the corresponding relationship between each channel and the weight parameter.

[0048] The convolutional neural network calculation module 304 uses the adjusted convolutional neural network model to perform convolution calculations on the feature data of the sparse channels and non-sparse channels separately. In some embodiments, the convolutional neural network calculation module 304 includes a sparse channel fast calculation module 3041, a non-sparse channel calculation module 3042 and a merging module 3043. The sparse channel fast calculation module 3041 performs sparse channel fast calculations on the rearranged adjacent one or more sparse channels based on the parameters in the trained convolutional neural network model. Sparse channel fast calculation refers to the calculation using the sparse characteristics of the feature data input in the sparse channel. That is: because the result of multiplying a zero value by any number is zero and can be ignored during accumulation, when calculating the convolution of the sparse channel, only the non-zero values ​​of the corresponding weight parameter files of the non-zero values ​​in the feature input can be calculated, and the non-zero values ​​in the feature input are accumulated to the corresponding positions of the results according to the definition of the convolution calculation. By using the sparse channel fast calculation, the amount of calculation is greatly reduced, and the purpose of fast calculation is achieved. The non-sparse channel calculation module 3042 performs conventional convolution calculations on the non-sparse channels. Thereafter, the merging module 3043 adds the calculation results of the sparse channel fast calculation module 3041 and the non-sparse channel calculation module 3042. The summed result is used for the calculation of the convolutional neural network.

[0049] By dividing or rearranging the sparse channels together, the solution according to the embodiment of the present invention can significantly reduce the amount of calculation while fully ensuring the calculation accuracy of the convolutional neural network model. In this way, on the one hand, the calculation speed can be improved, the time consumption of the game AI model calculation can be reduced, and the operation can be responded to more quickly; on the other hand, the resources and energy consumption of the central processing unit CPU can be reduced. At the same time, the use of sparse format data representation can also reduce the size of feature input and reduce the occupation of some memory resources.

[0050] Figure 4 A schematic diagram of a data flow 400 for performing calculations using a convolutional neural network according to an embodiment of the present invention is shown.

[0051] Sparse data representation: The corresponding sparse data representation and processing flow diagram is as follows Figure 4 As shown. The data of the channel feature input can be regarded as multiple square matrices (usually square for game scenes, such as 10×10), which are divided into sparse channels and non-sparse channels according to the sparsity of the data in the matrix. The convolutional neural network model label is divided into two parts: structure file and weight parameter. The structure file and weight parameters are used as inputs of the calculation module to obtain the results of sparse channels and non-sparse channels respectively. Finally, the results can be optionally merged with some weight parameters (bias in the weight parameters, each channel has the same bias value) to obtain the complete convolution calculation result. Here, merging refers to adding the results of sparse channels and dense channels.

[0052] The convolutional neural network model label is divided into two parts: the structure file and the weight parameter. The structure file defines the type, dimension and network topology of the specific layers used by the convolutional neural network, and the weight parameter file stores the parameter information related to the calculation of the feature input (such as the weight of each convolution layer and the bias of each channel). In the structure file of the convolutional neural network model, the description of the first convolution layer defines the number and dimension of the input feature channels. Only additional labels are needed to indicate which channels of the input feature are sparse channels to achieve channel-wise feature input.

[0053] Figure 5 A flowchart 500 of a convolutional neural network calculation method according to an embodiment of the present invention is schematically shown. In step 501, feature data with different attributes input by channels and a trained convolutional neural network model for a convolutional neural network network are first received. Exemplarily, the attributes may include the game path extracted from the game map screen as described above, the game character itself, other game characters fighting with the game character, teammates of the game character, accessories of the game character (such as pets, props, weapons, etc.), accessories of other game characters fighting with the game character, accessories of teammates of the game character, props in the game (such as reward props, penalty props, obstacles), etc. In one embodiment, the parameters of the trained convolutional neural network model include at least a structure file for defining the structure of the convolutional neural network and a weight parameter for defining the parameters when the convolutional neural network is used for calculation. In one embodiment, the weight parameter also includes a bias parameter for each channel. The trained convolutional neural network is pre-trained by a feature data set with different attributes and a set of data representations of the operation instructions to be executed by the player.

[0054] In step 502, it is determined whether each channel belongs to a sparse channel based on the attributes of the feature data input by the channels. As described above, different feature data attributes have different sparsities. Therefore, the sparsity of the channel and whether the channel belongs to a sparse channel can be determined based on the attributes of the feature data of different channel data.

[0055] In step 503, the channels are rearranged according to the criterion for arranging the sparse channels continuously. If the characteristic channels are not arranged according to the criterion for arranging the sparse channels continuously, rearrangement is required. In some embodiments, the sparse channels are grouped together to be arranged continuously, and then the remaining non-sparse channels are grouped together to be arranged continuously. In one embodiment, if the sparse channels in each characteristic channel are already arranged continuously, rearrangement is not required.

[0056] In step 504, the trained convolutional neural network model parameters are adjusted based on the rearranged channels. In some embodiments, the adjustment includes adjusting the structure of the trained convolutional neural network model and adjusting the corresponding weight parameters in the convolutional neural network model. The definition in the structure description file of the convolutional neural network model is modified according to the order of channel rearrangement, and the order of the weight parameters in the convolutional neural network model is adjusted accordingly, so that according to the principle of convolution calculation, the result of the convolution calculation can be guaranteed to be consistent with the calculation result before the rearrangement. In some embodiments, adjusting the trained convolutional neural network model includes: adjusting the model structure, adding identifiers for describing which channels in the feature input channel are sparse channels; adjusting the model weights, adjusting the corresponding weights in the model structure according to the order of rearrangement, so that the weight parameters still correspond to the feature channels before the rearrangement.

[0057] In step 505, the adjusted convolutional neural network model is used in the calculation to perform convolution calculations on the feature data of the sparse channels and the non-sparse channels separately.

[0058] Figure 6 A flow chart of a convolutional neural network calculation method according to an embodiment of the present invention is further shown. Figure 6 The specific implementation of convolution calculation in one embodiment is further given. In step 5051, sparse channel fast convolution calculation is performed on the rearranged adjacent one or more sparse channels based on the parameters in the adjusted convolutional neural network model. Fast convolution calculation refers to calculating only the non-zero items of feature data in the sparse channel. Since the feature data of the sparse channel input is sparse, and the structure of zero value multiplied by any number is zero, it can be ignored when performing accumulation operations. Therefore, only the non-zero values ​​in the feature input can be calculated, and then accumulated to the corresponding position of the result according to the definition of convolution calculation. In this way, the amount of calculation can be greatly reduced to achieve the purpose of fast calculation.

[0059] In one embodiment, the feature data of each sparse channel can be represented by a sparse representation form key-value pair. Whether a sparse format that only stores non-zero values ​​is used, or a non-sparse format (i.e., an array or matrix is ​​used to store all values), each eigenvalue will only be calculated accordingly with the corresponding weight, so the convolution calculation result will not change. The data representation of sparse channels is not limited to sparse data representation. Using non-sparse data representation, a sparse matrix fast convolution algorithm can also be used, which can also achieve the purpose of accelerating calculation.

[0060] In one embodiment, the sparse channel fast convolution calculation may not be limited to being performed in the first layer of the convolutional neural network. The sparse channel fast convolution calculation may be performed in other layers of the convolutional neural network according to the position of the convolutional layer. For example, if the first layer of the convolutional neural network is a fully connected layer, the second layer is a pooling layer, and the third layer is a convolutional layer, then the above-mentioned sparse channel fast convolution calculation is calculated in the third layer of the convolutional neural network. It can be understood that when deploying a convolutional neural network model, it can be determined whether the subsequent network layers use sparse calculations for acceleration based on the sparsity of each channel in the application scenario.

[0061] In step 5052, conventional convolution calculation is performed on the rearranged adjacent one or more sparse channels based on the parameters in the adjusted convolutional neural network model.

[0062] In step 5053, the convolution calculation results of the above two are added to obtain the calculation result. According to the principle of convolution calculation, the result of convolution calculation is the calculation result of all output channels, and each output channel is independent of each other. Adjusting the order of sparse channels and non-sparse channels in the feature channel and the order of corresponding model weights does not affect the result of each input channel. Therefore, the final convolution result is obtained by summing the results of sparse channels and non-sparse channels.

[0063] In one embodiment, the weight parameters of the convolutional neural network model also include a bias parameter. In one embodiment, merging the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel to obtain the calculation result also includes: weighting the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel based on the bias parameter in the weight parameter. For the case where there is a bias value in the weight parameter, the usual calculation method is to add the bias value to the result of each output channel. In this embodiment, when merging the results, it is only necessary to add the bias to any one of the convolution results of the sparse channel, the convolution results of the non-sparse channel, or the final convolution result.

[0064] In one embodiment, the method further comprises converting the calculation result into an operation instruction to be executed.

[0065] In one embodiment, the method further includes padding processing of the convolution layer. Usually, in order to ensure the consistency of dimensions or to process boundary information, the convolution layer will perform zero padding operations in rows and columns in the four directions of up, down, left, and right of each channel. This operation is called padding. For the padding of sparse channels, the 0 value itself will not be calculated accordingly and can be ignored. It is sufficient to adjust the coordinates corresponding to the original non-zero value.

[0066] In one embodiment, the method further includes stride processing of the convolutional layer. The stride parameter of the convolutional layer will affect whether the non-zero value of the feature input needs to be calculated with the model weight parameter, which can be judged by adding a decision condition before the calculation.

[0067] Figure 7 An example system 700 is generally illustrated, which includes an example computing device 710 that represents one or more systems and / or devices that can implement the various techniques described herein. Computing device 710 can be, for example, a device associated with a client (e.g., a client device), a system on a chip, a server of a service provider, and / or any other suitable computing device or computing system. Figure 3 The depicted device 300 for computing using a convolutional neural network may take the form of a computing device 710. Alternatively, the device 300 for computing using a convolutional neural network may be implemented as a computer program in the form of a video app 716. More specifically, the device 300 for computing using a convolutional neural network may be implemented as an integral part of a video player, or as a plug-in that may be downloaded and installed separately from a video player.

[0068] The example computing device 710 as shown includes a processing system 711, one or more computer-readable media 712, and one or more I / O interfaces 713 that are communicatively coupled to each other. Although not shown, the computing device 710 may also include a system bus or other data and command transmission system that couples various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures. Various other examples are also contemplated, such as control and data lines.

[0069] Processing system 711 represents the functionality of performing one or more operations using hardware. Thus, processing system 711 is illustrated as including hardware elements 714 that may be configured as processors, functional blocks, and the like. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 714 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0070] Computer readable media 712 is illustrated as including memory / storage 715. Memory / storage 715 represents memory / storage capacity associated with one or more computer readable media. Memory / storage 715 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 715 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Computer readable media 712 may be configured in various other ways as further described below.

[0071] One or more I / O interfaces 713 represent functionality that allows a user to input commands and information to the computing device 710 and optionally also allows information to be presented to the user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can detect motion that does not involve touch as gestures using visible or invisible wavelengths (such as infrared frequencies), etc.). Examples of output devices include a display device (e.g., a projector), a speaker, a printer, a network card, a tactile response device, etc. Thus, the computing device 710 can be configured in various ways as further described below to support user interaction.

[0072] The computing device 710 may also include a video app 716. The video app 716 may be, for example, Figure 3 A software instance of a device 300 for performing computations using a convolutional neural network, and in combination with other elements in a computing device 710, implements the techniques described herein.

[0073] Various techniques may be described herein in the general context of software hardware elements or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module" (e.g., input module 301, sparse data representation module 302, convolutional neural network model adjustment module 303, and convolutional neural network calculation module 304 in the previous paragraph), "function" and "component" used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on various computing platforms with various processors.

[0074] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 710. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0075] In contrast to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0076] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of the computing device 710, such as via a network. Signal media typically can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that encodes the information in the signal in such a manner to set or change one or more of its characteristics. By way of example and not limitation, communication media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared and other wireless media.

[0077] As previously described, hardware elements 714 and computer-readable media 712 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0078] The aforementioned combination may also be used to implement the various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 714. The computing device 710 may be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 714 of a processing system, the module may be implemented as a module executable by the computing device 710 as software, at least in part, in hardware. Instructions and / or functions may be executable / operable by one or more articles (e.g., one or more computing devices 710 and / or processing systems 711) to implement the techniques, modules and examples described herein.

[0079] In various embodiments, computing device 710 can be implemented in various configurations. For example, computing device 710 can be implemented as a computer-type device including a personal computer, a desktop computer, a multi-screen computer, a laptop computer, a netbook, etc. Computing device 710 can also be implemented as a mobile device-type device including mobile devices such as mobile phones, portable music players, portable game devices, tablet computers, multi-screen computers, etc. Computing device 710 can also be implemented as a television-type device, which includes a device with or connected to a generally larger screen in a casual viewing environment. These devices include televisions, set-top boxes, game consoles, etc.

[0080] The techniques described herein can be supported by these various configurations of computing device 710 and are not limited to the specific examples of the techniques described herein. Computing device 710 can also interact through a variety of communication technologies "cloud" 720.

[0081] Cloud 720 includes and / or represents a platform 722 for resources 724. Platform 722 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 720. Resources 724 may include applications and / or data that can be used when performing computer processing on a server remote from computing device 710. Resources 724 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network. Platform 722 may abstract resources and functionality to connect computing device 710 to other computing devices. Platform 722 may also be used to abstract the hierarchy of resources to provide a hierarchy of corresponding levels of demand for resources 724 implemented via platform 722 that is encountered.

[0082] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0083] The present invention is not limited to game AI, but can also be used in any other scenarios that use convolutional neural networks for calculation and prediction, such as image recognition, object detection, etc.

[0084] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "one", "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "include" and / or "comprise" specify the existence of the features, integral bodies, steps, operations, elements and / or parts when used in this specification, but do not exclude the existence of one or more other features, integral bodies, steps, operations, elements, parts and / or their groups or add one or more other features, integral bodies, steps, operations, elements, parts and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0085] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure, and the appended claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for computing using a convolutional neural network, comprising: Receiving feature data with different attributes inputted from at least two channels; determining a sparsity of each of the at least two channels based on attributes of the received feature data; Rearranging the at least two channels according to the determined sparsity so that sparse channels in the at least two channels are arranged adjacent to each other, and the sparsity of the sparse channels is greater than or equal to a preset value; adjusting the parameters of the trained convolutional neural network model based on the rearranged channels, and Using the adjusted convolutional neural network model, separately performing convolution calculations on the feature data of the sparse channel and the non-sparse channel, the sparsity of the non-sparse channel being less than the preset value; The method further comprises: performing convolution calculation on the feature data of the sparse channel and the non-sparse channel separately using the adjusted convolution neural network model, wherein the sparsity of the non-sparse channel is less than the preset value; performing sparse channel fast convolution calculation on the rearranged adjacent one or more sparse channels based on the parameters in the adjusted convolution neural network model; performing conventional convolution calculation on the rearranged adjacent one or more non-sparse channels based on the parameters in the adjusted convolution neural network model; and The result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel are combined to obtain a calculation result; wherein the sparse channel fast convolution calculation refers to calculating only the non-zero items in the sparse channel.

2. The method according to claim 1, wherein the parameters of the trained convolutional neural network model include at least a structure file for defining the structure of the convolutional neural network and weight parameters for defining parameters when using the convolutional neural network for calculation. The method according to claim 2 , wherein the weight parameters include bias parameters for each channel.

4. According to the method of claim 1, the adjusting the trained convolutional neural network model includes adjusting the structure of the trained convolutional neural network model to describe which channels among the channels are sparse channels.

5. According to the method of claim 1, adjusting the trained convolutional neural network model includes adjusting the corresponding weight parameters in the trained convolutional neural network model according to the order of channel rearrangement.

6. The method according to claim 1, wherein merging the result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel to obtain the calculation result further comprises: The result of the sparse channel fast convolution calculation of the sparse channel and the result of the conventional convolution calculation of the non-sparse channel are merged based on the bias parameter in the weight parameter.

7. The method according to claim 6, further comprising: The calculation result is converted into an operation instruction to be executed.

8. The method according to claim 1, wherein the trained convolutional neural network is pre-trained by a set of feature data sets with different attributes and data representations of operation instructions to be performed.

9. The method according to claim 1, further comprising: The feature data of each sparse channel is represented by a sparse representation of a key-value pair, wherein the key represents the position of a non-zero item in the sparse matrix, and the value represents the value of the non-zero item.

10. The method according to any one of claims 1-9, wherein each step of the method is performed in a convolutional layer of the convolutional neural network.

11. A device for performing calculations using a convolutional neural network, comprising: An input module receives feature data with different attributes inputted from at least two channels; a sparse data representation module that determines the sparsity of each of the at least two channels based on the attributes of the received feature data; and rearranging the at least two channels according to the determined sparsity so that sparse channels in the at least two channels are arranged adjacent to each other, and the sparsity of the sparse channels is greater than or equal to a preset value; as well as A convolutional neural network model adjustment module adjusts the parameters of the trained convolutional neural network model based on the rearranged channels, and The convolutional neural network calculation module uses the adjusted convolutional neural network model to perform convolution calculations on the feature data of the sparse channel and the non-sparse channel separately, and the sparsity of the non-sparse channel is less than the preset value.

12. The device according to claim 11, wherein the parameters of the trained convolutional neural network model include at least a structure file for defining the structure of the convolutional neural network and weight parameters for defining parameters when using the convolutional neural network for calculation.

13. A non-transitory computer readable medium of computer program instructions, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 10.

14. A computing device comprising a processor and a memory having a computer program stored thereon, the computer program being configured to, when executed on the processor, cause the processor to perform the method of any one of claims 1-10.

Images