A neural network processing method, device, equipment and medium

Abstract

The present invention discloses a neural network processing method, device, equipment and medium, the method comprising obtaining a source neural network object, the source neural network object comprising a source framework type parameter, the source framework type parameter characterizing the creation environment of the source neural network object; obtaining an equivalent construction target parameter, the equivalent construction target parameter characterizing the creation environment of a target neural network object; parsing the source neural network object to obtain a layer topology; obtaining a layer object corresponding to each layer according to the layer topology and the source neural network object; obtaining an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter; obtaining a target layer object corresponding to each layer object according to the equivalent construction rule set; and connecting each target layer object according to the layer topology to obtain a target neural network object. The present invention realizes the conversion of neural network models under different frameworks by equivalently constructing equivalent neural network objects.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and in particular to a neural network processing method, device, equipment and medium. Background Art

[0002] In recent years, with the rapid development of deep learning technology and neural networks, many deep learning frameworks have been proposed or open sourced. The main goal of the deep learning framework is to implement the efficient operation of the deep learning system on the central processing unit (CPU) or graphics processing unit (GPU) device by providing a set of specialized neural network application programming interfaces (API). Due to the different target users, framework positioning and design concepts of different deep learning frameworks, there are differences in the specific implementation details. Researchers often need to have a specific understanding of the implementation details of the deep learning framework used during experiments, which increases the cost of research experiments. In order to reduce the difficulty of deep learning based on deep learning frameworks, high-level APIs based on deep learning models have emerged, which can be used to expand the specific implementation of neural networks in deep learning frameworks and encapsulate their related preprocessing, training and post-training processing logic. However, the existing high-level APIs based on deep learning models cannot switch between different framework neural network models, nor can they optimize the deployment process of neural networks under a certain neural network framework to mobile terminals. Summary of the invention

[0003] In order to solve the problem in the prior art that it is impossible to realize the conversion of neural networks under different frameworks and the problem that it is impossible to optimize the deployment process of a neural network under a certain neural network framework to a mobile terminal, the present invention provides a neural network processing method, device, equipment and medium.

[0004] In one aspect, the present invention provides a neural network processing method, the method comprising:

[0005] Acquire a source neural network object, wherein the source neural network object includes a source framework type parameter, and the source framework type parameter represents a creation environment of the source neural network object;

[0006] Acquire equivalent construction target parameters, wherein the equivalent construction target parameters represent a creation environment of a target neural network object;

[0007] Parsing the source neural network object to obtain a layer topology, where the layer topology is used to characterize the type of each layer and the connection relationship between each layer;

[0008] According to the layer topology and the source neural network object, obtain a layer object corresponding to each layer;

[0009] Acquire an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter;

[0010] Obtain the target layer objects corresponding to each layer object according to the equivalent construction rule set;

[0011] Connect each target layer object according to the layer topology to obtain the target neural network object.

[0012] In another aspect, the present invention provides a neural network processing device, the device comprising:

[0013] A source neural network object acquisition module, used to acquire a source neural network object, wherein the source neural network object includes a source framework type parameter, and the source framework type parameter represents a creation environment of the source neural network object;

[0014] An equivalent construction target parameter acquisition module, used to acquire equivalent construction target parameters, wherein the equivalent construction target parameters represent a creation environment of a target neural network object;

[0015] A layer topology parsing module, used to parse the source neural network object to obtain a layer topology, wherein the layer topology is used to characterize the type of each layer and the connection relationship between each layer;

[0016] A layer object construction module, used for obtaining a layer object corresponding to each layer according to the layer topology and the source neural network object;

[0017] An equivalent construction rule set acquisition module, used to acquire an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter;

[0018] A target layer object construction module is used to obtain the target layer objects corresponding to each layer object according to an equivalent construction rule set;

[0019] The target neural network object building module is used to connect each target layer object according to the layer topology to obtain the target neural network object.

[0020] On the other hand, the present invention provides a device, characterized in that the device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement a neural network processing method.

[0021] On the other hand, the present invention provides a computer storage medium, characterized in that the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes a neural network processing method.

[0022] The present invention provides a neural network processing method, device, equipment and medium. In the present invention, for each layer object in the source neural network object, the corresponding target layer object is obtained respectively, and each target layer object is associated according to the layer topology to obtain the target neural network object. Because each layer object is equivalent to its corresponding target layer object, the target neural network object obtained by the layer topology association is also equivalent to the source neural network object. Therefore, the present invention realizes the conversion of neural network models under different frameworks by equivalently constructing equivalent neural network objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flow chart of a neural network processing method provided by the present invention;

[0025] Figure 2 It is a schematic diagram of the convolutional layer processing method provided by the present invention;

[0026] Figure 3 It is a schematic diagram of the target layer object structure mapping corresponding to the layer object provided by the present invention;

[0027] Figure 4 It is a flow chart of a method for obtaining target layer objects corresponding to each layer object according to an equivalent construction rule set provided by the present invention;

[0028] Figure 5 It is a flow chart of a method for converting an input layer in Tensorflow into an input layer in Caffe provided by the present invention;

[0029] Figure 6 The present invention provides a flowchart for converting a neural network layer in Tensorflow into a neural network layer in Caffe;

[0030] Figure 7 It is a schematic diagram of feature filling in Tensorflow provided by the present invention;

[0031] Figure 8 It is a schematic diagram of feature filling in Caffe provided by the present invention;

[0032] Fig. 9 It is a schematic diagram of the application program encapsulation logic provided by the present invention;

[0033] Fig.10 It is a flow chart of a method for optimizing a target neural network object provided by the present invention;

[0034] Fig.11 It is a flow chart of the image preprocessing method provided by the present invention;

[0035] Fig.12 It is a block diagram of a neural network processing device provided by the present invention;

[0036] Fig.13 It is a schematic diagram of the hardware structure of a device provided by the present invention for implementing the method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0039] In order to make the purpose, technical solution and advantages disclosed in the embodiments of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not used to limit the embodiments of the present invention. In order to facilitate the understanding of the subsequent embodiments of the present invention, the basic concepts involved in the embodiments of the present invention are first explained as follows:

[0040] Tensorflow: Tensorflow is a deep learning framework. It is a symbolic mathematical system based on data flow programming and is widely used in the programming implementation of various machine learning algorithms. Its predecessor is Google's neural network algorithm library. Tensorflow has a multi-level structure and can be deployed on various servers, terminals and web pages. Tensorflow is developed and maintained by Google Brain, Google's artificial intelligence team.

[0041] A deep neural network is a multi-layer network model that includes: input layer, hidden layer and output layer. The hidden layer is also a multi-layer structure. Through TensorFlow, a multi-layer neural network can be constructed. The operating mechanism of each layer of the neural network is to obtain a result after the neurons in this layer are processed and then pass it to the next layer.

[0042] TensorLayer: TensorLayer is a deep learning and reinforcement learning library developed based on TensorFlow. It provides a high-level deep learning API to facilitate the development of neural networks based on TensorFlow. TensorLayer extends all neural network implementations in the Tensorflow framework through concise code implementation, and provides an end-to-end workflow including data preprocessing, training, post-training processing, as well as service modules and database management, which are all the processes required for developers to build a complete model. In this way, developers can not only call all the functions provided by Tensorflow through TensorLayer, but also use more extended functions provided by multiple underlying APIs.

[0043] TensorLayer is completely based on the TensorFlow deep learning library. Although it has made some practical extensions to TensorFlow, its algorithm implementation details are only inherited, and no external adjustment to support the algorithm implementation details is considered. Therefore, although it can solve the problem of simple calling and flexible expansion of TensorFlow, it cannot solve the problem of switching between multiple deep learning framework models encountered by engineers in their actual work. This determines that it can only be a high-level API for solving a single deep learning framework scenario, and it is not a high-level API for general scenarios.

[0044] In addition, TensorLayer is positioned only as an advanced deep learning model API, and it provides an end-to-end workflow including related post-training processing, service modules, and database management, but does not involve mobile deployment processes.

[0045] Caffe: Convolutional neural network framework, full name Convolutional Architecture for Fast Feature Embedding, is a commonly used deep learning framework, mainly used in video and image processing. It is a clear, highly readable and fast deep learning framework. Based on Caffe, you can build a multi-level deep neural network.

[0046] Caffe and Tensorflow are two independent deep learning frameworks. In the prior art, the differences in internal logic and structure between neural networks built on Caffe and those built on Tensorflow make them unable to be converted to each other.

[0047] In the embodiment of the present invention, the neural networks constructed under different frameworks are referred to as heterogeneous neural networks. For example, neural network A constructed based on the TensorFlow framework and neural network B constructed based on the Caffe framework are heterogeneous neural networks. In the prior art, heterogeneous neural networks cannot be converted to each other. However, in the actual research and development process, there is an objective demand for the mutual conversion of heterogeneous neural networks. The embodiment of the present invention provides a neural network processing method. The neural network processing method can obtain equivalent neural networks under heterogeneous frameworks by equivalent construction, thereby meeting the objective demand for the mutual conversion of neural networks. The method is as follows: Figure 1 As shown, including:

[0048] S101. Obtain a source neural network object, wherein the source neural network object includes a source framework type parameter, and the source framework type parameter represents a creation environment of the source neural network object.

[0049] Specifically, the source neural network object may be an untrained neural network model or a trained neural network model. The embodiment of the present invention does not limit the training process of the neural network model.

[0050] The source neural network object includes the structure of the neural network and related parameters of the neural network.

[0051] S102. Obtain equivalent construction target parameters, where the equivalent construction target parameters represent a creation environment of a target neural network object.

[0052] The source framework type parameter is used to characterize the creation environment of the source neural network object. For example, the creation environment of the source neural network object can be Tensorflow, Caffe, or other deep learning frameworks. Neural network objects created under different frameworks may not be freely converted.

[0053] The equivalent construction target parameter is used to characterize the creation environment of the target neural network object. For example, the creation environment of the target neural network object can be Tensorflow, Caffe or other deep learning frameworks. Neural network objects created under different frameworks may not be freely converted.

[0054] The creation environment of the source neural network object in the embodiment of the present invention is different from the creation environment corresponding to the equivalent construction target parameters.

[0055] S103. Parse the source neural network object to obtain a layer topology, where the layer topology is used to characterize the type of each layer and the connection relationship between each layer.

[0056] The connection relationship between the various layers of the neural network model mentioned in the embodiment of the present invention can be described by "front" and "back". In the neural network model, according to the data flow direction, the layer that obtains data first is in front, and the layer that obtains data later is in the back.

[0057] The model structure of the source neural network object can be flexibly set according to the actual situation. The type of each layer and the relationship between the layers are not specifically limited. A neural network object can contain at least one input layer and at least one neural network layer. Each layer has input and output. The output of this layer will be used as the input of the next adjacent layer.

[0058] Specifically, various types of neural network layers in the embodiments of the present invention are briefly described as follows:

[0059] Convolutional layer: The convolutional layer is an important part of the neural network model. It extracts features through convolution and reduces the dimension of its input data.

[0060] like Figure 2 As shown in the figure, the dimension of the input image is 32*32*3, 3 is its depth, representing the three channels of the image, that is, R represents red, G represents green, and B represents blue. The convolution kernel is a 5*5*3 tensor, and the depth of the convolution kernel is the same as the depth of the input image. By convolving a convolution kernel with the input image, a 28*28*1 feature map can be obtained. Figure 2 Two convolution kernels are used to obtain two feature maps.

[0061] Multiple convolutional layers can be used in a neural network model to obtain deeper feature maps.

[0062] Pooling layer: The function of the pooling layer is similar to that of the convolutional layer. It compresses the data input to the pooling layer, which reduces the data dimension and simplifies the complexity of network calculations. On the other hand, it compresses features and extracts the main features.

[0063] The pooling layer can be set after the convolution layer to retain the main features, while reducing the parameters and calculation amount of the next layer to prevent overfitting. At the same time, it can also maintain a certain invariance. The data processing actions it performs include translation, rotation, scaling, etc. Commonly used are mean-pooling and max-pooling.

[0064] Fully connected layer: connects all features to get the output of the neural network model.

[0065] Usually, when designing a neural network model, the first few layers are set as convolutional layers, which have a small proportion of parameters but a large proportion of calculations; and then the fully connected layers are set, which have a large proportion of parameters and a large proportion of calculations. When performing calculation acceleration optimization, the focus is on the convolutional layers; when performing parameter optimization and weight pruning, the focus is on the fully connected layers.

[0066] S104. Obtain a layer object corresponding to each layer according to the layer topology and the source neural network object.

[0067] Specifically, each layer object includes the type, structure, and parameters of the layer object.

[0068] The purpose of obtaining each layer object in this step is to subsequently transform each layer object separately to obtain a corresponding target layer object equivalent to the layer object in other frameworks. In the embodiment of the present invention, equivalence can be understood as that for the same input, two equivalent layer objects have the same output.

[0069] S105. Acquire an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter.

[0070] Specifically, in the embodiment of the present invention, the parameter pair consisting of the source framework type parameter and the equivalent construction target parameter has a one-to-one correspondence with the equivalent construction rule set.

[0071] For example, if the source framework type parameter points to the Tensorflow framework, and the equivalent construction target parameter points to Caffe, a first equivalent construction rule set is obtained, which specifies the logic of how to convert each layer object of the neural network under the Tensorflow framework into the layer object in Caffe.

[0072] For example, if the source framework type parameter points to the Caffe framework, and the equivalent construction target parameter points to Tensorflow, a second equivalent construction rule set is obtained, which specifies the logic of how to convert each layer object of the neural network under the Caffe framework into the layer object in Tensorflow.

[0073] S106. Obtain the target layer objects corresponding to each layer object according to the equivalent construction rule set.

[0074] Specifically, the target layer object may include one or more target sub-layers. For example, a layer object is a single-layer structure, and its corresponding target layer object may be a multi-layer structure, and the target layer object includes multiple sub-layers. Figure 3 shown. Figure 3 In the method, a first normalization layer is converted into a target layer object having a four-layer structure, wherein the four-layer structure is a minimum mean layer, a variance calculation layer, a second normalization layer, and a scaling layer.

[0075] Specifically, the target layer objects corresponding to each layer object are obtained according to the equivalent construction rule set, such as Figure 4 As shown, including:

[0076] S1061. Extract equivalent construction rules from the equivalent construction rule set according to the layer type in the layer object.

[0077] Specifically, the equivalent construction rule may be encapsulated as an application program interface, and the equivalent construction of the target layer object may be implemented through the application program interface.

[0078] S1062. Construct a target layer object equivalent to the layer object according to the equivalent construction rule.

[0079] The layer object has structure and parameters. The structure and parameters of the target layer object may be the same as or different from those of the layer object. However, the target layer object is equivalent to the layer object.

[0080] Taking the layer object as an input layer, the equivalent construction rule is used to convert the input layer in Tensorflow into the input layer in Caffe as an example. The target layer object equivalent to the layer object is constructed according to the equivalent construction rule as follows: Figure 5 As shown, including:

[0081] S1. Create the target input layer under the Caffe framework;

[0082] S2. Get a four-dimensional tensor in the layer object, where the four-dimensional tensor identifies a batch parameter, a length, a width, and a channel, respectively;

[0083] S3. Setting the properties of the target input layer according to the four-dimensional tensor to obtain a target layer object.

[0084] Taking the layer object as a neural network layer, the equivalent construction rule is used to convert the neural network layer in Tensorflow into the neural network layer in Caffe as an example, the target layer object equivalent to the layer object is constructed according to the equivalent construction rule as follows Figure 6As shown, including:

[0085] S10. Create the target neural network layer under the Caffe framework.

[0086] For different types of neural network layers, the mapping relationship from Tensorflow to Caffe is different, which may be a one-to-one mapping or a one-to-many mapping. The one-to-one mapping results in that the target neural network layer corresponding to a single-layer neural network layer only includes one layer of neural network, while the one-to-many mapping results in that the target neural network layer corresponding to a single-layer neural network layer includes multiple layers of neural networks.

[0087] S20. Obtain data processing logic executed in the neural network layer, wherein the data processing logic includes at least one data processing action.

[0088] S30. Obtain a first data processing logic and a second data processing logic according to the data processing logic, wherein the first data processing logic is a data processing logic composed of data processing actions incompatible with the Caffe framework in the data processing logic, and the second data processing logic is a data processing logic composed of data processing actions compatible with the Caffe framework.

[0089] Specifically, the first data processing logic or the second data processing logic may be empty.

[0090] S40. If the first data processing logic is not empty, a compatible processing logic is obtained according to the first data processing logic; the target neural network is corrected to obtain a target layer object, so that the data processing result output by the target layer object reflects the joint action of the compatible processing logic and the second data processing logic; the joint action of the compatible processing logic and the second data processing logic is equivalent to the action of the neural network layer in Tensorflow.

[0091] S50. If the first data processing logic is empty, the target neural network is modified to obtain a target layer object, so that the data processing result output by the target layer object reflects the effect of the second data processing logic.

[0092] In the embodiment of the present invention, the data processing logic includes a data processing action of feature padding as an example for explanation:

[0093] Since the Padding implementation methods of Caffe and Tensorflow are different, a brief introduction is given below:

[0094] Tensorflow provides two types of Padding operations: SAME Padding and VALID Padding.

[0095] The execution logic of VALID Padding is: the input feature map slides from left to right according to the preset step size. If the number of rows or columns of the feature map is odd, the rightmost column or bottommost row that cannot be processed is discarded.

[0096] SAME Padding: When the number of columns or rows of the feature map to be processed is odd, add a column to the right of the rightmost column or add a row to the bottom of the bottommost row. The elements in the newly added row or column are all zero, and the added feature map is used for processing.

[0097] Figure 7 The image shows a 2x3 input feature map matrix, with a maximum pooling window of 2x2 and a window movement stride of 2. The gray area can meet the processing requirements because the window can cover the area for the first time. After sliding two steps to the right, the remaining window in the VALID mode is less than 2x2, so the third column is directly discarded. The SAME mode does not discard the extra column, but fills the fourth column to meet the processing needs of the 2x2 window.

[0098] Caffe only provides symmetrical padding, which requires the columns or rows of left and right or top and bottom padding to be the same, so it does not support Figure 7 The SAME Padding method of Tensorflow in the text processing is not compatible with the data processing action of Padding. Therefore, it is necessary to design a corresponding compatible processing logic for it. The compatible processing logic in the embodiment of the present invention is to sequentially execute pre-filling and VALID Padding. That is, the SAME Padding (first data processing logic) in Tensorflow is converted into sequentially executing pre-filling and VALID Padding (compatible processing logic), such as Figure 8 As shown in the figure, VALID Padding is the same as VALID Padding in Tensorflow. The pre-filling + VALID Paddin method of compatible processing logic avoids calling the SAME method that Caffe does not support, solves the compatibility problem, and ensures that the neural network layer in Tensorflow can be converted into the target neural network layer in Caffe.

[0099] S107. Connect each target layer object according to the layer topology to obtain a target neural network object.

[0100] Specifically, for each layer object in the source neural network object, its corresponding target layer object is obtained respectively, and each target layer object is associated according to the layer topology to obtain the target neural network object. Because each layer object is equivalent to its corresponding target layer object, the target neural network object obtained by associating according to the layer topology is also equivalent to the source neural network object. Therefore, the embodiment of the present invention realizes the conversion of neural network models under different frameworks by equivalently constructing equivalent neural network objects.

[0101] Specifically, step S107 can also be implemented by setting an application program interface. In a preferred embodiment, Fig. 9 As shown, the equivalent construction rule set and step S107 can be encapsulated as an application program interface. The encapsulated application program interface is equivalent to an upper-level interface compatible with various neural network development frameworks, thereby realizing mutual conversion of neural network models created under different frameworks.

[0102] Furthermore, the application program interface may specifically include a layer initialization interface and a conversion interface. The layer initialization interface is used to construct a certain type of layer under each framework, and the conversion interface is used to perform different-framework conversion on the layer.

[0103] Taking the Tensorflow framework as an example, a __init__ method and a to_caffe_prototxt method are implemented for each layer to complete the call to the Tensorflow underlying interface and initialize the layer, as well as convert it into a layer in the Caffe framework.

[0104] The embodiment of the present invention provides a neural network processing method that is compatible with different deep learning frameworks, so that neural network models developed under different deep learning frameworks can be converted to each other in an equivalent structure. Based on the encapsulation of a layer of application program interface in the existing low-level deep learning framework (such as Tensorflow or Caffe), a neural network model definition API that is easier to modify and expand can be provided, and the conversion across deep learning frameworks is realized based on layer granularity.

[0105] Furthermore, in order to facilitate the user to perform one-click automatic conversion of models under different frameworks, a model automatic conversion interface can be provided in a preferred embodiment, and the model automatic conversion interface can encapsulate the logic of the above steps S101-S107 in a higher layer, thereby realizing one-click conversion. This conversion can take a neural network model with specific parameters and structure as input, and take another neural network model with specific parameters and structure under a different framework as output.

[0106] Specifically, the embodiment of the present invention provides a one-key conversion interface in a conversation mode and a one-key conversion interface in a non-conversation mode.

[0107] The one-click conversion interface in session mode completes the one-click conversion by loading the complete neural network object created externally.

[0108] The one-click conversion interface in non-session mode can create a complete neural network object internally by loading the definition and parameters of the model to be created, and complete the one-click conversion based on the created neural network object.

[0109] In an embodiment of the present invention, a method for converting neural networks to different frameworks is provided, and the applicable scenarios of neural networks under different frameworks may also be different. One of the typical scenarios is the implementation of a neural network model in a mobile terminal scenario. Specifically, the model generated by training different deep learning frameworks running on a workstation is converted into a neural network model in a mobile terminal. Since the hardware resources of the mobile terminal are limited, the neural network model in the mobile terminal not only needs to be recognized by the neural network framework running on the mobile terminal, but also should have the characteristics of small operating resource occupancy. The problem of identifying the neural network framework running on the mobile terminal can be solved by generating the target neural network object through the above-mentioned neural network processing method in the embodiment of the present invention, and the small operating resource occupancy can be solved by optimizing the structure of the target neural network object.

[0110] The embodiment of the present invention believes that the fewer the number of layers of the target neural network object, the smaller the target neural network object operation resource occupation. Therefore, after obtaining the target neural network object based on the above neural network processing method, the embodiment of the present invention also provides a method for optimizing the target neural network object, such as Fig.10 As shown, the method includes:

[0111] S201. Extract at least one layer structure to be optimized of the target neural network object, each of the layer structures to be optimized includes a data processing layer and a scaling layer.

[0112] The layer structure to be optimized is a multi-layer structure. Taking the target neural network object belonging to the Caffe framework as an example, the layer structure to be optimized includes at least one data processing layer and at least one scaling layer, and the data processing layer and the scaling layer can also perform multiple data processing actions. For example, the data processing layer (Batchnorm Layer) and its corresponding data processing actions may include minimum mean calculation (mini-batch mean) and variance calculation (mini-batch variance parameter calculation), and the scaling layer (Scale Layer) realizes normalization (normalize) and scaling (scale).

[0113] S202. Merge each layer in each layer structure to be optimized to obtain a target single layer corresponding to the layer structure to be optimized.

[0114] Specifically, the target single layer performs the following processing on the input data:

[0115] (1) Calculate the mean of the input data

[0116] (2) Calculate the variance of the input data

[0117] (3) Calculate the normalized eigenvalues ​​of the input data where ε is a built-in parameter in the target neural network object;

[0118] (4) Calculate the output data corresponding to the input data where γ, β are built-in parameters in the target neural network object.

[0119] S203. Use the target single layer to replace the layer structure to be optimized to obtain the optimization result of the target neural network.

[0120] The embodiment of the present invention not only provides a higher-level deep learning model API relative to the existing deep learning framework, so as to flexibly realize the model conversion of the neural network, thereby encapsulating a high-level API without usage scenario restrictions; it can also obtain a neural network model object suitable for mobile terminals, thereby opening up the complete process from workstation model training to mobile terminal application deployment, which can speed up the experimental speed of researchers and improve the engineering landing efficiency of research results. It can support one-click automatic conversion of different framework models, hiding the differences in the implementation details of different deep learning frameworks; based on the training results of the neural network model, a mobile terminal inference model generation method is encapsulated, which can make up for the engineering shortcomings of quickly launching mobile terminal applications for research and experimental results.

[0121] The neural network processing method based on the embodiment of the present invention can meet the needs of training and using neural networks under different frameworks, and the optimization method of the target neural network object can meet the needs of training and using neural networks in mobile terminals, which is convenient for the implementation of neural networks. The target neural network obtained in the embodiment of the present invention can be widely used in computer vision-related scenarios, such as image processing, face-pinching of game characters, and artistic filters. In order to apply the neural network trained in the background for computer vision-related scenarios to the mobile terminal, in a preferred embodiment, the mobile terminal and the background can pre-process the image based on the same image pre-processing method before training or using the neural network, so as to improve the training speed of the neural network and facilitate the application of the training results of the neural network. Specifically, the image pre-processing method is such as Fig.11As shown, including:

[0122] S301. Obtain the image to be processed.

[0123] Specifically, the channel order of the image to be processed is RGB.

[0124] S302. Crop a target image from the image to be processed, wherein the ratio of the area of ​​the target image to the area of ​​the image to be processed is within a first preset range, and the aspect ratio of the target image is within a second preset range.

[0125] Specifically, the ratio of the target image area to the image to be processed in the embodiment of the present invention can be randomly selected between [0.05, 1.0], and the aspect ratio of the target image in the embodiment of the present invention can be randomly selected between [3 / 4, 4 / 3].

[0126] S303. Adjust the size of the target image to a target size.

[0127] Specifically, the target size is preferably 224*224.

[0128] S304. Flip the target image horizontally with a first probability.

[0129] Specifically, the first probability is preferably 0.5.

[0130] S305. Rotate the target image with a second probability, wherein the rotation angle is within a third preset interval.

[0131] Specifically, the second probability is preferably 0.25, and the second preset interval is preferably [-45°, 45°].

[0132] S306. Perturb the saturation, contrast, brightness and chroma of the target image with a third probability.

[0133] The embodiment of the present invention not only proposes a processing method for a neural network, but also encapsulates the processing method as a high-level API for defining a neural network model. It provides compatibility with different deep learning frameworks and proposes a neural network optimization processing method, so that the neural network can be directly implemented on a mobile terminal. Relying on the embodiment of the present invention, developers can quickly iterate the neural network model, train the model, and deploy it on the mobile terminal, improve the efficiency of researchers in conducting deep learning algorithm experiments, and provide convenience for researchers to implement algorithms and deploy mobile terminal projects, solving engineering problems that have long plagued researchers.

[0134] The embodiment of the present invention also provides a neural network processing device, such as Fig.12 As shown, the device comprises:

[0135] A source neural network object acquisition module 401 is used to acquire a source neural network object, wherein the source neural network object includes a source framework type parameter, and the source framework type parameter represents a creation environment of the source neural network object;

[0136] An equivalent construction target parameter acquisition module 402 is used to acquire equivalent construction target parameters, wherein the equivalent construction target parameters represent a creation environment of a target neural network object;

[0137] A layer topology parsing module 403 is used to parse the source neural network object to obtain a layer topology, where the layer topology is used to characterize the type of each layer and the connection relationship between each layer;

[0138] A layer object construction module 404, used to obtain a layer object corresponding to each layer according to the layer topology and the source neural network object;

[0139] An equivalent construction rule set acquisition module 405 is used to acquire an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter;

[0140] A target layer object construction module 406 is used to obtain the target layer object corresponding to each layer object according to the equivalent construction rule set;

[0141] The target neural network object construction module 407 is used to connect each target layer object according to the layer topology to obtain the target neural network object.

[0142] Specifically, the neural network processing device and method embodiments described in the embodiments of the present invention are based on the same inventive concept.

[0143] An embodiment of the present invention further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of a neural network processing method described in an embodiment of the present invention. The specific execution process can be described in detail in the method embodiment, and will not be repeated here.

[0144] Further, Fig.13 A schematic diagram of the hardware structure of a device for implementing the method provided by an embodiment of the present invention is shown. The device may be a computer terminal, a mobile terminal or other device. The device may also participate in constituting or include the apparatus provided by an embodiment of the present invention. Fig.13As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.13 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations are shown.

[0145] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0146] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the above-mentioned neural network processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0148] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0149] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0150] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0151] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural network processing method, It is characterized in that The method comprises: Acquire a source neural network object, wherein the source neural network object includes a source framework type parameter, and the source framework type parameter represents a creation environment of the source neural network object; Acquire equivalent construction target parameters, wherein the equivalent construction target parameters represent a creation environment of a target neural network object; Parsing the source neural network object to obtain a layer topology, where the layer topology is used to characterize the type of each layer and the connection relationship between each layer; According to the layer topology and the source neural network object, obtain a layer object corresponding to each layer; Acquire an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter; Extracting equivalent construction rules from the equivalent construction rule set according to the layer type in the layer object; constructing a target layer object equivalent to the layer object according to the equivalent construction rules; Connect each target layer object according to the layer topology to obtain the target neural network object; Wherein, the input data of the source neural network object and the target neural network object are both images, and constructing a target layer object equivalent to the layer object according to the equivalent construction rule comprises: Create the target input layer under the convolutional neural network framework; Get the four-dimensional tensor in the layer object, wherein the four-dimensional tensor respectively identifies the batch processing parameter, length, width and channel; The properties of the target input layer are set according to the four-dimensional tensor to obtain the target layer object.

2. The method according to claim 1, It is characterized in that The step of constructing a target layer object equivalent to the layer object according to the equivalent construction rule includes: Creating a target neural network layer under the convolutional neural network framework; Obtaining data processing logic executed in the neural network layer, the data processing logic comprising at least one data processing action; According to the data processing logic, a first data processing logic and a second data processing logic are obtained, wherein the first data processing logic is a data processing logic composed of data processing actions incompatible with the convolutional neural network framework in the data processing logic, and the second data processing logic is a data processing logic composed of data processing actions compatible with the convolutional neural network framework; If the first data processing logic is not empty, the compatible processing logic is obtained according to the first data processing logic; the target neural network is corrected to obtain the target layer object, so that the data processing result output by the target layer object reflects the joint action of the compatible processing logic and the second data processing logic; the joint action of the compatible processing logic and the second data processing logic is equivalent to the action of the target layer object.

3. The method according to claim 2, It is characterized in that The step of constructing a target layer object equivalent to the layer object according to the equivalent construction rule further includes: If the first data processing logic is empty, the target neural network is modified to obtain the target layer object so that the data processing result output by the target layer object reflects the effect of the second data processing logic.

4. The method according to claim 1, It is characterized in that Also includes: The target neural network object is optimized to reduce the number of layers of the target neural network object.

5. The method according to claim 4, It is characterized in that The optimizing the target neural network object to reduce the number of layers of the target neural network object includes: Extracting at least one layer structure to be optimized of the target neural network object, each of the layer structures to be optimized includes a data processing layer and a scaling layer; Merging each layer in each of the layer structures to be optimized to obtain a target single layer corresponding to the layer structure to be optimized; The target single layer is used to replace the layer structure to be optimized to obtain the optimization result of the target neural network.

6. A neural network processing device, It is characterized in that The device comprises: A source neural network object acquisition module, used to acquire a source neural network object, wherein the source neural network object includes a source framework type parameter, and the source framework type parameter represents a creation environment of the source neural network object; An equivalent construction target parameter acquisition module, used to acquire equivalent construction target parameters, wherein the equivalent construction target parameters represent a creation environment of a target neural network object; A layer topology parsing module, used to parse the source neural network object to obtain a layer topology, wherein the layer topology is used to characterize the type of each layer and the connection relationship between each layer; A layer object construction module, used for obtaining a layer object corresponding to each layer according to the layer topology and the source neural network object; An equivalent construction rule set acquisition module, used to acquire an equivalent construction rule set according to the source framework type parameter and the equivalent construction target parameter; A target layer object construction module is used to extract an equivalent construction rule from the equivalent construction rule set according to the layer type in the layer object; and to construct a target layer object equivalent to the layer object according to the equivalent construction rule; A target neural network object building module, used for connecting each target layer object according to the layer topology to obtain the target neural network object; Wherein, the input data of the source neural network object and the target neural network object are both images, and constructing a target layer object equivalent to the layer object according to the equivalent construction rule comprises: Create the target input layer under the convolutional neural network framework; Get the four-dimensional tensor in the layer object, where the four-dimensional tensor identifies the batch processing parameter, length, width, and channel respectively; The properties of the target input layer are set according to the four-dimensional tensor to obtain a target layer object.

7. A device, It is characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a neural network processing method as described in any one of claims 1 to 5.

8. A computer storage medium, It is characterized in that The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor according to a neural network processing method as described in any one of claims 1-5.

Images