METHOD AND DEVICE FOR OPERATION OF A DEEP NEURAL NETWORK

Description

[0001] The present application claims the priority to a Chinese patent application No. 201811307229.4, filed with the China National Intellectual Property Administration on November 05, 2018 and entitled "Deep Neural Network Operation Method and Apparatus".Technical field

[0002] The present application relates to the field of machine learning technology, and in particular to a deep neural network operation method and apparatus.Background

[0003] As an emerging field in machine learning research, Deep Neural Network (DNN) analyzes data by imitating the mechanism of human brain, and is an intelligent model for analysis and learning by establishing and simulating the human brain. At present, DNNs such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Long Short Term Memory (LSTM) have been well applied in target detection and segmentation, behavior detection and recognition, speech recognition or the like. With the increasing amount of data, the calculation complexity of the DNN is increasing.

[0004] At present, in order to reduce the calculation complexity of DNN, lightweight neural network structures are generally used, such as mobile-end convolutional neural networks MobileNet and ShuffleNet or the like. These lightweight neural network structures implement convolution operation through Depthwise Convolution (DWConv). Specifically, each of channels of the input feature map is convoluted separately, and then the convolution result of each of channels is convoluted by using 1×1 convolution kernel to realize information aggregation on the channel dimension. DWConv splits complex convolution operation, convolves the feature map of a channel, and then uses 1×1 convolution kernel to realize information aggregation on the channel dimension, thus reducing the complexity of each convolution operation.

[0005] However, as mentioned above, multiple convolution operations are required in DWConv. This reduces DNN operation efficiency.

[0006] YUNHO JEON ET AL: "Constructing Fast Network through Deconstruction of Convolution", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 28 May 2018 (2018-05-28) discloses constructing fast network through deconstruction of convolution. Convolutional neural networks have achieved great success in various vision tasks; however, they incur heavy resource costs. By using deeper and wider networks, network accuracy can be improved rapidly. However, in an environment with limited resources (e.g., mobile applications), heavy networks may not be usable. This study shows that naive convolution can be deconstructed into a shift operation and pointwise convolution. To cope with various convolutions, we propose a new shift operation called active shift layer (ASL) that formulates the amount of shift as a learnable function with shift parameters. This new layer can be optimized end-to-end through backpropagation and it can provide optimal shift values. Finally, we apply this layer to a light and fast network that surpasses existing state-of-the-art networks.

[0007] WU BICHEN ET AL: "Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions", 2018 IEEE / CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, IEEE, 18 June 2018 (2018-06-18), pages 9127-9135 , DOI: 10.1109 / CVPR.2018.00951 discloses Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions. Neural networks rely on convolutions to aggregate spatial information. However, spatial convolutions are expensive in terms of model size and computation, both of which grow quadratically with respect to kernel size. In this paper, we present a parameter-free, FLOP-free "shift" operation as an alternative to spatial convolutions. We fuse shifts and point-wise convolutions to construct end-to-end trainable shift-based modules, with a hyperparameter characterizing the tradeoff between accuracy and efficiency. To demonstrate the operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters; we additionally demonstrate the operation's resilience to parameter reduction on ImageNet, outperforming ResNet family members. We finally show the shift operation's applicability across domains, achieving strong performance with fewer parameters on classification, face verification and style transfer.Summary

[0008] Embodiments of the present application provide a deep neural network operation method and apparatus. Operation efficiency of DNN can be improved. The following aspects are provided.

[0009] The invention is set forth in the appended claims.

[0010] In a first aspect, a computer-implemented method for operating a deep neural network according to the appended claim 1 is provided.

[0011] Optionally, a displacement amount is an average distribution of multiple preset different offsets.

[0012] In a second aspect, an apparatus for operating a deep neural network according to the appended claim 3 is provided.

[0013] Optionally, a displacement amount is an average distribution of multiple preset different offsets.

[0014] In a third aspect, a machine-readable storage medium according to the appended claim 4 is provided.

[0015] In a fourth aspect, a computer program according to the appended claim 6 is provided.

[0016] In embodiments of the present application, a solution is provided as follows. An input feature map of a network layer is obtained. Each of channels of the input feature map of the network layer is displaced respectively, along each of axes, according to a preset displacement parameter. In this way, a displacement feature map is obtained. Convolution operation is performed on the displacement feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer. Each of channels of the input feature map of the network layer is displaced along axes respectively. A misplacement of the input feature map of the network layer is used, to communicate information in the receptive field.

[0017] As such, the subsequent 1×1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information on a spatial domain dimension. The whole process requires only a single 1×1 convolution. This effectively improves the operation efficiency of the DNN.Brief Description of the Drawings

[0018] In order to more clearly describe the technical solution of the embodiments of the present application and the prior art, drawings needed in the embodiments and the prior art will be briefly described below. Fig. 1 is a schematic flowchart of a depth neural network operation method according to an embodiment of the present application; Fig. 2 is a schematic diagram of channel displacement according to an embodiment of the present application; Fig. 3a is a schematic diagram of a deep neural network structure according to one embodiment of the present application; Fig. 3b is a schematic diagram of a deep neural network structure according to another embodiment of the present application; Fig. 4 is a schematic structural diagram of a depth neural network operation apparatus according to an embodiment of the present application; Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Detailed Description

[0019] In order to make the objectives, technical solutions and advantages of the application more clearly, the present application will be further described in detail with reference to the accompanying drawings and examples.

[0020] The embodiments of the present application provide a deep neural network operation method, apparatus, electronic device and machine-readable storage medium, so as to improve the operation efficiency of the DNN.

[0021] The depth neural network operation method according to the embodiments of the present application will be first introduced below.

[0022] The deep neural network operation method according to embodiments of the present application is performed on an electronic device that can implements intelligent algorithms. The electronic device is an intelligent device capable of target detection and segmentation, behavior detection and recognition or speech recognition, such as a remote computer, a remote server, an intelligent camera, or an intelligent voice device. The device includes at least a processor equipped with a core processing chip. The deep neural network operation method according to embodiments of the present application may be embodied in software, hardware, and logic circuits disposed in the entity.

[0023] As shown in Fig. 1, the depth neural network operation method according to an embodiment of the present application may include the following steps.

[0024] S101, obtaining an input feature map of a network layer.

[0025] The network layer may be any operation layer in a depth neural network. In this embodiment, the network layer includes a displacement layer and a 1×1 convolution kernel, wherein one displacement layer corresponds to one displacement structure, that is, the displacement layer may be a displacement structure that can display an input feature map of the network layer. Of course, the displacement layer may also be a structure obtained by combining basic structures, such as a convolution layer and a pooling layer, with the displacement structure.

[0026] S102, displacing respectively, according to a preset displacement parameter, each of channels of the input feature map of the network layer along axes, to obtain a displaced feature map.

[0027] The preset displacement parameter includes displacement amounts of a channel in axes. If the input feature map of the network layer has C channels, the displacement layer that displaces the input feature map may be a structure with 2C parameters. A displacement parameter (DX (c), Dy (c)) includes displacement amounts of a C-th channel in x and y axes. For example, as shown in Fig. 2, a displacement parameter of (1, 1) for a single channel moves the feature map of the channel towards upleft by one unit. The part that exceeds the boundary is discarded, and the new vacant position is filled with zero. This displacement operation may be expressed as formula (1): O C , x , y = I C , x + dx C , y + dy C wherein, I C is the C-th channel of the input feature map of the network layer, Oc is the C-th channel of the displaced feature map, (x,y) is a coordinate point on the displacement feature map, (DX(c), Dy(c)) is the displacement parameter for the C-th channel, and (x+DX(c), y+Dy(c)) and (x,y) corresponds to coordinate points before the input feature map of the network layer is displaced.

[0028] Each channel has a respective preset displacement parameter, and is displaced according to the displacement parameter, to obtain a displaced feature map. The preset displacement parameter may be constant, for example, a calculated average attribution of multiple preset different offsets, and the preset displacement parameter may also be obtained through learning.

[0029] Optionally, a displacement amount is an average distribution of multiple preset different offsets.

[0030] The displacement amount in the preset displacement parameters may be obtained by calculating an average distribution of multiple preset different offsets. The multiple preset different offsets may be set empirically. A greater number of offsets can ensure higher accuracy of the preset displacement parameters obtained by calculating average distribution. Of course, in addition to calculation of the average distribution, the preset displacement parameter can be obtained through Gaussian distribution.

[0031] The preset displacement parameters are obtained through learning.

[0032] The preset displacement parameters are obtained by learning through the following steps.

[0033] In a first step, an initial displacement parameter is obtained.

[0034] The initial displacement parameter may be in various forms. For example, the initial displacement parameter can be (0, 0) Displacement amounts in the initial displacement parameter may also be the averages of multiple different offsets. The initial displacement parameter may also be emprically set by the user.

[0035] In a second step, displacement amounts in the displacement parameter are converted into floating-point displacement amounts.

[0036] The initial displacement parameter typically includes discrete integer variables. To allow the displacement parameter to be obtained by learning, in this embodiment, displacement amounts in the displacement parameter are converted to the floating-point displacement amounts.

[0037] In a third step, an output function of the network layer is obtained by means of bilinear interpolation algorithm according to the input feature map of the network layer and floating-point displacement amounts.

[0038] In order to obtain the displacement parameter by learning, in addition to converting the displacement amounts in the displacement parameter to floating-point displacement amounts, the displacement operation in this embodiment is also converted to differential bilinear interpolation sampling, so that the formula (1) may be transformed into: O C , x , y = ∑ n m ∈ Ω I C , n , m 1 − x + dx C − n 1 − y + dy C − m wherein, I C is the C-th channel of the input feature map of the network layer, Oc is the C-th channel of the displaced feature map, (x,y) is a coordinate point on the displacement feature map, Ω is the neighborhood where the floating-point coordinates (x+dx(C),y+dy(C)) in the input feature map of the network layer are located, (n,m) is the coordinates of four boundary points of the neighborhood, (dx(C),dy(C)) is the displacement parameter corresponding to the C-th channel. Formula (2) gives the principle of bilinear interpolation, that is, using four real coordinate points around the target point in the original image to jointly determine a coordinate point in a target image.

[0039] In a fourth step, a network loss value of the deep neural network is determined according to input data of the deep neural network, the 1×1 convolution kernel, the initial displacement parameter, and a label corresponding to the input data.

[0040] The whole depth neural network may be regarded as a non-linear mapping function, which is a stack of displacement layers, convolution layers, activation layers, pooling layers and / or some other customized network layers, and is used to learn the non-linear mapping between input data of the deep neural network and corresponding labels.

[0041] Network loss values are task-related. Different tasks have different loss functions. A classification task have a loss function of softmax, for example. The network loss value represents approximation between an output value or output image of the whole depth neural network and labels (such as category, segmentation result, detection result, etc.) of the input data. Image detection and segmentation may have repective loss functions, examples of which are not provided here.

[0042] The operation of determining the network loss value of the depth neural network according to input data of the depth neural network, the 1×1 convolution kernel, the displacement parameter and a label corresponding to the input data includes: calculating the network loss function of the deep neural network according to the input data of the deep neural network, the 1×1 convolution kernel, the initial displacement parameter, and the label for the input data; calculating a first regularization term according to the 1×1 convolution kernel; calculating a second regularization term according to the displacement parameter; and weighting the network loss function, the first regularization term, and the second regularization term to obtain the network loss value of the deep neural network.

[0043] In order to keep input feature maps in place to the maximum extent, i.e. (dx,dy)=(0,0), and further reduce the loss caused by displacement, the displacement parameter in this embodiment is subject to regularized constraint. The network loss value is shown in formula (3). Loss = L f input w 0 , … , w p , dx 0 , dy 0 , … , dx q , dy q , T + λ 1 ∑ i p R w i + λ 2 ∑ i q R dx i dy i wherein, Loss is the network loss value of the deep neural network, L(.) is the network loss function, f(.) is the nonlinear mapping function represented by the deep neural network, w is the 1x1 convolution layer weight parameter, input is the input data of the deep neural network, (dx,dy) is the displacement parameter, p is the number of convolutional layers, q is the number of displacement layers, T is the label corresponding to the input data of the deep neural network, and λ1, λ2 are weighting coefficients, ∑ i n R w i is the first regularation term, ∑ i m R dx i dy i is the second regularation term. The regularation term in this embodiment can be L1 regularation or L2 regularation.

[0044] In a fifth step, a gradient of the network loss value with respect to the displacement parameter is calculated according to the output function of the network layer, and the displacement parameter are updated in multiple iterations by a gradient descent method to obtain preset displacement parameter.

[0045] The network loss value may be the above network loss value with or without regularized constraint. In particular, the gradient of the network loss value with respect to the output function of the network layer may be calculated according to the chain rule and in a sequence from the last network layer forward. The gradient reflects the adjustment direction and size of the displacement parameter. As such, the preset displacement parameter can be obtained by updating displacement parameter in multiple iterations through a gradient descent method. Specifically, calculation of the gradient of the network loss value with respect to the output function is a process of inverse partial differentiation of the network loss value to the output function, as shown in formula (4): ∂ Loss ∂ dx c = ∑ x width ∑ y height ∂ Loss ∂ O c , x , y ∑ n m ∈ Ω I c , n , m 1 − y + dy c − m sign n − x − dx c + λ 2 ∂ R dx c dx c ∂ Loss ∂ dy c = ∑ x width ∑ y height ∂ Loss ∂ O c , x , y ∑ n m ∈ Ω I c , n , m 1 − x + dx c − n sign m − y − dy c + λ 2 ∂ R dy c dy c wherein, sign (.) is a sign function, which outputs 1 when the input data is positive and outputs -1 when the input data is negative; height and width are the height and width of the displacement feature map. Formula (4) calculates the gradient of the network loss value to the displacement parameter according to the gradient of the network loss value to the displacement feature map, to update the displacement parameters.

[0046] Since the displacement layer is expected to be used for displacement operation instead of bilinear interpolation operation during the test phase, and the displacement parameter need to be maintained as the integer parameter during the forward derivation, the forward displacement parameter is always the rounded value of the floating-point parameter. The integer parameter would not cover floating-point parameter, and is only used for forward derivation to obtain a loss value during training process. When calculating gradient by inverse derivation, the updated parameter is a floating-point displacement parameter. If the displacement change of the inverse partial derivative is too small, the forward derivation of the network is not to be affected. Only when the updated displacement change accumulated in multiple iterations causes a substantial change, the forward derivation of the network will be affected. When the training of the network is completed, only the rounded displacement parameters are retained.

[0047] S102 further comprises: rounding off the floating-point displacement amounts in the preset displacement parameter to obtain integer displacement amounts; and displacing respectively, according to the integer displacement amounts in the preset displacement parameter, each of channels of the input feature map along the axes, to obtain the displaced feature map.

[0048] When the forward derivation or training is completed, each displacement operation shall be based on the integer displacement amount. Therefore, it is necessary to round floating-point displacement amounts in the preset displacement parameter and perform displacement based on the integer displacement amount to obtain a displacement feature map.

[0049] S103, performing convolution operation on the displaced feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer.

[0050] After obtaining the displaced feature map, the convolution operation is performed on the displaced feature map with the 1×1 coanvolution kernel. Since the displaced feature map can communicate information in the receptive field, the 1 × 1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information aggregation.

[0051] Fig. 3a and Fig. 3b are network structures of the deep neural network in the embodiment. Fig. 3a is a network structure with a stride of 2, and Fig. 3b is the network structure with a stride of 1. Both network structures do not contain any convolution layer with convolution core greater than 1. The right branches of Fig. 3a and Fig. 3b relate to a channel attention mechanism, the focus of which on some useful information in the channel. This is not the focus of the embodiment and will not be described in detail here. In addition to the network structure shown in Figs. 3a and 3b, the network structure of the deep neural network may have other variants, further examples of which are not provided here.

[0052] In embodiments of the present application, a solution is provided as follows. An input feature map of a network layer is obtained. Each of channels of the input feature map of the network layer is displaced respectively, along each of axes, according to a preset displacement parameter. In this way, a displacement feature map is obtained. Convolution operation is performed on the displacement feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer. Each of channels of the input feature map of the network layer is displaced along axes respectively. A misplacement of the input feature map of the network layer is used, to communicate information in the receptive field. As such, the subsequent 1×1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information on a spatial domain dimension. The whole process requires only a single 1×1 convolution. This effectively improves the operation efficiency of the DNN. Through the combination of the displacement structure and 1 × 1 convolution core, a lightweight network structure that meets the accuracy of lightweight network can be consructed, while avoiding the use of DWConv. The network structure according to this embodiment may run efficiently on graphics processing unit (GPU) platform or other hardware platforms, and has an important industrial and engineering value. By applying the network structure according to the embodiment on a GPU platform or other hardware platforms, the execution efficiency of the hardware platform for object detection and segmentation, behavior detection and recognition, and speech recognition can be improved, such that performance of the hardware platform can be improved.

[0053] Corresponding to the above method embodiment, an embodiment of the present application provides a deep neural network operation apparatus. As shown in FIG. 4, the deep neural network operation apparatus may include: an obtaining module 410, configured for obtaining an input feature map of a network layer; a displacement module 420, configured for displacing respectively, according to a preset displacement parameter, each of channels of the input feature map of the network layer along axes, to obtain a displaced feature map, wherein the preset displacement parameter comprises displacement amounts of the channel in the axes; and an operation module 430, configured for performing convolution operation on the displaced feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer.

[0054] Optionally, a displacement amount is an average distribution of multiple preset different offsets.

[0055] The preset displacement parameter is obtained by learning in advance.

[0056] The apparatus further includes: a learning module configured for: obtaining an initial displacement parameter; converting the displacement amounts in the initial displacement parameter into floating-point displacement amounts; obtaining an output function of the network layer through a bilinear interpolation algorithm according to the input feature map of the network layer and the floating-point displacement amounts; determining a network loss value of the depth neural network according to input data of the depth neural network, the 1×1 convolution kernel, the displacement parameter and a label corresponding to the input data; and calculating, according to the output function of the network layer, a gradient of the network loss value with respect to the displacement parameter, and updating the displacement parameter in multiple iterations by a gradient descent method to obtain the preset displacement parameter.

[0057] The learning module is specifically configured for: calculating a network loss function of the deep neural network according to the input data of the deep neural network, the 1×1 convolution kernel, the initial displacement parameter, and the label corresponding to the input data; calculating a first regularization term according to the 1×1 convolution kernel; calculating a second regularization term according to the displacement parameters; and weighting the network loss function, the first regularization term, and the second regularization term to obtain the network loss value of the deep neural network.

[0058] The displacement module 420 is specifically configured for: rounding off the floating-point displacement amounts in the preset displacement parameter to obtain an integer displacement amounts; and displacing respectively, according to the integer displacement amounts in the preset displacement parameter, each of channels of the input feature map along the axes, to obtain the displaced feature map.

[0059] In embodiments of the present application, a solution is provided as follows. An input feature map of a network layer is obtained. Each of channels of the input feature map of the network layer is displaced respectively, along each of axes, according to a preset displacement parameter. In this way, a displacement feature map is obtained. Convolution operation is performed on the displacement feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer. Each of channels of the input feature map of the network layer is displaced along axes respectively. A misplacement of the input feature map of the network layer is used, to communicate information in the receptive field. As such, the subsequent 1×1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information on a spatial domain dimension. The whole process requires only a single 1×1 convolution. This effectively improves the operation efficiency of the DNN.

[0060] In order to improve the operation efficiency of the DNN, an embodiment of the present application further provides an electronic device including, as shown in Fig. 5 a processor 501 and a machine-readable storage medium 502.

[0061] The machine-readable storage medium 502 is configured to store machine executable instructions that can be executed by the processor 501.

[0062] The processor 501 is configured to perform the method according to the embodiment of the present application when executing the machine executable instructions stored on the machine-readable storage medium 502.

[0063] The data transmission between the machine-readable storage medium 502 and the processor 501 may be conducted by wired connection or wireless connection, and the electronic device may communicate with other devices through wired communication interface or wireless communication interface.

[0064] The above machine-readable storage medium may include a Random Access Memory (RAM), and may also include a Non-volatile memory (NVM), such as at least one disk memory. Optionally, the machine-readable storage medium may also be at least one storage device located away from the processor.

[0065] The above processor may be a general purpose processor, which includes a Central Processing Unit (CPU), or a Network Processor (NP). It may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0066] In embodiments of the present application, a processor of an electronic device can read and execute machine-executable instructions stored in a machine reable storage medium so as to implement the following operations. An input feature map of a network layer is obtained. Each of channels of the input feature map of the network layer is displaced respectively, along each of axes, according to a preset displacement parameter. In this way, a displacement feature map is obtained. Convolution operation is performed on the displacement feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer. Each of channels of the input feature map of the network layer is displaced along axes respectively. A misplacement of the input feature map of the network layer is used, to communicate information in the receptive field. As such, the subsequent 1×1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information on a spatial domain dimension. The whole process requires only a single 1×1 convolution. This effectively improves the operation efficiency of the DNN.

[0067] In addition, in correspondence with the deep neural network operation method according to above embodiments, embodiments further provide a machine-readable storage medium having machine executable instructions stored thereon which, when executed by a processor, cause the processor to perform the deep neural network operation method described above.

[0068] In embodiments of the present application, machine-executable instructions stored in a machine reable storage medium can be executed so as to perform the following operations. An input feature map of a network layer is obtained. Each of channels of the input feature map of the network layer is displaced respectively, along each of axes, according to a preset displacement parameter. In this way, a displacement feature map is obtained. Convolution operation is performed on the displacement feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer. Each of channels of the input feature map of the network layer is displaced along axes respectively. A misplacement of the input feature map of the network layer is used, to communicate information in the receptive field. As such, the subsequent 1×1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information on a spatial domain dimension. The whole process requires only a single 1×1 convolution. This effectively improves the operation efficiency of the DNN.

[0069] In addition, embodiments of the present application further provide an application program which, when executed, performing the deep neural network operation method according to the embodiments of the present application.

[0070] In embodiments of the present application, an application program can be executed to perform the following operations. An input feature map of a network layer is obtained. Each of channels of the input feature map of the network layer is displaced respectively, along each of axes, according to a preset displacement parameter. In this way, a displacement feature map is obtained. Convolution operation is performed on the displacement feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer. Each of channels of the input feature map of the network layer is displaced along axes respectively. A misplacement of the input feature map of the network layer is used, to communicate information in the receptive field. As such, the subsequent 1×1 convolution kernel may not only aggregate information on a channel dimension, but also aggregate information on a spatial domain dimension. The whole process requires only a single 1×1 convolution. This effectively improves the operation efficiency of the DNN.

[0071] The embodiments of the electronic device, machine-readable storage media and application program are described briefly since they are substantially similar to the above method embodiment, and details of related parts can be known with reference to the description of the method embodiment.

[0072] It should be noted that the relationship terms used herein such as "first", "second", and the like are only used for distinguishing one entity or operation from another entity or operation, but do not necessarily require or imply that there is any actual relationship or order between these entities or operations. Moreover, the terms "comprise", "comprise" or any other variants thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or devices comprising a series of elements comprise not only those elements listed but also those not specifically listed or the elements intrinsic to these processes, methods, articles, or devices. Without further limitations, elements defined by the sentences "comprise(s) a..." or "comprise(s) a..." do not exclude that there are other identical elements in the processes, methods, articles, or devices which comprise these elements.

[0073] All the embodiments are described in corresponding ways, same or similar parts in each of the embodiments can be referred to one another, and the parts emphasized are differences to other embodiments. In particular, the embodiment of the system is described briefly since it is substantially similar to the method embodiment, and the related parts can be referred to the corresponding description of the method embodiment.

[0074] The embodiments described above are simply preferable embodiments of the present application.

Claims

1. A computer-implemented method for operating a deep neural network, the method comprising: obtaining (S101) an input feature map of a network layer of the deep neural network; obtaining a preset displacement parameter by learning in advance; displacing (S102) respectively, according to the preset displacement parameter, each channel of the input feature map of the network layer along axes, to obtain a displaced feature map; and performing (S103) a convolution operation on the displaced feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer; wherein the preset displacement parameter comprises displacement amounts of the channels along the axes, and wherein the learning comprises: obtaining an initial displacement parameter; converting the displacement amounts in the initial displacement parameter into floating-point displacement amounts; obtaining an output function of the network layer through a bilinear interpolation algorithm according to the input feature map of the network layer and the floating-point displacement amounts; determining a network loss value of the deep neural network according to input data of the deep neural network, the 1×1 convolution kernel, the initial displacement parameter and a label corresponding to the input data, wherein the input data comprises an image and the label indicates a segmentation result for the image or an object present in the image, and wherein integer displacement amounts obtained by rounding off the floating-point displacement amounts are used in determining the network loss value, and the integer displacement amounts would not cover the floating-point displacement amounts; and calculating, according to the output function of the network layer, a gradient of the network loss value with respect to the displacement parameter, and updating the displacement parameter in multiple iterations by a gradient descent method to obtain the preset displacement parameter, wherein when updating the displacement parameter in multiple iterations, the floating-point displacement amounts are updated, and wherein determining the network loss value comprises: calculating a network loss function of the deep neural network according to the input data of the deep neural network, the 1×1 convolution kernel, the initial displacement parameter, and the label corresponding to the input data; calculating a first regularization term according to the 1×1 convolution kernel; calculating a second regularization term according to the displacement parameter; and weighting the network loss function, the first regularization term, and the second regularization term to obtain the network loss value of the deep neural network, wherein, the displacing (S102) comprises: rounding off the floating-point displacement amounts in the preset displacement parameter which is obtained in the process of obtaining the preset displacement parameter by learning, to obtain integer displacement amounts; and displacing respectively, according to the integer displacement amounts in the preset displacement parameter, each of channels of the input feature map along the axes, to obtain the displaced feature map.

2. The method according to claim 1, wherein, the displacement amounts are an average distribution of multiple preset different offsets.

3. An apparatus for operating a deep neural network, the apparatus comprising: an obtaining module (410) configured for obtaining an input feature map of a network layer of the deep neural network; a displacement module (420) configured for displacing respectively, according to a preset displacement parameter, each channel of the input feature map of the network layer along axes, to obtain a displaced feature map, wherein the preset displacement parameter comprises displacement amounts of the channel in the axes, and the preset displacement parameter is obtained by learning in advance; an operation module (430) configured for performing convolution operation on the displaced feature map with a 1×1 convolution kernel to obtain an output feature map of the network layer; and a learning module configured for: obtaining an initial displacement parameter; converting the displacement amounts in the initial displacement parameter into floating-point displacement amounts; obtaining an output function of the network layer through a bilinear interpolation algorithm according to the input feature map of the network layer and the floating-point displacement amounts; determining a network loss value of the deep neural network according to input data of the deep neural network, the 1×1 convolution kernel, the displacement parameter and a label corresponding to the input data, wherein the input data comprises an image and the label indicates a segmentation result for the image or an object present in the image, and wherein integer displacement amounts obtained by rounding off the floating-point displacement amounts are used in determining the network loss value, and the integer displacement amounts would not cover the floating-point displacement amounts; and calculating, according to the output function of the network layer, a gradient of the network loss value with respect to the displacement parameter, and updating the displacement parameter in multiple iterations by a gradient descent method to obtain the preset displacement parameter, wherein when updating the displacement parameter in multiple iterations, the floating-point displacement amounts are updated; and wherein the learning module is further configured for: calculating a network loss function of the deep neural network according to the input data of the deep neural network, the 1×1 convolution kernel, the initial displacement parameter, and the label corresponding to the input data; calculating a first regularization term according to the 1×1 convolution kernel; calculating a second regularization term according to the displacement parameters; and weighting the network loss function, the first regularization term, and the second regularization term to obtain the network loss value of the deep neural network, wherein the displacement module (420) is further configured for: rounding off the floating-point displacement amounts in the preset displacement parameter which is obtained in the process of obtaining the preset displacement parameter by learning, to obtain an integer displacement amounts; and displacing respectively, according to the integer displacement amounts in the preset displacement parameter, each of channels of the input feature map along the axes, to obtain the displaced feature map.

4. The apparatus according to claim 3, wherein, the displacement amounts are an average distribution of multiple preset different offsets.

5. A machine-readable storage medium having machine-executable instructions stored thereon which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 2.

6. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 2.