Data processing device, data processing system and data processing method

Abstract

The system comprises a data processing unit (10) for learning a neural network, and an encoding unit (11) that generates encoded data in which model header information for identifying a model of the neural network, layer header information for identifying a layer of the neural network, and edge weight information of the layer unit are encoded. The encoding unit (11) encodes layer structure information indicating the layer structure of the neural network and a new layer flag indicating whether each layer being encoded is an update from a layer of a reference model or a new layer.

Description

Technology Field

[0001] The present invention relates to a data processing device, a data processing system, and a data processing method for generating encoded data in which information regarding the configuration of a neural network is encoded. Background Technology

[0002] Machine learning is a method for solving the problems of classification (identification) and regression of input data. Machine learning includes a method called a neural network that simulates the neural circuits (neurons) of the brain. In a neural network (hereinafter referred to as NN), classification (identification) or regression of input data is performed by a probability model (identification model, generative model) represented by a network in which neurons are connected to each other.

[0003] In addition, NNs can be made more high-performance by optimizing their parameters through learning using large amounts of data. However, recent NNs are becoming larger in scale, and as the size of NN data tends to increase, the computational load of computers using NNs is also increasing.

[0004] For example, Non-patent Literature 1 describes a technique for scalar-quantizing and then encoding edge weights (including bias values), which are information representing the configuration of a NN. By scalar-quantizing and then encoding the edge weights, the data size of the data regarding the edges is compressed. Prior art literature

[0005] (Non-patent Document 1) Vincent Vanhoucke, Andrew Senior, Mark Z. Mao, "Improving the speed of neural networks on CPUs", Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011. The problem to be solved

[0006] In a system where multiple clients are connected to a server via a data transmission network, data representing the structure of a learned NN is encoded on the server side, and the encoded data is decoded on the client side, allowing each of the multiple clients to perform data processing using the NN learned on the server. In conventional systems, when the structure of the NN is updated, information regarding layers that have not been updated is transmitted to the clients in addition to information regarding the updated layers. Consequently, there was a problem in that the size of the transmitted data could not be reduced.

[0007] The present invention aims to solve the above problem by obtaining a data processing device, a data processing system, and a data processing method capable of reducing the data size of data representing the structure of an NN. means of solving the problem

[0008] A data processing device according to the present invention comprises a data processing unit for learning a neural network, a model header information for identifying a model of the neural network, a layer header information for identifying a layer of the neural network, and a encoding unit for generating encoded data in which weight information of each edge belonging to a layer identified by the layer header information is encoded, and the encoding unit encodes layer structure information representing the layer structure of the neural network. Effects of the invention

[0009] According to the present invention, an encoding unit encodes layer structure information representing the layer structure of an NN and a new layer flag indicating whether each layer being encoded is an update from a layer of a reference model or a new layer. Since only the information regarding the updated layer among the data representing the structure of the NN is encoded and transmitted, the data size of the data representing the structure of the NN can be reduced. Brief explanation of the drawing

[0010] FIG. 1 is a block diagram showing the configuration of a data processing system according to Embodiment 1. FIG. 2 is a diagram showing an example of the configuration of an NN. FIG. 3 is a block diagram showing the configuration of a data processing device (encoder) according to Embodiment 1. FIG. 4 is a block diagram showing the configuration of a data processing device (decoder) according to Embodiment 1. FIG. 5 is a flowchart showing the operation of a data processing device (encoder) according to Embodiment 1. FIG. 6 is a flowchart showing the operation of a data processing device (decoder) according to Embodiment 1. FIG. 7 is a diagram showing an example of encoded data in Embodiment 1. FIG. 8 is a diagram showing another example of encoded data in Embodiment 1. FIG. 9 is a diagram showing an example of convolution processing of one-dimensional data in Embodiment 1. FIG. 10 is a diagram showing an example of convolution processing of two-dimensional data in Embodiment 1. FIG. 11 is of an NN Fig. 12 is a diagram showing a matrix of edge weight information for each node in the first layer. Fig. 13 is a diagram showing a matrix of edge weight information for each node in the first layer of the NN. Fig. 14 is a diagram showing a matrix of edge weight information for the quantization stage in the convolution layer. Fig. 15 is a block diagram showing a modified example of a data processing device (encoder) according to Embodiment 1. Fig. 16 is a diagram showing an overview of the update of encoded data in Embodiment 1. Fig. 17 is a diagram showing the configuration of a network model corresponding to the update of encoded data shown in Fig. 16. Fig. 18 is a diagram showing an example of layer structure information included in the model information header. Fig. 19 is a diagram showing an example of layer identification information corresponding to the layer structure information included in the model information header.FIG. 20a is a block diagram showing a hardware configuration for realizing the function of a data processing device according to Embodiment 1. FIG. 20b is a block diagram showing a hardware configuration for executing software for realizing the function of a data processing device according to Embodiment 1. Specific details for implementing the invention

[0011] Embodiment 1.

[0012] FIG. 1 is a block diagram showing the configuration of a data processing system according to embodiment 1. In the data processing system shown in FIG. 1, a server (1) is connected to a client (3-1, 3-2, ..., 3-N) via a data transmission network (2). N is a natural number greater than or equal to 2. The server (1) is a data processing device that generates a high-performance NN by optimizing the parameters of a neural network (NN) through learning using a large amount of data, and is the first data processing device provided by the data processing system shown in FIG. 1.

[0013] The data transmission network (2) is a network through which data exchanged between a server (1) and clients (3-1, 3-2, ..., 3-N) is transmitted, and is the Internet or an intranet. For example, in the data transmission network (2), information for generating NN is transmitted from the server (1) to the clients (3-1, 3-2, ..., 3-N).

[0014] A client (3-1, 3-2, ..., 3-N) is a device that generates an NN learned by a server (1) and performs data processing using the generated NN. For example, a client (3-1, 3-2, ..., 3-N) is a device having communication functions and data processing functions, such as a personal computer (PC), a camera, or a robot. Each of the clients (3-1, 3-2, ..., 3-N) is a second data processing device equipped with the data processing system shown in FIG. 1.

[0015] In the data processing system shown in FIG. 1, each of the clients (3-1, 3-2, ..., 3-N) has a different value representing the data size and appropriate performance of the NN model and parameters. For this reason, even if the NN model and parameters are encoded using the technique described in Non-Patent Literature 1, it is necessary to compress them to a data size suitable for each of the clients (3-1, 3-2, ..., 3-N), which increases the processing load of encoding.

[0016] Thus, in the data processing system according to embodiment 1, the server (1) generates encoded data in which information including model header information for identifying the model of the NN, layer header information for identifying the layer of the NN, and edge weights including layer-unit bias values ​​(hereinafter, unless specifically stated otherwise, edge weights are assumed to include bias values) is encoded, and transmits the data to clients (3-1, 3-2, ..., 3-N) via a data transmission network (2). Each of the clients (3-1, 3-2, ..., 3-N) can decode only the information regarding the necessary layer from the encoded data transmitted from the server (1) via the data transmission network (2). By doing so, the processing load of encoding in the server (1) is reduced, and the size of the data transmitted from the server (1) to the data transmission network (2) can be reduced.

[0017] Here, the configuration of the NN is described. Figure 2 is a diagram showing an example of the configuration of the NN. As shown in Figure 2, input data (x1, x2,…, x N1 ) is processed in each layer of the NN, and the processing result (y1,…, y NL ) is printed. N l(l=1, 2,…, L) represents the number of nodes in the lth layer, and L represents the number of layers in the NN. As shown in FIG. 2, the NN has an input layer, a hidden layer, and an output layer, and each of these layers has a structure in which multiple nodes are connected by edges. The output value of each of the multiple nodes can be calculated from the output value of the node of the previous layer connected by the edge, the weight of the edge, and an activation function set for each layer.

[0018] In NNs, for example, there are Convolutional Neural Networks (CNNs) that have not only fully-connected layers but also convolutional layers and pooling layers. In CNNs, it is possible to create networks that realize data processing other than classification and regression, such as networks that realize filter processing of data.

[0019] For example, by taking an image or voice as input, filter processing of the image or voice to achieve noise removal or high quality improvement of the input signal, high-frequency restoration processing of voice in which high frequencies are lost in compressed voice, restoration processing (inpainting) of an image in which some image regions are missing, or super-resolution processing of an image can be realized by CNN. In CNN, a NN can be constructed that combines a generative model and an identification model to determine the authenticity of data using an identification model that determines whether the data was generated by the generative model.

[0020] Recently, a new NN called a Generative Adversarial Network has also been proposed, in which the identification model is adversarially trained so that the generative model does not generate data that is detected as not true data by the identification model, and the identification model detects that the data generated by the generative model is not true data. In this NN, it is possible to generate high-precision generative and identification models.

[0021] FIG. 3 is a block diagram showing the configuration of a data processing device (encoder) according to embodiment 1. The data processing device shown in FIG. 3 is a first data processing device that learns an NN using a training data set and an evaluation data set, and generates encoded data of model information (hereinafter referred to as model information) representing the configuration of the NN, and is, for example, the server (1) shown in FIG. 1.

[0022] The data processing device shown in FIG. 3 comprises a data processing unit (10) and an encoding unit (11). The data processing unit (10) is a first data processing unit that learns an NN and comprises a learning unit (101), an evaluation unit (102), and a control unit (103). The encoding unit (11) generates encoded data in which model header information identifying the model of the NN learned by the learning unit (101), layer header information identifying the layer of the NN, and edge weight information at the layer level are encoded. In addition, the encoding unit (11) encodes layer structure information of the layer being encoded (encoded layer) and encodes a new layer flag. The layer structure information is information indicating the layer structure of the NN. The new layer flag is flag information for identifying whether the layer is a newly added layer or a layer that updates an existing layer, and details will be described later.

[0023] The learning unit (101) performs learning processing of the NN using a learning data set and generates model information of the learned NN. The model information is output from the learning unit (101) to the evaluation unit (102). Additionally, the learning unit (101) has model information for encoding controlled by the control unit (103) described later, and outputs the model information for encoding to the encoding unit (11) when it receives a learning completion instruction from the control unit (103). The evaluation unit (102) generates an NN using the model information and performs inference processing from the evaluation data set using the generated NN. The value of the evaluation index obtained as a result of the inference processing is the evaluation result, and the evaluation result is output from the evaluation unit (102) to the control unit (103). The evaluation index is set in the evaluation unit (102) and is, for example, an inference precision or an output value of a loss function.

[0024] The control unit (103) determines whether the model of the NN learned by the learning unit (101) is updated and whether the learning of the NN by the learning unit (101) is completed based on the evaluation value obtained as an evaluation result by the evaluation unit (102), and controls the learning unit (101) according to the determination result. For example, the control unit (103) compares the evaluation value with a model update determination criterion and determines whether to update the model information as encoding model information based on the result of this comparison. Also, the control unit (103) compares the evaluation value with a learning completion determination criterion and determines whether to complete the learning of the NN by the learning unit (101) based on the result of this comparison. Also, such determination criteria are determined from the history of the evaluation value.

[0025] FIG. 4 is a block diagram showing the configuration of a data processing device (decoder) according to embodiment 1. The data processing device shown in FIG. 4 is a second data processing device that decodes encoded data generated by the encoding unit (11) shown in FIG. 3 to generate an NN, and uses the generated NN to infer one or more evaluation data, and is, for example, a client (3-1, 3-2, ..., 3-N) shown in FIG. 1.

[0026] The data processing device shown in FIG. 4 is equipped with a decoding unit (201) and an inference unit (202). The decoding unit (201) decodes model information from encoded data generated by the encoding unit (11). For example, the decoding unit (201) can decode only the information necessary for the data processing device shown in FIG. 4 from the encoded data.

[0027] The inference unit (202) is a second data processing unit that generates an NN using model information decoded by the decoding unit (201) and performs data processing using the generated NN. For example, the data processing is inference processing for evaluation data using the NN. The inference unit (202) performs inference processing for evaluation data using the NN and outputs an inference result.

[0028] Next, the operation of the data processing system according to Embodiment 1 will be described. FIG. 5 is a flowchart showing the operation of a data processing device (encoder) according to Embodiment 1, and illustrates a data processing method using the data processing device shown in FIG. 3. A learning unit (101) learns the NN (step ST1). For example, the learning unit (101) performs learning of the NN using a learning data set and outputs model information obtained through this learning to an evaluation unit (102).

[0029] Model information is information representing the configuration of a NN model and is composed of layer structure information representing the structure of each layer and weight information for each edge belonging to the layer. The layer structure information includes layer type information, configuration information regarding layer type, and information necessary to configure the layer in addition to the edge weights. Information necessary to configure the layer in addition to the edge weights includes, for example, an activation function. Layer type information is information representing the type of layer, and by referring to the layer type information, it is possible to identify the type of layer, such as a convolution layer, a pooling layer, or a total combination layer.

[0030] Configuration information regarding the layer type is information indicating the configuration of the layer of the type corresponding to the layer type information. For example, if the layer type corresponding to the layer type information is a convolution layer, the configuration information regarding the layer type includes the number of channels performing convolution, the data size and shape of the convolution filter (kernel), the convolution interval (stride), the presence or absence of padding at the boundaries of the input signal for convolution processing, and, if padding is required, the padding method. Furthermore, if the layer type corresponding to the layer type information is a pooling layer, the configuration information regarding the layer type includes a pooling method such as max pooling or average pooling, the shape of the kernel performing the pooling processing, the pooling interval (stride), the presence or absence of padding at the boundaries of the input signal for pooling processing, and, if padding is required, the padding method.

[0031] In the information representing the weights of each edge, there are cases where weights are set independently at each edge, such as in the entire combined layer. On the other hand, there are cases where the edge weights are common at the convolution filter (kernel) level (channel level), such as in the convolution layer, that is, where the edge weights are common at a single filter.

[0032] The evaluation unit (102) evaluates the NN (step ST2). For example, the evaluation unit (102) generates the NN using model information generated by the learning unit (101) and performs inference processing on an evaluation dataset using the generated NN. The evaluation result is output from the evaluation unit (102) to the control unit (103). The evaluation result is, for example, an output value of the inference precision or loss function.

[0033] Next, the control unit (103) determines whether to update the model information (step ST3). For example, the control unit (103) determines that if the evaluation value generated by the evaluation unit (102) does not satisfy the model update determination criteria, the model information for encoding held by the learning unit (101) is not updated, and if the evaluation value satisfies the model update determination criteria, the model information for encoding is updated.

[0034] As an example of a criterion for model update, when the evaluation value is the output of a loss function, the evaluation value from the current training is smaller than the minimum evaluation value in the training history since the start of training. As another example, when the evaluation value is inference precision, the evaluation value from the current training is larger than the maximum evaluation value in the training history since the start of training.

[0035] In addition, the conversion unit of the training history may be arbitrary. For example, a training history is provided for each model identification number (model_id) described below. In this case, if the model in question does not have the reference model identification number (reference_model_id) described below, training is initiated with no training history. That is, in the first step ST3, the model information is necessarily updated. On the other hand, if the model in question has the reference model identification number, the training history (History A) of the model indicated by the reference model identification number is referenced. By doing so, it is possible to prevent the model in question from being updated to a model with a worse evaluation value (low inference precision, high loss function value, etc.) than the model indicated by the reference model identification number during training. At this time, if the model identification number of the model in question and the reference model identification number are the same, the training history (History A) corresponding to the reference model identification number is updated whenever training of the model in question is performed. Meanwhile, if the model identification number of the model in question and the reference model identification number are different, the learning history (History A) corresponding to the reference model identification number is copied as the initial value of the learning history (History B) of the model identification number of the model in question, and then the learning history (History B) of the model in question is updated whenever the model in question is trained.

[0036] When it is determined by the control unit (103) that model information is to be updated (step ST3; YES), the learning unit (101) updates the model information for encoding with the model information (step ST4). For example, the control unit (103) generates model update instruction information indicating that there is an update to the model information and outputs learning control information including the model update instruction information to the learning unit (101). The learning unit (101) updates the model information for encoding with the model information according to the model update instruction information included in the learning control information.

[0037] Meanwhile, if it is determined that the model information is not to be updated (step ST3; NO), the control unit (103) generates model update instruction information indicating that there is no update to the model information, and outputs learning control information including the model update instruction information to the learning unit (101). The learning unit (101) does not update the model information for encoding according to the model update instruction information included in the learning control information.

[0038] Next, the control unit (103) compares the evaluation value with the learning completion determination criterion and determines whether to complete the learning of the NN by the learning unit (101) based on the result of this comparison (step ST5). For example, if the learning completion determination criterion is whether the evaluation value generated by the evaluation unit (102) has reached a specific value, the control unit (103) determines that the learning of the NN by the learning unit (101) is completed if the evaluation value generated by the evaluation unit (102) satisfies the learning completion determination criterion, and determines that the learning of the NN by the learning unit (101) is not completed if the evaluation value does not satisfy the learning completion determination criterion. Alternatively, if the learning completion determination criteria are based on recent learning history, such as determining that learning is complete when M consecutive times (M is a predetermined integer greater than or equal to 1) or when no model information update (step ST3; No) is selected, the control unit (103) determines that the learning of the NN by the learning unit (101) is not complete if the learning history does not satisfy the learning completion determination criteria.

[0039] If the control unit (103) determines that the learning of the NN is complete (step ST5; yes), the learning unit (101) outputs model information for encoding to the encoding unit (11) and proceeds to the processing of step ST6. Meanwhile, if the control unit (103) determines that the learning of the NN is not complete (step ST5; no), the processing from step ST1 is executed.

[0040] The encoding unit (11) encodes the model information for encoding input from the learning unit (101) (step ST6). The encoding unit (11) encodes the model information for encoding generated by the learning unit (101) in layers of the NN and generates encoding data composed of header information and layer-unit encoding data. In addition, the encoding unit (11) encodes the layer structure information and encodes the new layer flag.

[0041] FIG. 6 is a flowchart illustrating the operation of a data processing device (decoder) according to embodiment 1, and illustrates the operation of the data processing device shown in FIG. 4. The decoding unit (201) decodes model information from encoded data encoded by the encoding unit (11) (step ST11). Next, the inference unit (202) generates an NN from the model information decoded by the decoding unit (201) (step ST12). The inference unit (202) performs inference processing on evaluation data using the generated NN and outputs an inference result (step ST13).

[0042] Next, the encoding of model information by the encoding unit (11) in step ST6 of FIG. 5 will be described in detail. For the encoding of model information by the encoding unit (11), encoding methods (1) or (2) may be used, for example. Alternatively, it may be defined which of the encoding methods (1) or (2) is used for each parameter. For example, by making the header information (1) and the weight data (2), the decoder can easily interpret the header information without variable-length decoding, while high compression can be achieved by variable-length decoding of the weight data, which occupies most of the data size of the encoded data, thereby suppressing the data size of the entire encoded data.

[0043] (1) If a parameter that constitutes each piece of information included in the model information is a bit sequence itself described with bit precision defined in the parameter, and if header information exists, the data arranged in a pre-set order including the header information is used as encoded data. Bit precision is, for example, 8 bits of int or 32 bits of float, which is the bit precision defined in the parameter.

[0044] (2) The bit sequence itself, which is encoded by a variable length encoding method set for each parameter of the information included in the model information, is the encoded data that is arranged in a pre-set order including header information.

[0045] FIG. 7 is a diagram showing an example of encoded data in embodiment 1, and the encoded data of (1) or (2) may be arranged in the order shown in FIG. 7. The encoded data shown in FIG. 7 is composed of a collection of data called a data unit, and the data unit includes a non-layer data unit and a layer data unit. The layer data unit is a data unit in which layer data, which is encoded data in a layer unit, is stored.

[0046] Layer data consists of a start code, a data unit type, a layer information header, and weight data. The layer information header contains encoded layer header information for identifying the layers of the NN. The weight data contains encoded weight information of the edges belonging to the layer indicated by the layer information header. In addition, in the encoded data shown in FIG. 7, the order of each layer data unit does not necessarily have to be the same as the order of each layer of the NN and is arbitrary. This is because it is possible to identify which layer of the NN each layer data unit belongs to by the layer identification number (layer_id) described later.

[0047] A non-layer data unit is a data unit in which data other than layer data is stored. For example, a non-layer data unit stores a start code, a data unit type, and a model information header. The model information header is encoded model header information for identifying the NN model.

[0048] A start code is a code stored at the leading position of a data unit and used to identify the leading position of a data unit. A client (3-1, 3-2, ..., 3-N) (hereinafter referred to as the decoding side) can identify the leading position of a non-layer data unit or a layer data unit by referring to the start code. For example, if 0x000001 is defined as the start code, data other than the start code stored in the data unit is set so that 0x000001 does not occur. By doing so, the leading position of the data unit can be identified from the start code.

[0049] To prevent 0x000001 from occurring, for example, 03 is inserted into the third byte of encoded data ranging from 0x000000 to 0x000003 to make it 0x00000300 to 0x00000303, and the original state can be restored by converting 0x000003 to 0x0000 during decoding. Additionally, as long as the start code is a uniquely identifiable bit sequence, a bit sequence other than 0x000001 may be defined as the start code. Also, if there is a method to identify the leading position of a data unit, a start code may not be used. For example, a bit sequence that identifies the end of a data unit may be attached to the end of the data unit. Alternatively, a start code may be attached only to the beginning of non-layer data units, and the data size of each layer data unit may be encoded as part of the model information header. By doing so, it is possible to identify the short-circuit location of each layer data unit from the above information.

[0050] The data unit type is data stored after the start code in the data unit to identify the type of data unit. The data unit type has a predefined value for each type of data unit. By referencing the data unit type stored in the data unit, the decoding side can identify whether the data unit is a non-layer data unit or a layer data unit, and can also identify which non-layer data unit or layer data unit it is.

[0051] The model information header in the non-layer data unit includes a model identification number (model_id), the number of layer data units within the model (num_layers), and the number of coded layer data units (num_coded_layers). The model identification number is a number used to identify the model of the NN. Therefore, basically, each individual model has an independent number, but if the data processing device (decoder) according to Embodiment 1 receives a new model having the same model identification number as a previously received model, the model having the said model identification number is overwritten. The number of layer data units within the model is the number of layer data units constituting the model identified by the model identification number. The number of coded layer data units is the number of layer data units actually existing within the coded data. In the example of FIG. 7, since layer data units (1) to (n) exist, the number of coded layer data units is n. Furthermore, the number of coded layer data units must be less than or equal to the number of layer data units within the model.

[0052] The layer information header in the layer data unit includes a layer identification number (layer_id) and layer structure information. The layer identification number is a number used to identify a layer. To enable identification of which layer a layer belongs to by the layer identification number, the method for assigning values ​​to the layer identification number is fixedly defined in advance. For example, numbers are assigned sequentially starting from the layer closest to the input layer, such as assigning 0 to the input layer of the NN and 1 to the next layer. The layer structure information represents the configuration of each layer of the NN and includes information necessary to construct the layer, in addition to layer type information, configuration information regarding layer types, and edge weights. For example, it is the information regarding only the corresponding layer portion of the model_structure_information and layer_id_information described later. Additionally, as layer structure information, it has weight_bit_length, which indicates the bit precision of the edge weight of the corresponding layer. For example, if weight_bit_length=8, it indicates that the weight is 8-bit data. Therefore, the bit precision of the edge weights can be set on a layer-by-layer basis. This enables adaptive control, such as changing the bit precision on a layer-by-layer basis according to the importance of the layer (the degree to which bit precision affects the output result).

[0053] In addition, although a layer information header containing layer structure information has been described so far, the model information header may include all layer structure information (model_structure_information) included in the encoded data and layer identification information (layer_id_information) corresponding to this layer structure information. The decoding side can determine the composition of the layer for each layer identification number by referring to the model information header. Furthermore, since the composition of the layer for each layer identification number can be determined by referring to the model information header in the above case, the layer information header may contain only the layer identification number. By doing so, when the data size of a layer data unit is larger than the data size of a non-layer data unit, the data size of each layer data unit can be reduced, and the maximum data size of the data unit within the encoded data can be reduced.

[0054] In the layer data unit, layer-unit encoded weight data is stored after the layer information header. The weight data includes a non-zero flag and non-zero weight data. The non-zero flag is a flag indicating whether the value of the edge weight is zero, and the non-zero flag is set for all edge weights belonging to the corresponding layer.

[0055] Non-zero weight data is data set following the non-zero flag in the weight data, and the value of the weight is set for the weight where the non-zero flag indicates non-zero (significant). In FIG. 7, weight data (1) to weight data (m), each representing a non-zero weight value, are set as non-zero weight data. The number of non-zero weight data m is the total number of weights M of the corresponding layer l. lThe following applies. Also, regarding weight data for layers where the weight values ​​are non-zero edges sparse, the non-zero weight data is small and consists almost entirely of non-zero flags, so the data size of the weight data is significantly reduced.

[0056] FIG. 8 is a diagram showing another example of encoded data in embodiment 1, and the encoded data of (1) or (2) may be arranged in the order shown in FIG. 8. Unlike FIG. 7, the data structure of the weight data in FIG. 8 is such that in the non-zero weight data, the weights of all edges belonging to the corresponding layer are arranged and listed by bit plane in order from the most significant bit. Additionally, in the layer information header, bit plane data location identification information is set to indicate the leading position of each bit representing the edge weight.

[0057] For example, if the bit precision defined by the edge weight is X, then the weights of all edges belonging to the corresponding layer are each described with bit precision X. The encoding unit (11) sets the 1st bit weight data (1), 1st bit weight data (2), ..., 1st bit weight data (m), which are 1st bit non-zero weight data among the bit sequences of these weights, as the 1st bit each non-zero weight data. This process is repeated from the 2nd bit non-zero weight data to the X-bit non-zero weight data. Also, the 1st bit weight data (1), 1st bit weight data (2), ..., 1st bit weight data (m) are non-zero weight data constituting the 1st bit plane.

[0058] The decoding side can identify the necessary encoded data among the layer-unit encoded data based on bit plane data location identification information and decode the identified encoded data with arbitrary bit precision. That is, the decoding side can select only the necessary encoded data from the encoded data and can decode the NN model information according to the environment of the decoding side. In addition, the bit plane data location identification information may be information capable of identifying the short-circuit location between bit plane data, information indicating the leading position of each bit plane data, or information indicating the data size of each bit plane data.

[0059] In order to transmit all encoded data representing the configuration of the NN to the decoding side, if the transmission bandwidth of the data transmission network (2) is insufficient, the encoding unit (11) may limit the non-zero weight data transmitted according to the transmission bandwidth of the data transmission network (2) among the encoded data. For example, among the bit sequence of weight information described with 32-bit precision, the non-zero weight data of the upper 8 bits is selected as the transmission target. The decoding side can recognize that, from the start code listed after this non-zero weight data, a layer data unit corresponding to the next layer is listed after the 8th non-zero weight data in the encoded data. Also, the decoding side can accurately decode the weights with a value of zero by referring to the non-zero flag in the weight data.

[0060] In order to improve the inference precision at a given bit precision when weight data is decoded at an arbitrary bit precision on the decoding side, the encoding unit (11) may include an offset that is added to the weight when decoded at each bit precision in the layer information header. For example, the encoding unit (11) adds a constant offset on a layer-by-layer basis to a bit sequence of weights described by bit precision, obtains the offset that results in the highest precision, and includes the obtained offset in the layer information header for encoding.

[0061] Additionally, the encoding unit (11) may include the offset of the edge weights in all layers of the NN in the model information header for encoding. Additionally, the encoding unit (11) may set a flag indicating whether to include the offset in the layer information header or the model information header, and, for example, include the offset in the encoded data only when the flag is valid.

[0062] The encoding unit (11) may encode the difference between the edge weight value and a specific value. As for the specific value, for example, the weight of the previous encoding order may be used. Also, the weight of the corresponding edge belonging to a layer one level higher (a layer close to the input layer) may be used as the specific value, or the weight of the corresponding edge of the model before the update may be used as the specific value.

[0063] In addition, the encoding unit (11) has the functions shown in (A), (B) and (C).

[0064] (A) The encoding unit (11) has an extended encoding function that divides the base encoding data and the reinforcement encoding data into encodings.

[0065] (B) The encoding unit (11) has the function of encoding the difference with the edge weights in the reference NN.

[0066] (C) The encoding unit (11) has the function of encoding only partial information (e.g., layer-level information) in the reference NN as information for updating the NN.

[0067] Explain the example of (A).

[0068] The encoding unit (11) quantizes the edge weights using a quantization method that is predefined for the edge weights, and sets the data encoded with the weights after quantization as base encoded data, and sets the data encoded with the quantization error as a weight as reinforcement encoded data. Since the bit precision of the weights that become the base encoded data is reduced compared to the weights before quantization due to quantization, the data size is reduced. When the transmission bandwidth for transmitting the encoded data to the decoding side is insufficient, the data processing device according to embodiment 1 transmits only the base encoded data to the decoding side. On the other hand, when the transmission bandwidth for transmitting the encoded data to the decoding side is sufficient, the data processing device according to embodiment 1 transmits the reinforcement encoded data to the decoding side in addition to the base encoded data.

[0069] Reinforcement encoding data can be two or more. For example, the encoding unit (11) sets the quantization value obtained when the quantization error is additionally quantized as the first reinforcement encoding data, and the quantization error as the second reinforcement encoding data. In addition, the quantization error of the second reinforcement encoding data may be divided by the additionally quantized quantization value and the quantization error to form the number of reinforcement encoding data of the target. In this way, by using extended encoding, it is possible to transmit encoded data according to the transmission bandwidth and transmission allowable time of the data transmission network (2).

[0070] Additionally, the encoding unit (11) may encode up to the upper M bits of the non-zero weighted data shown in FIG. 8 as base encoded data, and divide the remaining bit sequence into one or more to form one or more reinforced encoded data. In this case, the encoding unit (11) resets the non-zero flag in each of the base encoded data and the reinforced encoded data. The weight that becomes 0 in the upper bit of the reinforced encoded data must be 0.

[0071] Explain the example of (B).

[0072] In the case where a model of the NN before retraining by the learning unit (101) exists, the encoding unit (11) may encode the difference between the edge weights in the NN model after retraining and the corresponding edge weights in the model before retraining. Additionally, retraining includes transfer learning or additional learning. In a data processing system, if the configuration of the NN is updated at a high frequency or if the change in the distribution of the training data with each retraining is small, the difference in edge weights is small, so the data size of the encoded data after retraining is reduced.

[0073] The encoding unit (11) includes a reference model identification number (reference_model_id) in the model information header to identify the model prior to update to be referenced, in addition to the model identification number. In the example of (B), it becomes possible to identify the model prior to retraining from the reference model identification number. Additionally, the encoding unit (11) may set a flag (reference_model_present_flag) in the model information header to indicate whether there is a reference source in the encoded data. At this time, the encoding unit (11) first encodes the flag (reference_model_present_flag), and additionally sets the reference model identification number in the model information header only when the flag indicates that the encoded data is for updating the model.

[0074] For example, in the data processing system shown in FIG. 1, even if the update frequency of NNs differs between clients or data processing is performed using NNs of different models, the client can correctly identify which model the encoded data is for updating by referring to the reference model identification number. If it is identified from the reference model identification number that the encoded data is for updating a model that is not on the client side, the client can also transmit it to the server (1).

[0075] Explain the example of (C).

[0076] The learning unit (101) may fix one or more arbitrary layers from the upper part (input layer side) of the NN and retrain only some layers when there is a model of the NN before retraining, for example, for the purpose of fine-tuning. In this case, the encoding unit (11) encodes only information indicating the configuration of the layers updated by retraining. By doing so, the data size of the encoded data transmitted to the decoding side during the update of the NN is reduced. In addition, the number of encoded layer data units (num_coded_layers) in the encoded data becomes less than or equal to the number of layer data units (num_layers) within the model. On the decoding side, the layer to be updated can be specified by referring to the reference model identification number included in the model information header and the layer identification number included in the layer information header.

[0077] Next, data processing by the learning unit (101), evaluation unit (102), and inference unit (202) is explained.

[0078] FIG. 9 is a diagram showing an example of convolution processing of one-dimensional data in embodiment 1, and shows a convolution layer that performs convolution processing of one-dimensional data. One-dimensional data includes, for example, voice data and time series data. The convolution layer shown in FIG. 9 has 9 nodes 10-1 to 10-9 in the previous layer and 3 nodes 11-1 to 11-3 in the next layer. Edges 12-1, 12-6, and 12-11 are assigned the same weight, edges 12-2, 12-7, and 12-12 are assigned the same weight, edges 12-3, 12-8, and 12-13 are assigned the same weight, edges 12-4, 12-9, and 12-14 are assigned the same weight, and edges 12-5, 12-10, and 12-15 are assigned the same weight. Also, there are cases where the weights from edges 12-1 to 12-5 are all different values, and cases where multiple weights are the same value.

[0079] Of the nine nodes 10-1 to 10-9 of the previous layer, five nodes are connected to one node of the next layer with the above weights. The kernel size K is 5, and the kernel is defined by a combination of these weights. For example, as shown in FIG. 9, node 10-1 is connected to node 11-1 via edge 12-1, node 10-2 is connected to node 11-1 via edge 12-2, node 10-3 is connected to node 11-1 via edge 12-3, node 10-4 is connected to node 11-1 via edge 12-4, and node 10-5 is connected to node 11-1 via edge 12-5. The kernel is defined by a combination of the weights of edges 12-1 to 12-5.

[0080] Node 10-3 is connected to Node 11-2 via Edge 12-6, Node 10-4 is connected to Node 11-2 via Edge 12-7, Node 10-5 is connected to Node 11-2 via Edge 12-8, Node 10-6 is connected to Node 11-2 via Edge 12-9, and Node 10-7 is connected to Node 11-2 via Edge 12-10. The kernel is defined by a combination of the weights of Edges 12-6 through 12-10.

[0081] Node 10-5 is connected to Node 11-3 via Edge 12-11, Node 10-6 is connected to Node 11-3 via Edge 12-12, Node 10-7 is connected to Node 11-3 via Edge 12-13, Node 10-8 is connected to Node 11-3 via Edge 12-14, and Node 10-9 is connected to Node 11-3 via Edge 12-15. The kernel is defined by a combination of the weights of Edges 12-11 through 12-15.

[0082] The learning unit (101), evaluation unit (102), and inference unit (202) perform convolution operations at intervals of a number of steps (S=2 in FIG. 9) for each kernel using a combination of edge weights in the convolution layer when processing input data using a CNN. The combination of edge weights is determined by learning for each kernel. Also, in CNNs for image recognition purposes, the NN is often constructed from a convolution layer having multiple kernels.

[0083] FIG. 10 is a diagram showing an example of convolution processing of two-dimensional data in Embodiment 1, illustrating convolution processing of two-dimensional data such as image data. Among the two-dimensional data shown in FIG. 10, the kernel (20) has a size in the x-direction of K x , the size in the y-direction is K y It is the block region of. The kernel size K is, K=K x ×K yThe learning unit (101), evaluation unit (102), or inference unit (202) has, in 2D data, a number of steps in the x-direction S x The interval and number of steps in the y-direction S y At intervals, a convolution operation of the data for each kernel (20) is performed. Here, step S x , S y is an integer greater than or equal to 1.

[0084] FIG. 11 is a diagram showing the matrix of edge weight information for each node in the l-th layer (l=1, 2,…, L), which is the total coupling layer of the NN. FIG. 12 is a diagram showing the matrix of the quantization step of edge weight information for each node in the l-th layer (l=1, 2,…, L), which is the total coupling layer of the NN.

[0085] In the case of the NN, the weight w for each layer shown in Fig. 11 ij The combination of these becomes the data that constitutes the network. For this reason, in multi-layered neural networks such as deep neural networks, the amount of data generally exceeds several hundred Mbytes, requiring a large memory size. i is the node index, and i=1, 2,…, N l is. j is an edge index, and j=1, 2,…, N l-1 +1 (including offset).

[0086] Therefore, in the data processing device according to embodiment 1, weight information is quantized to reduce the amount of data of edge weight information. For example, as shown in FIG. 12, the quantization step q ij is the edge weight w ij It is set for each. The quantization step may be multiple node indices or multiple edge indices, or multiple node indices and edge indices may be common. By doing so, the quantization information to be encoded is reduced.

[0087] FIG. 13 is a diagram showing a matrix of edge weight information in a convolution layer. FIG. 14 is a diagram showing a matrix of the quantization step of edge weight information in a convolution layer. In a convolution layer, the edge weight for a single kernel is common across all nodes, and the kernel can be made into a small region by reducing the number of edges combined per node, i.e., the kernel size K. FIG. 13 shows the edge weight w i＇j＇ g is data set for each kernel, and FIG. 14 is quantization step q i＇j＇ This is data configured for each kernel. Also, i' is the kernel index, and i'=1, 2,… , M l (l=1, 2,…, L). j' is an edge index, and j'=1, 2,…, K l +1 (including offset).

[0088] The quantization steps may be commonized into multiple kernel indices, multiple edge indices, or multiple kernel indices and edge indices. By doing so, the quantization information to be encoded is reduced. For example, all quantization steps within a layer may be commonized so that there is one quantization step per layer, or all quantization steps within a model may be commonized so that there is one quantization step per model.

[0089] FIG. 15 is a block diagram showing the configuration of a modified example of a data processing device (encoder) according to embodiment 1. The data processing device shown in FIG. 15 is a first data processing device that learns an NN using a learning data set and an evaluation data set and generates encoded data of the NN's model information, and is, for example, the server (1) shown in FIG. 1. The data processing device shown in FIG. 15 is equipped with a data processing unit (10A), an encoding unit (11), and a decoding unit (12).

[0090] The data processing unit (10A) is a data processing unit that generates and learns an NN, and is equipped with a learning unit (101A), an evaluation unit (102), and a control unit (103). The encoding unit (11) encodes model information generated by the learning unit (101A) and generates encoded data composed of header information and layer-unit encoded data. The decoding unit (12) decodes model information from the encoded data generated by the encoding unit (11). In addition, the decoding unit (12) outputs model information of completed decoding to the learning unit (101A).

[0091] The learning unit (101A) performs training of the NN using a training data set, similar to the learning unit (101), and generates model information representing the configuration of the trained NN. Additionally, the learning unit (101A) generates the NN using the model information of the completed decoding and retrains the parameters of the generated NN using the training data set.

[0092] When retraining as described above, it is possible to achieve high precision while maintaining a small data size of the encoded data by fixing the weights of some edges during retraining. For example, by performing retraining with the weights of non-zero flags fixed at 0, it is possible to optimize the weights while preventing the data size of the encoded data from exceeding the weights of the edges prior to retraining.

[0093] A data processing device is equipped with a decoding unit (12), and the data processing unit (10A) learns an NN using information decoded by the decoding unit (12). By doing so, for example, even if the encoding unit (11) performs irreversible encoding that causes encoding distortion, the data processing device can generate and learn an NN based on the actual decoding result of the encoded data, and under conditions where constraints are imposed on the data size of the encoded data, it is possible to learn an NN with the influence of encoding error suppressed to the minimum.

[0094] In a data processing system having a configuration as shown in FIG. 1, equipped with a data processing device shown in FIG. 3 as a server (1) and a data processing device shown in FIG. 4 as a client (3-1, 3-2, ..., 3-N), the data output from the intermediate layer of the NN can be used as a feature quantity for data processing of image data and voice data, such as image retrieval or matching described in Reference 1 below.

[0095] (Reference 1) ISO / IEC JTC1 / SC29 / WG11 / m39219, "Improved retrieval and matching with CNN feature for CDVA", Chengdu, China, Oct. 2016.

[0096] For example, when using output data of an intermediate layer of an NN as an image feature for image processing such as image search, matching, or object tracking, the image feature of HOG (Histogram of Oriented Gradients), SIFT (Scale Invariant Feature Transform), or SURF (Speeded Up Robust Features), which is an image feature used in the conventional image processing, is replaced or added. By doing so, the image processing can be realized in the same processing order as the image processing using the conventional image feature. In the data processing system according to Embodiment 1, the encoding unit (11) encodes model information representing the configuration of the NN up to the intermediate layer that outputs the image feature.

[0097] Additionally, a data processing device functioning as a server (1) performs data processing such as image search using the feature quantity of the data processing. A data processing device functioning as a client generates an NN from the encoded data to the intermediate layer, and performs data processing such as image search using the data output from the intermediate layer of the generated NN as a feature quantity.

[0098] In a data processing system, the encoding unit (11) encodes model information representing the configuration up to the intermediate layer of the NN, thereby increasing the compression rate of parameter data by quantization and reducing the amount of data of weight information before encoding. The client generates an NN using the model information decoded by the decoding unit (201) and performs data processing using the data output from the intermediate layer of the generated NN as a feature quantity.

[0099] Additionally, the data processing system according to embodiment 1 has a configuration as shown in FIG. 1, and may have a data processing device shown in FIG. 3 or FIG. 15 as a server (1), and a data processing device shown in FIG. 4 as a client (3-1, 3-2, ..., 3-N). In the data processing system having this configuration, a new_layer_flag is set in the encoded data. When the new_layer_flag is 0 (invalid), the layer corresponding to the new_layer_flag is a layer that is updated based on the reference layer. When the new_layer_flag is 1 (valid), the layer corresponding to the new_layer_flag is a layer that is newly added.

[0100] If the new layer flag is 0 (invalid), a flag (channel_wise_update_flag) is set to identify whether the edge weights of the layer corresponding to the new layer flag are updated on a channel-by-channel basis. If this flag is 0 (invalid), the edge weights of all channels are encoded. If this flag is 1 (valid), a channel-by-channel weight update flag (channel_update_flag) is set. This update flag indicates whether there is an update from the reference layer for each channel. If this update flag is 1 (valid), the channel weights are encoded, and if it is 0 (invalid), they become the same weights as the reference layer.

[0101] Additionally, as a layer information header, information indicating the number of channels of the layer (num_channels) and information indicating the number of edge weights per channel (weights_per_channels) are set. The weights_per_channels of a certain layer l is kernel size K l N, the number of indices from layer l-1, which is +1 or 1 layer prior. l-1 It becomes +1.

[0102] Since the encoded data has the aforementioned new layer flag, it is possible to determine the number of channels and the number of channel-unit weights solely from the encoded data of the layer data unit. Therefore, as a decoding process of the layer data unit, the update flag of the channel-unit weights can be decoded.

[0103] In addition, the case where a flag for identifying whether weights are updated on a channel-by-channel basis is set to 1 (valid) is restricted to when the reference layer and the number of channels are the same. This is because when the reference layer and the number of channels are different, the correspondence relationship between each channel between the reference layer and the layer corresponding to the flag becomes unclear.

[0104] FIG. 16 is a diagram showing an overview of the update of encoded data in embodiment 1. In FIG. 16, the data shown above consists of a non-layer data unit and layer data units (1) to (4), and is encoded sequentially starting from the layer data unit (4), as in FIG. 7. In the non-layer data unit, as model header information, a model identification number (model_id)=0, the number of layer data units in the model (num_layers)=4, layer structure information (model_structure_information) and layer identification information (layer_id_information) are set, and a flag (reference_model_present_flag) indicating whether there is a reference source in the encoded data is set to 0 (invalid).

[0105] In the layer data unit (1), the layer identification number (layer_id) is set to 0, the information (num_channels) indicating the number of channels (filters, kernels) of the layer is set to 32, and the information (weights_per_channels) indicating the number of weights (including bias values) per channel (filters, kernels) is set to 76. Also, in the layer data unit (2), the layer identification number (layer_id) is set to 1, the information (num_channels) indicating the number of channels of the layer is set to 64, and the information (weights_per_channels) indicating the number of weights per channel is set to 289.

[0106] In the layer data unit (3), the layer identification number (layer_id) is set to 2, the information indicating the number of channels of the layer (num_channels) is set to 128, and the information indicating the number of weights per channel (weights_per_channels) is set to 577. Also, in the layer data unit (4), the layer identification number (layer_id) is set to 3, the information indicating the number of channels of the layer (num_channels) is set to 100, and the information indicating the number of weights per channel (weights_per_channels) is set to 32769.

[0107] In FIG. 16, the data shown on the lower side is data updated from the data shown on the upper side using layer structure information, a layer update flag, and a new layer flag, and is composed of a non-layer data unit and layer data units (1'), (2), (3), (5), and (4'). For a client to which the data shown on the upper side is transmitted, the non-layer data unit and the layer data units (1'), (5), and (4') need to be transmitted, but the layer data units (2) and (3) are not updated and do not need to be transmitted.

[0108] In the non-layer data unit shown at the bottom of FIG. 16, as model header information, the model identification number (model_id) = 10, the number of layer data units in the model (num_layers) = 5, the layer structure information (model_structure_information) and the layer identification information (layer_id_information) are set, the flag (reference_model_present_flag) indicating whether there is a reference source in the encoded data is set to 1 (valid), the reference model identification number (reference_model_id) is set to 0, and the number of encoded layer data units (num_coded_layers) is set to 3.

[0109] In the layer data unit (1'), the layer identification number (layer_id) is 0, the new layer flag (new_layer_flag) is set to 0, the information indicating the number of channels of the layer (num_channels) is set to 32, and the information indicating the number of weights per channel (weights_per_channels) is set to 76. Also, since the flag for identifying whether weights are updated per channel (channel_wise_update_flag) is set to 1 (valid), the channel-wise weight update flag (channel_update_flag) is set.

[0110] Layer data units (2) with a layer identification number (layer_id) of 1 and layer data units (3) with a layer identification number (layer_id) of 2 are not subject to update and are therefore not included in the encoded data. Accordingly, the above model header information is set such that the number of layer data units (num_layers) in the model is 5 and the number of encoded layer data units (num_coded_layers) is 3.

[0111] In the layer data unit (5), the layer identification number (layer_id) is 4, and the new layer flag (new_layer_flag) is set to 1 (valid). Also, the information indicating the number of channels of the layer (num_channels) is set to 256, and the information indicating the number of weights per channel (weights_per_channels) is set to 1153.

[0112] In the layer data unit (4'), the layer identification number (layer_id) is 3, the new layer flag (new_layer_flag) is set to 0, the information indicating the number of channels of the layer (num_channels) is set to 100, and the information indicating the number of weights per channel (weights_per_channels) is set to 16385. In addition, the flag for identifying whether weights are updated per channel (channel_wise_update_flag) is set to 0 (invalid), so there is no update to the weights per channel.

[0113] In the data shown below, the layer data units (1) and (4) in the data shown above are updated to layer data units (1’) and (4’), and a layer data unit (5) with a layer identification number of 4 is added.

[0114] FIG. 17 is a diagram showing the configuration of a network model corresponding to the update of encoded data shown in FIG. 16. In FIG. 17, the network model shown on the left is a network model realized by decoding the data shown on the upper side of FIG. 16. Also, the network model shown on the right is a network model realized by decoding the data shown on the lower side of FIG. 16.

[0115] In the layer data unit (1'), since the channel_wise_update_flag for identifying whether the weight is updated on a channel-by-channel basis is 1, the weights of several channels are updated from the layer data unit (1). Also, as the layer data unit (5) is added and the layer data unit (4) is updated to the layer data unit (4'), in the network model shown on the right, a 2D convolution layer and a 2D max pooling layer are added between the entire connected layer and the entire connected layer.

[0116] FIG. 18 is a diagram showing an example of layer structure information included in a model information header. As the overall layer structure information (model_structure_information) included in the model information header, text information such as that shown in FIG. 18 may be set. The text information shown in FIG. 18 is text information representing the layer structure of a model according to the standard specification described in Reference 2, called NNEF (Neural Network Exchange Format).

[0117] (Reference 2) "Neural Network Exchange Format", The Khronos NNEF Working Group, Version 1.0, Revision 3, 2018-06-13.

[0118] In FIG. 18, (A) the network model of model_id=0 is the network model corresponding to the data shown on the upper side of FIG. 16 (the network model shown on the left side of FIG. 17). (B) The network model of model_id=10 is the network model corresponding to the data shown on the lower side of FIG. 16 (the network model shown on the right side of FIG. 17).

[0119] FIG. 19 is a diagram showing an example of layer identification information (layer_id_information) corresponding to layer structure information included in a model information header, and shows layer identification information with a layer identification number set corresponding to the layer structure information of FIG. 18. In FIG. 19, (A) the network model of model_id=0 is layer identification information corresponding to the network model shown on the left side of FIG. 17. (B) the network model of model_id=10 is layer identification information corresponding to the network model shown on the right side of FIG. 17. The weight and bias values ​​of each layer are assigned to the layer identification number, and the value corresponds to the data shown in FIG. 16.

[0120] File data, such as a file containing model_structure_information, which is overall layer structure information, and layer_id_information, which is information indicating the layer identification number corresponding to the overall layer structure information, is configured as encoded data by being inserted after the information indicating the byte count of the file data in the model information header. Alternatively, it is also possible to configure the model information header to include a URL (Uniform Resource Locator) indicating the source of the file data. Furthermore, to allow selection of which of these configurations is used, a flag identifying which configuration is used may be set before the file data or the URL in the model information header. The identification flag may be common to both model_structure_information and layer_id_information, or may be provided individually. If the former is used, the amount of information in the model information header can be reduced, and if the latter is used, it can be set independently depending on the preconditions for use.

[0121] In addition, the model information header includes information indicating the format of the text information. For example, NNEF is information with an index of 0, and other formats are information with an index of 1 or higher. By this, it is possible to identify which format is used and to decode it accurately.

[0122] In addition, the layer structure information represented by text information as shown in FIGS. 18 and 19, and the information representing a layer identification number corresponding to the layer structure information, can be applied to all systems shown in Embodiment 1. Furthermore, from model_structure_information and layer_id_information, it is possible to identify which layer of data within the model each layer data unit belongs to solely from the corresponding encoded data. Therefore, when updating the model (when the reference_model_present_flag is valid), it is possible to set a model that is not generated from the encoded data shown in the present embodiment as a reference model. That is, the encoded data shown in the present embodiment can set any model as a reference model by having model_structure_information and layer_id_information as part of the model information header. However, in this case, the correspondence between the reference model identification number (reference_model_id) and the reference model needs to be defined separately.

[0123] Next, a hardware configuration for realizing the functions of a data processing device according to Embodiment 1 will be described. In the data processing device according to Embodiment 1, the functions of the data processing unit (10) and the encoding unit (11) are realized by a processing circuit. That is, the data processing device according to Embodiment 1 is equipped with a processing circuit for executing the processing from step ST1 to step ST6 of FIG. 5. The processing circuit may be dedicated hardware, but may also be a CPU (Central Processing Unit) that executes a program stored in memory.

[0124] FIG. 20a is a block diagram showing a hardware configuration for realizing the function of a data processing device according to Embodiment 1. In FIG. 20a, the processing circuit (300) is a dedicated circuit that functions as the data processing device shown in FIG. 3. FIG. 20b is a block diagram showing a hardware configuration for executing software that realizes the function of a data processing device according to Embodiment 1. In FIG. 20b, the processor (301) and the memory (302) are connected to each other by a signal bus.

[0125] In the case where the processing circuit is dedicated hardware as shown in FIG. 20a, the processing circuit (300) corresponds to, for example, a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a FPGA (Field-Programmable Gate Array), or a combination thereof. In addition, the functions of the data processing unit (10) and the encoding unit (11) may be realized by other processing circuits, or these functions may be combined and realized by a single processing circuit.

[0126] When the processing circuit is the processor shown in FIG. 20b, the functions of the data processing unit (10) and the encoding unit (11) are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in memory (302). The processor (301) realizes the functions of the data processing unit (10) and the encoding unit (11) by reading and executing the program stored in memory (302). That is, the data processing device according to embodiment 1 is provided with memory (302) for storing a program in which the processing from step ST1 to step ST6 shown in FIG. 5 is executed as a result when executed by the processor (301). Such a program causes the computer to execute the sequence or method of the data processing unit (10) and the encoding unit (11). The memory (302) may be a computer-readable storage medium in which a program is stored to enable the computer to function as the data processing unit (10) and the encoding unit (11).

[0127] Memory (302) includes, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically-EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD, etc.

[0128] Additionally, regarding the functions of the data processing unit (10) and the encoding unit (11), some parts may be realized with dedicated hardware, and some parts may be realized with software or firmware. For example, the function of the data processing unit (10) may be realized with a processing circuit as dedicated hardware, and the function of the encoding unit (11) may be realized by the processor (301) reading and executing a program stored in memory (302). In this way, the processing circuit can realize each of the above functions by hardware, software, firmware, or a combination thereof.

[0129] Also, although the data processing device shown in FIG. 3 has been described, the same applies to the data processing device shown in FIG. 4. For example, the data processing device shown in FIG. 4 is equipped with a processing circuit for executing the processing from step ST11 to step ST13 of FIG. 6. This processing circuit may be dedicated hardware, or it may be a CPU that executes a program stored in memory.

[0130] In the case where the processing circuit is dedicated hardware as shown in FIG. 20a, the processing circuit (300) corresponds to, for example, a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. In addition, the functions of the decoding unit (201) and the inference unit (202) may be realized by other processing circuits, or these functions may be realized by consolidating them into a single processing circuit.

[0131] If the processing circuit is the processor shown in FIG. 20b, the functions of the decoding unit (201) and the inference unit (202) are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in memory (302). The processor (301) realizes the functions of the decoding unit (201) and the inference unit (202) by reading and executing the program stored in memory (302). That is, the data processing device shown in FIG. 4 is equipped with memory (302) for storing a program in which the processing from step ST11 to step ST13 shown in FIG. 6 is executed as a result when executed by the processor (301). Such a program causes the computer to execute the sequence or method of the decoding unit (201) and the inference unit (202). The memory (302) may be a computer-readable medium in which a program is stored to enable the computer to function as the decoding unit (201) and the inference unit (202).

[0132] Additionally, regarding the functions of the decoding unit (201) and the inference unit (202), some parts may be realized in dedicated hardware, and some parts may be realized in software or firmware. For example, the function of the decoding unit (201) may be realized in a processing circuit as dedicated hardware, and the function of the inference unit (202) may be realized by the processor (301) reading and executing a program stored in memory (302).

[0133] As described above, in the data processing device according to embodiment 1, the encoding unit (11) encodes layer structure information and encodes a layer update flag, and when the layer update flag indicates an update of the layer structure, encodes a new layer flag. Since only the information regarding the updated layer among the data representing the structure of the NN is encoded and transmitted, the data size of the data representing the structure of the NN can be reduced.

[0134] Additionally, the encoding unit (11) encodes information indicating the configuration of the NN and generates encoded data consisting of header information and layer-unit encoded data. Since only the information regarding the layers required by the decoding side can be encoded, the processing load for encoding information regarding the configuration of the NN is reduced, and the size of the data transmitted to the decoding side can be reduced.

[0135] In a data processing device according to embodiment 1, the encoding unit (11) encodes the weight information of the edge belonging to the layer of the NN in bit plane units starting from the upper bit. By doing so, the data size of the encoded data transmitted to the decoding side can be reduced.

[0136] In a data processing device according to embodiment 1, an encoding unit (11) encodes information regarding one or more layers specified as header information. By doing so, only information regarding layers required by the decoding side is encoded, and the data size of the encoded data transmitted to the decoding side can be reduced.

[0137] In a data processing device according to embodiment 1, the encoding unit (11) encodes the difference between the value of the edge weight belonging to the layer specified as header information and a specific value. By doing so, the data size of the encoded data transmitted to the decoding side can be reduced.

[0138] In a data processing device according to embodiment 1, the encoding unit (11) encodes edge weight information by dividing it into base encoded data and reinforcement encoded data. By doing so, it is possible to realize the transmission of encoded data according to the transmission bandwidth and transmission allowance time of the data transmission network (2).

[0139] Furthermore, the present invention is not limited to the above embodiments, and within the scope of the present invention, any combination of each embodiment, any modification of any component of each embodiment, or any omission of any component in each embodiment is possible.

[0140] (Industrial Applicability)

[0141] The data processing device according to the present invention can be used, for example, in image recognition technology. Explanation of the symbols

[0142] 1: Server 2: Data transmission network 3-1~3-N: Clients 10, 10A: Data Processing Unit 10-1~10-9, 11-1~11-3: Node 11: Encoding section 12: Decoding part 12-1~12-15: Edges 20: Colonel 101, 101A: Learning Department 102: Evaluation Unit 103: Control Unit 201: Decoding unit 202: Inference unit 300: Processing circuit 301: Processor 302: Memory

Claims

Claim 1 A data processing device comprising a data processing unit for learning a neural network, a model header information for identifying a model of the neural network, a layer header information for identifying a layer of the neural network, and a encoding unit for generating encoded data in which weight information of each edge belonging to a layer identified by the layer header information is encoded, wherein the encoding unit encodes layer structure information indicating the layer structure of the neural network and a new layer flag indicating whether each layer being encoded is an update from a layer of a reference model or a new layer. Claim 2 A data processing device according to claim 1, wherein the encoding unit encodes weight information of edges belonging to a layer in bit plane units starting from the upper bit. Claim 3 A data processing device according to claim 1 or 2, wherein the encoding unit encodes edge weight information belonging to one or more layers identified by the layer header information. Claim 4 A data processing device according to claim 1 or 2, wherein the encoding unit encodes the difference between the value of the edge weight and a specific value. Claim 5 A data processing device according to claim 1 or 2, wherein the encoding unit encodes edge weight information by dividing it into base encoded data and reinforcement encoded data, wherein the base encoded data is data encoded by quantizing the edge weights and encoding the weights after quantization, and the reinforcement encoded data is data encoded by considering the quantization error as a weight. Claim 6 A data processing device according to claim 1 or 2, comprising a decoding unit that decodes encoded data generated by the encoding unit, and wherein the data processing unit learns the neural network using the information decoded by the decoding unit. Claim 7 A data processing system comprising: a first data processing unit for learning a neural network; a first data processing device having a model header information for identifying a model of the neural network, a layer header information for identifying a layer of the neural network, and a first data processing unit for generating encoded data in which weight information of each edge belonging to a layer identified by the layer header information is encoded; a decoding unit for decoding from the encoded data generated by the first data processing unit; and a second data processing device having a second data processing unit for generating the neural network and performing data processing using the neural network using the information decoded by the second data processing unit, wherein the first data processing unit encodes layer structure information indicating the layer structure of the neural network and a new layer flag indicating whether each layer being encoded is an update from a layer of a reference model or a new layer. Claim 8 A data processing system according to claim 7, wherein the encoding unit encodes information up to the intermediate layer of the neural network, and the second data processing device performs data processing using the data output from the intermediate layer of the neural network as a feature quantity. Claim 9 A data processing method comprising: a data processing unit having a step of learning a neural network; an encoding unit having a step of generating encoded data in which model header information for identifying a model of the neural network, layer header information for identifying a layer of the neural network, and weight information of each edge belonging to the layer identified by the layer header information are encoded; wherein the encoding unit encodes layer structure information indicating the layer structure of the neural network and a new layer flag indicating whether each layer being encoded is an update from a layer of a reference model or a new layer.

Images