Instruction processing method, data processing method and device

Abstract

The present invention discloses an instruction processing device, comprising: a first register adapted to store a plurality of first source data; a second register adapted to store a plurality of second source data; a third register adapted to store sequence information, the sequence information being adapted to indicate the execution order of multiplication operations on the second source data; a decoder adapted to receive and decode a multiply-accumulate instruction, the multiply-accumulate instruction indicating that the first, second, and third registers are used as first, second, and third operands, respectively; and an execution unit adapted to execute the decoded multiply-accumulate instruction, so as to obtain the plurality of first source data from the first register, obtain the plurality of second source data from the second register, obtain the sequence information from the third register, multiply the plurality of first source data respectively by the corresponding second source data indicated by the sequence information, obtain a plurality of multiplication results, and add the plurality of multiplication results to obtain a multiply-accumulate result. The present invention also discloses a corresponding instruction processing method, a data processing method, and a device.

Description

Technical Field

[0001] The present invention relates to the field of processor technology, and in particular to an instruction processing device having an instruction set including multiply-accumulate instructions, and a data processing method for protecting neural network weights through multiply-accumulate instructions. Background Art

[0002] Neural networks are widely used in fields such as pattern recognition, image processing, natural language processing, and automatic control. A neural network model consists of two parts: the network structure, which includes the number of processing layers and the connections between processing nodes in each layer; and the model parameters, which include the weights and biases of each processing node in the neural network.

[0003] Currently, for different application scenarios, developers typically directly adopt open-source, general-purpose neural network structures (or make minor modifications). They then use private, application-specific datasets to train the neural network, determine parameters such as weights, and thus generate a neural network model suitable for that specific scenario. Because the weights are generated using private datasets for specific application scenarios, it is necessary to protect the weights of the neural network model to prevent competitors from copying them.

[0004] There are two main existing weight protection methods:

[0005] One approach is to use a trusted execution environment (such as TrustZone) to save neural network weights, but this approach will introduce more complex software and hardware design and increase the development costs of terminal equipment manufacturers.

[0006] The other approach is to encrypt and store the weights, and then decrypt them during the neural network forward calculation. Since the weight data volume is large (much larger than the network structure data volume), decrypting the weights takes a long time and requires a lot of memory resources, making it difficult to meet the real-time requirements of the neural network forward calculation.

[0007] To this end, a simple and efficient neural network weight protection scheme is needed. Summary of the Invention

[0008] To this end, the present invention provides an instruction processing method, a data processing method and an apparatus, in an effort to solve or at least alleviate the above problems.

[0009] According to a first aspect of the present invention, an instruction processing device is provided, comprising: a first register, suitable for storing a plurality of first source data; a second register, suitable for storing a plurality of second source data; a third register, suitable for storing sequence information, the sequence information being suitable for indicating the execution order of multiplication operations of the second source data; a decoder, suitable for receiving and decoding a multiplication-accumulation instruction, the multiplication-accumulation instruction indicating: the first register as a first operand, the second register as a second operand, and the third register as a third operand; an execution unit, respectively coupled to the first, second, and third registers and the decoder, and suitable for executing the decoded multiplication-accumulation instruction so as to obtain a plurality of first source data from the first register, obtain a plurality of second source data from the second register, obtain the sequence information from the third register, multiply the plurality of first source data with the corresponding second source data indicated by the sequence information, respectively, to obtain a plurality of multiplication results, and add the plurality of multiplication results to obtain a multiplication-accumulation result.

[0010] According to a second aspect of the present invention, there is provided an instruction processing method, comprising: receiving a multiply-accumulate instruction and decoding the multiply-accumulate instruction, wherein the multiply-accumulate instruction indicates a first register as a first operand, a second register as a second operand, and a third register as a third operand; executing the decoded multiply-accumulate instruction to obtain a plurality of first source data from the first register, obtain a plurality of second source data from the second register, obtain sequence information indicating the execution order of multiplication operations on the second source data from the third register, multiply the plurality of first source data respectively with the corresponding second source data indicated by the sequence information to obtain a plurality of multiplication results, and add the plurality of multiplication results to obtain a multiply-accumulate result.

[0011] According to a third aspect of the present invention, there is provided a data processing method suitable for performing forward calculations on a neural network, wherein the neural network includes a plurality of processing nodes, and the weight data of the neural network is stored in an order different from the arrangement order of the processing nodes. The method comprises: storing a plurality of input data, a plurality of weight data and sequence information of the processing nodes in a first register, a second register and a third register, respectively, wherein the sequence information is suitable for indicating the execution order of multiplication operations of the plurality of weight data; and using the above-mentioned instruction processing method to calculate the multiplication and accumulation results of the plurality of input data and the plurality of weight data to obtain the output data of the processing node.

[0012] According to a fourth aspect of the present invention, there is provided a data processing method, comprising: obtaining a plurality of first source data, a plurality of second source data and sequence information for indicating the execution order of multiplication operations of the plurality of second source data; determining a correspondence between the first source data and the second source data based on the sequence information; multiplying the plurality of first source data with the corresponding second source data respectively to obtain a plurality of multiplication results, and adding the plurality of multiplication results to obtain a multiplication-accumulation result.

[0013] According to a fifth aspect of the present invention, there is provided a data processing method suitable for performing forward calculations of a neural network, wherein the neural network includes a plurality of processing nodes, and the weight data of the neural network is stored in an order different from the arrangement order of the processing nodes. The method comprises: obtaining a plurality of input data, a plurality of weight data and sequence information of the processing node, wherein the sequence information is suitable for indicating the execution order of multiplication operations of the plurality of weight data; determining the correspondence between the input data and the weight data based on the sequence information; and performing weighted summation on the plurality of input data based on the weight data corresponding to each input data to determine the output data of the processing node.

[0014] According to a sixth aspect of the present invention, there is provided a computing device comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device executes the above-mentioned instruction processing method or data processing method.

[0015] According to a seventh aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the above-mentioned instruction processing method or data processing method.

[0016] According to an eighth aspect of the present invention, there is provided a system on chip comprising the above-mentioned instruction processing device.

[0017] According to a ninth aspect of the present invention, there is provided an intelligent device comprising the above-mentioned system on chip. According to the instruction processing solution of the present invention, the second source data can be protected.

[0018] In the instruction processing scheme of the present invention, the instruction processing device performs multiplication and accumulation operations on the first source data and the second source data through multiplication and accumulation instructions, that is: the first source data, the second source data and the sequence information are read into the first, second and third registers respectively, and then the decoder decodes the multiplication and accumulation instructions, the execution unit executes the decoded multiplication and accumulation instructions, and determines the execution order of the multiplication operation of the second source data according to the sequence information, so that the second source data corresponds to the first source data, and each first source data is multiplied with the corresponding second source data to obtain multiple multiplication results, and then the multiple multiplication results are added to obtain the multiplication and accumulation result.

[0019] The instruction processing scheme of the present invention implements protection of the second source data through multiplication-accumulation instructions. The second source data is stored in a random order in the memory, and sequence information is provided to indicate the execution order of the multiplication operation of the second source data. When a multiplication-accumulation operation is required on the second source data, the execution order of the multiplication operation of the second source data can be determined by the sequence information, that is, the multiplication correspondence between the first source data and the second source data is determined. The second source data is stored in a random order in the memory. Even if someone steals the second source data through illegal means, they cannot use the second source data because they cannot know the correct order of the second source data, thereby achieving protection of the second source data.

[0020] In addition, the instruction processing scheme of the present invention only adjusts the arrangement order of the second source data. The second source data can still be read by the instruction processing device and directly used for calculation through multiplication and addition instructions. No additional software and hardware structure is required, nor is there a need for complex and time-consuming encryption and decryption steps, thereby achieving simple and efficient protection of the second source data.

[0021] The second source data can be the weight data of the neural network. Accordingly, the data processing method of the present invention can achieve simple and efficient protection of the weight data. The weight data is stored in a disordered order in the memory, and sequence information is set to indicate the execution order of the multiplication operation of the weight data. When the forward calculation of the neural network is performed, the correspondence between the weight data and the input data is determined by the sequence information and a multiplication and accumulation operation is performed. The weight data is stored in a disordered order in the memory. Even if others steal the weight data by illegal means, they cannot use the weight data because they cannot know the correct arrangement order of the weight data, thereby achieving protection of the weight data.

[0022] Moreover, the data processing method of the present invention only adjusts the arrangement order of the weight data. The weight data can still be read in and directly used for multiplication and accumulation operations. No additional software and hardware structure is required, nor is there a need for complex and time-consuming encryption and decryption steps. This achieves simple and efficient protection of the weight data and can meet the real-time requirements of the forward calculation of the neural network.

[0023] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other objects, features, and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

[0025] Figure 1 FIG2 shows a schematic diagram of an instruction processing device 100 according to an embodiment of the present invention;

[0026] Figure 2 shows a schematic diagram of a register architecture 200 according to one embodiment of the present invention;

[0027] Figure 3 FIG2 shows a schematic diagram of an instruction processing device 300 according to an embodiment of the present invention;

[0028] Figure 4 A schematic diagram showing a processing procedure of a multiply-accumulate instruction according to an embodiment of the present invention is shown;

[0029] Figure 5 A schematic diagram showing a processing procedure of a multiply-accumulate instruction according to another embodiment of the present invention is shown;

[0030] Figure 6 1 shows a flow chart of an instruction processing method 600 according to one embodiment of the present invention;

[0031] Figure 7 shows a flow chart of a data processing method 700 according to one embodiment of the present invention;

[0032] Figure 8 A schematic diagram of a neural network according to an embodiment of the present invention is shown;

[0033] Figure 9 A schematic diagram showing a data processing process according to an embodiment of the present invention;

[0034] Figure 10 1000 is a flowchart of a data processing method according to an embodiment of the present invention;

[0035] Figure 11 shows a flow chart of a data processing method 1100 according to one embodiment of the present invention;

[0036] Figure 12 FIG2 shows a schematic diagram of a processor 1200 according to an embodiment of the present invention;

[0037] Figure 13shows a schematic diagram of a computing device 1300 according to one embodiment of the present invention;

[0038] Figure 14 A schematic diagram of a system on chip (SoC) 1400 is shown according to one embodiment of the present invention. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0040] Figure 1 is a schematic diagram of an instruction processing device 100 according to an embodiment of the present invention. The instruction processing device 100 has an execution unit 140, which includes a circuit operable to execute instructions (including the multiplication and accumulation instructions according to the present invention). In some embodiments, the instruction processing device 100 can be a single-core processor, a processor core of a multi-core processor, or a processing element in an electronic system. It should be noted that the processor here includes but is not limited to a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a neural network processor (NPU), etc.

[0041] In addition, it should be noted that Figure 1 The division of the various modules in the present invention is functional and can be rearranged and combined for physical implementation without departing from the scope of protection of the present invention.

[0042] Decoder 130 receives incoming instructions in the form of high-level machine instructions or macroinstructions, and decodes these instructions to generate low-level micro-operations, microcode entry points, microinstructions or other low-level instructions or control signals. Low-level instructions or control signals can implement the operation of high-level instructions by low-level (e.g., circuit level or hardware level) operations. Decoder 130 can be implemented in different ways, including but not limited to microcode, lookup table, hardware implementation, programmable logic array (PLA) etc. The present invention is not limited to the mode of implementing decoder 130, and any mode that can implement decoder 130 is within the scope of protection of the present invention.

[0043] Decoder 130 may receive incoming instructions from cache 110, memory 120, or other sources. The decoded instructions may include one or more micro-ops, microcode entry points, microinstructions, other instructions, or other control signals that reflect or are derived from the received instructions. These decoded instructions are sent to execution unit 140 and executed by execution unit 140. While executing these instructions, execution unit 140 receives data inputs from register file 150, cache 110, and / or memory 120 and generates data outputs to these.

[0044] In some embodiments, register file 150 includes architectural registers, also referred to as registers. Unless otherwise specified or clearly apparent, the phrases architectural registers, register file, and register are used herein to refer to registers that are visible to software and / or programmers (e.g., software-visible) and / or specified by macroinstructions to identify operands. These registers are distinct from other non-architectural registers in a given microarchitecture (e.g., temporary registers, reorder buffers, etc.).

[0045] To avoid obscuring the description, a relatively simple instruction processing apparatus 100 has been shown and described. It should be understood that other embodiments may have more than one execution unit. For example, the instruction processing apparatus 100 may include multiple different types of execution units, such as an arithmetic unit, an arithmetic logic unit (ALU), an integer unit, a floating point unit, etc. Other embodiments of the instruction processing apparatus may have multiple cores, logical processors, or execution engines. Figure 12-14 Several embodiments of the instruction processing apparatus 100 are provided.

[0046] According to one embodiment, register group 150 includes a vector register group 152. Vector register group 152 includes multiple vector registers 152A, which can store operands for multiply-accumulate instructions. The bit width of each vector register 152A can be 128 bits, 256 bits, 512 bits, or other values, and the present invention does not limit the bit width of vector registers 152A. Register group 150 can also include a general register group 154. General register group 154 ​​includes multiple general registers 154A, which can also store operands for multiply-accumulate instructions. In an embodiment of the present invention, multiply-accumulate instructions can be used as SIMD (Single Instruction Multiple Data) instructions to operate on vectors.

[0047] Figure 2 FIG. 2 shows a schematic diagram of a register architecture 200 according to an embodiment of the present invention. Figure 2As shown, the register architecture 200 defines 16 128-bit vector registers VR0 to VR15, as well as a series of data processing SIMD instructions for these 16 vector registers. Depending on the definition of the specific instruction, each vector register can be regarded as an array consisting of several 8-bit, 16-bit, 32-bit or even 64-bit elements. In addition, 32 32-bit general registers GR0 to GR31 are also defined in the register architecture 200. The general registers GR0 to GR31 can store some control state values ​​during SIMD instruction processing (such as the "sequence information" below), and can also store operands during general instruction processing.

[0048] According to one embodiment, Figure 1 The vector register set 152 described in the Figure 2 One or more of the vector registers VR0 to VR15 shown, Figure 1 The general register set 154 described in the Figure 2 One or more of the general registers GR0-GR31 shown.

[0049] Those skilled in the art will understand that Figure 2 The register architecture 200 shown is only an example. In other alternative embodiments, the register architecture may use register groups or registers of different bit widths, different types, and different numbers. The present invention does not limit the register architecture of the instruction processing device 100.

[0050] Figure 3 FIG. 3 is a schematic diagram of an instruction processing device 300 according to an embodiment of the present invention. Figure 3 The instruction processing device 300 shown is Figure 1 The instruction processing apparatus 100 shown is further expanded, and some components are omitted for ease of description. Figure 1 The same reference numerals are used throughout to designate the same and / or similar components.

[0051] The instruction processing device 300 is adapted to execute a multiply-accumulate instruction. According to one embodiment of the present invention, the multiply-accumulate instruction indicates the first register A0 as the first operand, specifies the second register A1 as the second operand, and specifies the third register A2 as the third operand.

[0052] A plurality of first source data and a plurality of second source data to be used for multiplication operations are stored in first register A0 and second register A1, respectively. Unless otherwise specified, hereinafter, the array consisting of the plurality of first source data in first register A0 is denoted as src0, and the i-th first source data in first register A0 is denoted as src0[i-1]; the array consisting of the plurality of second source data in second register A1 is denoted as src1, and the i-th second source data in second register A1 is denoted as src1[i-1].

[0053] For each first source data in the first register A0, there is a corresponding second source data in the second register A1 to be multiplied with it. However, the storage order of the first source data in the first register A0 is different from the storage order of the second source data in the second register A1. The first source data in the first register A0 does not necessarily correspond to the second source data in the same position in the second register A1 (for example, the third first source data src0[2] in the first register A0 does not correspond to the third second source data src1[2] in the second register A1, but corresponds to the first second source data src1[0]).

[0054] It should be noted that either the order of the first source data src0 or the order of the second source data src1 can be considered the correct order, and the other order can be considered a random order. Those skilled in the art will understand that the correct order and the random order are relative terms. Once a certain order is designated as the correct order, all other orders different from the correct order are considered random orders. For example, the order of the first source data src0 in the first register A0 can be considered the correct order, and the second source data src1 can be stored in the second register A1 in a random order.

[0055] The third register A2 stores sequence information seq, which is suitable for indicating the order in which multiplication operations are performed on the second source data src1, so as to determine the corresponding multiplication relationship between the first source data src0 and the second source data src1. According to one embodiment, the sequence information seq includes multiple sequence identifiers corresponding to the multiple first source data, each sequence identifier is suitable for indicating the storage order of the second source data corresponding to the corresponding first source data in the second register, that is, the i-th sequence identifier seq[i-1] is suitable for indicating the storage order of the second source data corresponding to the i-th first source data src0[i-1] in the second register A1.

[0056] According to one embodiment, the multiply-accumulate instruction further includes a fourth operand specifying a fourth register A3. The fourth register A3 is adapted to store a multiply-accumulate result.

[0057] According to one embodiment, the multiply-accumulate instruction can be used as a type of SIMD instruction, that is, it acts on vector data. In this case, the first, second and third registers are all vector registers. In another embodiment, the multiply-accumulate instruction can also be a conventional instruction that acts on scalar data. Accordingly, the first, second and third registers can also be general registers. The present invention is not limited to the type of operands, and all data that can perform multiply-accumulate operations are within the scope of protection of the present invention. The following describes the multiply-accumulate instruction using vector registers as an example. Those skilled in the art will understand that the various aspects described subsequently can also be applied to other types of data.

[0058] like Figure 3 As shown, decoder 130 is adapted to decode the multiply-accumulate instruction to determine a first register corresponding to A0, a second register corresponding to A1, and a third register corresponding to A2 in vector register file 152. According to one embodiment, decoder 130 further determines a fourth register corresponding to A3.

[0059] The execution unit 140 is coupled to the first register A0 , the second register A1 , the third register A2 and the decoder 130 , respectively, and is adapted to execute the decoded multiply-accumulate instruction.

[0060] Specifically, the execution unit 140 obtains a plurality of first source data src0[0], src0[1], src0[2], ... from the first register A0, obtains a plurality of second source data src1[0], src1[1], src1[2], ... from the second register A1, and obtains sequence information seq from the third register A2. The execution order of the multiplication operation of the second source data src1 is determined according to the sequence information seq, thereby determining the multiplication correspondence between the first source data and the second source data. The plurality of first source data src0[0], src0[1], src0[2], ... are respectively multiplied by the corresponding second source data indicated by the sequence information seq to obtain a plurality of multiplication results, and the plurality of multiplication results are added to obtain a multiplication-accumulation result.

[0061] For example, Figure 4 As shown, the first register A0 stores four first source data src0[0] to src0[3], and the second register A1 stores four second source data src1[0] to src1[3]. The first register A0 and the second register A1 can be, for example, 128-bit vector registers, and the first source data and the second source data can be, for example, 32-bit floating-point numbers.

[0062] The third register A2 stores sequence information seq, which includes four sequence identifiers seq[0] to seq[3].

[0063] Sequence identifiers seq[0] to seq[3] are used to indicate the storage order of the second source data corresponding to the first source data src0[0] to src0[3] in the second register A1. For example, seq[0]=2 indicates that the first source data src0[0] corresponds to the second source data src1[2]; seq[1]=1 indicates that the first source data src0[1] corresponds to the second source data src1[1]; seq[2]=3 indicates that the first source data src0[2] corresponds to the second source data src1[3]; seq[3]=0 indicates that the first source data src0[3] corresponds to the second source data src1[0].

[0064] According to the multiplication operation execution order indicated by the sequence information seq, the second source data corresponding to each first source data is determined, and the first source data is multiplied by the corresponding second source data to obtain multiple multiplication results. That is, src0[0] is multiplied by src1[2], src0[1] is multiplied by src1[1], src0[2] is multiplied by src1[3], and src0[3] is multiplied by src1[0] to obtain four multiplication results m0 to m3. The four multiplication results are then added to obtain a multiplication-accumulation result result, and the multiplication-accumulation result result is stored in the fourth register A3.

[0065] According to one embodiment, the fourth register A3 stores the accumulated data src3. The multiply-accumulate instruction also includes a selector acc. The selector acc is an immediate value. When decoding the multiply-accumulate instruction, the decoder 130 also determines the value of the selector acc and sends the value of acc to the execution unit 140. The execution unit 140 determines how to perform the addition operation based on the value of the selector acc.

[0066] For example, if the selection flag acc does not exist or the value of the selection flag acc is 0, the execution unit 140 does not use the accumulated data src3, but directly stores the multiplication-accumulation result result in the fourth register A3 to overwrite the accumulated data src3. If the selection flag acc exists or the value of the selection flag acc is 1, the execution unit 140 uses the accumulated data src3, adds the multiplication-accumulation result result to the accumulated data src3 to obtain an addition result, and stores the addition result in the fourth register A3 to replace the accumulated data src3.

[0067] According to one embodiment, the plurality of first source data in the first register A0 are divided into a plurality of data groups, and the execution unit 140 is adapted to add the multiplication results in each data group to obtain a plurality of multiplication-accumulation results.

[0068] For example, Figure 5As shown, the first register A0 stores 16 first source data a0 to a15. These 16 first source data are divided into four data groups, each of which includes four first source data. That is, the first data group includes first source data a0 to a3, the second data group includes first source data a4 to a7, the third data group includes first source data a8 to a11, and the fourth data group includes first source data a12 to a15. In order to facilitate the illustration of the grouping of the first source data, Figure 5 The second and fourth data groups are displayed with a gray background.

[0069] The second register A1 stores 16 second source data b0~b15. The arrangement order of the second source data b0~b15 is different from the arrangement order of the first source data a0~a15. The second source data multiplied by the first source data ai (a=0, 1,..., 15) is not necessarily bi.

[0070] The third register A2 stores sequence information, which can determine the execution order of the multiplication operation of the second source data b0-b15, that is, determine the second source data corresponding to the first source data a0-a15. Figure 5 As shown, the sequence information includes 16 sequence identifiers, which are used to indicate the storage order of the second source data corresponding to the first source data a0 to a15 in the second register A1. According to the indication of the sequence information in the third register A2, the first source data a0 to a15 correspond to the second source data b0, b14, b2, b13, b1, b5, b9, b7, b8, b4, b10, b3, b12, b11, b6, and b15, respectively.

[0071] Each first source data is multiplied by the corresponding second source data (for example, multiplying a0 and b0, multiplying a1 and b14, multiplying a2 and b2, etc.) to obtain 16 multiplication results m0-m15.

[0072] Subsequently, the multiplication results within each data group are added together, resulting in one multiplication-accumulation result for each data group, for a total of four multiplication-accumulation results. For example, the four multiplication results m0-m3 within the first data group are added together to obtain the first multiplication-accumulation result c0; the four multiplication results m4-m7 within the second data group are added together to obtain the second multiplication-accumulation result c1; the four multiplication results m8-m11 within the third data group are added together to obtain the third multiplication-accumulation result c2; and the four multiplication results m12-m15 within the fourth data group are added together to obtain the fourth multiplication-accumulation result c3. The four multiplication-accumulation results c0-c3 are stored in the fourth register A3.

[0073] According to one embodiment, the length of the multiplication-accumulation result is the sum of the lengths of the plurality of first source data. Figure 4As shown, the first register A0 can be, for example, a 128-bit vector register, which stores four 32-bit first source data, and the multiplication-accumulation result is the sum of the lengths of the four first source data, that is, the length of the multiplication-accumulation result is 128 bits.

[0074] exist Figure 5 In the illustrated embodiment, the first register A0, the second register A1, and the fourth register A3 are, for example, all 128-bit vector registers, and the first source data a0-a15 and the second source data b0-b15 are, for example, all 8-bit unsigned integer data. The first source data a0-a15 are divided into four data groups, each of which includes four first source data. The multiplication-accumulation result of each data group is the sum of the lengths of all the first source data in that data group, that is, the length of the multiplication-accumulation result of each data group is 4*8 (the number of first source data included in the data group * the length of the first source data) = 32 bits.

[0075] The instruction processing device of the present invention (for example, the aforementioned instruction processing device 100, 300) performs accumulation operations on the first source data and the second source data through multiplication and accumulation instructions, that is, the first source data, the second source data and the sequence information are read into the first, second and third registers respectively, and then the decoder decodes the multiplication and accumulation instructions, and the execution unit executes the decoded multiplication and accumulation instructions, and determines the execution order of the multiplication operation of the second source data according to the sequence information, so that the second source data corresponds to the first source data, and each first source data is multiplied with the corresponding second source data to obtain multiple multiplication results, and then the multiple multiplication results are added to obtain the multiplication and accumulation result.

[0076] The instruction processing scheme of the present invention implements protection of the second source data through multiplication-accumulation instructions. The second source data is stored in a random order in the memory, and sequence information is set to indicate the execution order of the multiplication operation of the second source data. When a multiplication-accumulation operation is required on the second source data, the execution order of the multiplication operation of the second source data can be determined by the sequence information, that is, the multiplication correspondence between the second source data and the first source data is determined. The second source data is stored in a random order in the memory. Even if others steal the second source data through illegal means, they cannot use the second source data because they cannot know the correct order of the second source data, thereby achieving protection of the second source data.

[0077] Moreover, the instruction processing scheme of the present invention only adjusts the arrangement order of the second source data. The second source data can still be read by the instruction processing device and directly used for calculation through multiplication and accumulation instructions. No additional software and hardware structure is required, nor is there any need for complex and time-consuming encryption and decryption steps, thereby achieving simple and efficient protection of the second source data.

[0078] According to one embodiment, the first source data is the input data of the processing node in the neural network, and the second source data is the weight data of the processing node. The weight data is stored in a memory (such as the aforementioned Figure 1 The weight data is stored in a random order in the memory 120 in the memory, and sequence information is set to indicate the execution order of the multiplication operation of the weight data (i.e., the correct arrangement order). When the forward calculation of the neural network is performed, the correspondence between the weight data and the input data is determined by the sequence information and the multiplication and accumulation operations are performed. The weight data is stored in a random order in the memory. Even if someone steals the weight data by illegal means, they cannot use the weight data because they cannot know the correct arrangement order of the weight data, thereby achieving the protection of the weight data.

[0079] Moreover, the data processing method of the present invention only adjusts the arrangement order of the weight data. The weight data can still be read in and directly used for multiplication and accumulation operations. No additional software and hardware structure is required, nor is there a need for complex and time-consuming encryption and decryption steps. This achieves simple and efficient protection of the weight data and can meet the real-time requirements of the forward calculation of the neural network.

[0080] Figure 6 FIG. 6 is a flowchart of an instruction processing method 600 according to an embodiment of the present invention. Figure 6 The instruction processing method shown is suitable for Figure 1 、 Figure 3 、 Figure 12-14 The described instruction processing device, processor core, computing device and system on chip are executed and are suitable for executing the multiply-accumulate instruction described above.

[0081] like Figure 6 As shown, the method 600 begins at step S610 .

[0082] In step S610 , a multiply-accumulate instruction is received and decoded.

[0083] Refer to the above Figure 3 As described in the related description, the multiply-accumulate instruction indicates the first register A0 as the first operand, specifies the second register A1 as the second operand, and specifies the third register A2 as the third operand.

[0084] The first register A0 and the second register A1 respectively store a plurality of first source data src0 and a plurality of second source data src1 to be used for multiplication operation.

[0085] According to one embodiment, the first source data in the first register A0 is divided into a plurality of data groups. For example, as mentioned above Figure 5 As shown, the first source data a0-a15 are divided into four data groups, each data group includes four first source data.

[0086] The arrangement order of the plurality of second source data in the second register A1 is different from the arrangement order of the plurality of first source data in the first register A0. The first source data in the first register A0 does not necessarily correspond to the second source data at the same position in the second register A1 (for example, the third first source data src0[2] in the first register A0 does not correspond to the third second source data src1[2] in the second register A1, but corresponds to the first second source data src1[0]). For example, the arrangement order of the plurality of first source data in the first register A0 can be correct, and accordingly, the plurality of second source data are stored in a disordered order in the second register.

[0087] According to one embodiment, the first source data may be input data of a processing node in a forward calculation process of a neural network, and the second source data may be weight data of the processing node.

[0088] The third register A2 stores sequence information seq, which is used to indicate the order in which multiplication operations are performed on the second source data, so as to determine the corresponding multiplication relationship between the first source data and the second source data. According to one embodiment, the sequence information includes multiple sequence identifiers corresponding to the multiple first source data, each sequence identifier is suitable for indicating the storage order of the second source data corresponding to the corresponding first source data in the second register. For example, the sequence identifier seq[i-1] is suitable for indicating the storage order of the second source data corresponding to the first source data src0[i-1] in the second register A1.

[0089] According to one embodiment, the multiply-accumulate instruction further indicates a fourth register A3 as a fourth operand. The fourth register A3 is suitable for storing a multiply-accumulate result.

[0090] Optionally, the fourth register A3 stores the accumulated data src3, and the multiply-accumulate instruction further includes a select flag acc. The select flag acc is an immediate value used to indicate whether to perform an addition operation on the accumulated data src3. Accordingly, when decoding the multiply-accumulate instruction in step S610, the value of the select flag acc is also determined.

[0091] Subsequently, in step S620 , a plurality of first source data are obtained from the first register, a plurality of second source data are obtained from the second register, and sequence information indicating the execution order of the multiplication operation of the second source data is obtained from the third register.

[0092] For example, multiple first source data src0[0], src0[1], src0[2], ... are obtained from the first register A0, multiple second source data src1[0], src1[1], src1[2], ... are obtained from the second register A1, and sequence information seq for indicating the execution order of the multiplication operation of the second source data is obtained from the third register A2.

[0093] Then, in step S630 , the plurality of first source data are multiplied by the corresponding second source data indicated by the sequence information to obtain a plurality of multiplication results.

[0094] The sequence information seq includes multiple sequence identifiers, each of which is suitable for indicating the storage order of the second source data corresponding to the corresponding first source data in the second register. For example, the sequence identifier seq[i-1] is suitable for indicating the storage order of the second source data corresponding to the first source data src0[i-1] in the second register A1. Based on the sequence information, the execution order of the multiplication operation of the second source data can be determined, that is, the second source data corresponding to each first source data can be determined. Each first source data is multiplied by the corresponding second source data to obtain multiple multiplication results.

[0095] Then, in step S640 , the multiple multiplication results are added together to obtain a multiplication-accumulation result.

[0096] According to one embodiment, after the multiplication-accumulation result result is obtained, the multiplication-accumulation result is stored in the fourth register A3.

[0097] According to one embodiment, the fourth register A3 stores accumulated data src3. If the select flag acc is not present in the multiply-accumulate instruction or the value of the select flag acc is 0, the accumulated data src3 is not used, and the multiplication-accumulation result result is directly stored in the fourth register A3, overwriting the accumulated data src3. If the select flag acc is present or the value of the select flag acc is 1, the accumulated data src3 is used, the multiplication-accumulation result result is added to the accumulated data src3 to obtain an addition result, and the addition result is stored in the fourth register A3 to replace the accumulated data src3.

[0098] According to one embodiment, referring to the aforementioned Figure 5 , the first source data is divided into multiple data groups, and accordingly, in step S640, the multiplication results in each data group are added together to obtain multiple multiplication and accumulation results (each data group corresponds to one multiplication result).

[0099] According to one embodiment, the length of the multiplication-accumulation result is the sum of the lengths of the plurality of first source data. Figure 4As shown, the first register A0 can be, for example, a 128-bit vector register, which stores four 32-bit first source data, and the multiplication-accumulation result is the sum of the lengths of the four first source data, that is, the length of the multiplication-accumulation result is 128 bits.

[0100] exist Figure 5 In the illustrated embodiment, the first register A0, the second register A1, and the fourth register A3 are, for example, all 128-bit vector registers, and the first source data a0-a15 and the second source data b0-b15 are, for example, all 8-bit unsigned integer data. The first source data a0-a15 are divided into four data groups, each of which includes four first source data. The multiplication-accumulation result of each data group is the sum of the lengths of all the first source data in that data group, that is, the length of the multiplication-accumulation result of each data group is 4*8 (the number of first source data included in the data group * the length of the first source data) = 32 bits.

[0101] Figure 7 FIG. 7 is a flowchart of a data processing method 700 according to an embodiment of the present invention. Figure 6 The data processing method shown is suitable for Figure 1 、 Figure 3 、 Figure 12-14 The described instruction processing device, processor core, computing device and system on chip are executed, which is suitable for performing forward calculation of the neural network according to the above-mentioned instruction processing method 600, and protecting the weight data of the neural network in a simple and efficient manner.

[0102] The neural network includes a plurality of processing layers, each of which includes at least one processing node. A processing node in a processing layer is adapted to perform weighted summation on output data of nodes in a previous layer to generate its own output data.

[0103] For example, Figure 8 A relatively simple neural network structure is shown, in which the output data y4 of node 4 is the weighted sum of the output data x0~x3 of nodes 0~3 and the bias b4 of node 4, that is, y4=x0*w40+x1*w41+x2*w42+x3*w43+b4; the output data y8 of node 8 is the weighted sum of the output data y4~y7 of nodes 4~7 and the bias b8 of node 8, that is, y8=y4*w84+y5*w85+y6*w86+y7*w87+b8.

[0104] At present, the weight data of neural networks are usually stored in the order of the processing nodes. Figure 8Taking the neural network in

[15] as an example, the storage order of the weight data for processing nodes 4 to 9 (a total of 24) is as follows: w40, w41, w42, w43 (weight data for processing node 4), w50, w51, w52, w53 (weight data for processing node 5), w60, w61, w62, w63 (weight data for processing node 6), w70, w71, w72, w73 (weight data for processing node 7), w84, w85, w86, w87 (weight data for processing node 8), and w94, w95, w96, w97 (weight data for processing node 9). According to this storage order, if the network structure file and weight data file of the neural network model are illegally stolen by others, they can derive the neural network model based on these two files, causing the neural network model to be leaked.

[0105] To achieve simple and efficient protection for neural network weight data, in an embodiment of the present invention, the neural network weight data is stored in an order different from the order in which the processing nodes are arranged (i.e., out of order), and sequence information is provided to indicate the order in which multiplication operations on the weight data are performed (i.e., the correct arrangement order). The weight data and sequence information are stored in different files. In this way, even if someone illegally steals the weight data, they cannot use it because they cannot know the correct arrangement order of the weight data, thereby achieving protection for the weight data and the neural network model.

[0106] For example, Figure 8 The weight data of the neural network can be stored in the file in the following random order:

[0107] w40, w42, w43, w41, w50, w52, w53, w51, w60, w62, w63, w61, w70, w72, w73, w71, w84, w86, w87, w85, w94, w96, w97, w95.

[0108] The corresponding sequence information can be, for example, 0, 3, 1, 2, which is used to indicate the execution order of the multiplication operations of the weight data of processing nodes 4 to 9, that is, the correct arrangement order. Taking processing node 4 as an example, according to the sequence information 0, 3, 1, 2, the correct arrangement order of the weight data of node 4 is: the first weight data w40 in the weight file, the fourth weight data w41, the second weight data w42, and the third weight data w43. (Because the current program usually starts counting from 0 instead of 1, the sequence information 0, 3, 1, 2 in the program represents the first, fourth, second, and third data respectively).

[0109] In the above embodiment, the weight data of processing nodes 4 to 9 are stored in the same order and share the same sequence information 0, 3, 1, 2. According to one embodiment, to further enhance the protection of weight data, the weight data of different processing nodes can be stored in different orders, and accordingly, different processing nodes correspond to different sequence information. This can make the storage order of weight data even more disordered, making it more difficult to restore the correct order.

[0110] In one embodiment, the weight data of the processing nodes in the same processing layer are stored in the same order, and the weight order of the processing nodes in different processing layers is stored in different orders; accordingly, the processing nodes in the same processing layer use the same order information, and the processing nodes in different processing layers use different order information. For example, referring to Figure 8 The weight data of nodes 4 to 7 are stored in the order of order1, and the corresponding sequence information is seq1; the weight data of nodes 8 and 9 are stored in the order of order2, and the corresponding sequence information is seq2.

[0111] In another embodiment, the processing nodes in the neural network are divided into multiple groups, and the weights of the processing nodes in each group are stored in the same order and use the same sequence information; the weights of the processing nodes in different groups are stored in different orders and use different sequence information. Figure 8 , processing nodes 4 to 9 can be divided into 3 groups, each group includes 2 processing nodes. The weight data of nodes 4 and 5 are stored in order 1, and the corresponding sequence information is seq1; the weight data of nodes 6 and 7 are stored in order 2, and the corresponding sequence information is seq2; the weight data of nodes 8 and 9 are stored in order 3, and the corresponding sequence information is seq3.

[0112] In another embodiment, the weight data of each node in the neural network can even be stored in a different order, using different order information. Figure 8 ,The weight data of processing nodes 4 to 9 are stored in different orders, and accordingly, different sequence information is also used.

[0113] Those skilled in the art will appreciate that the greater the number of weight data storage orders in a neural network, the greater the corresponding amount of sequence information, and the greater the protection strength of the weight data. Those skilled in the art can set the weight data storage order and the corresponding sequence information based on actual needs. The present invention does not limit the weight data storage order or the amount or granularity of the sequence information.

[0114] like Figure 7 As shown, the method 700 begins at step S710 .

[0115] In step S710 , multiple input data, multiple weight data and sequence information of the processing node are stored in the first register, the second register and the third register respectively, wherein the sequence information is suitable for indicating the execution order of the multiplication operation of the multiple weight data.

[0116] Then, in step S720, the above-mentioned instruction processing method 600 is used to calculate the multiplication and accumulation results of the multiple input data and the multiple weight data to obtain the output data of the processing node.

[0117] Still Figure 8 Taking the neural network structure shown in the figure as an example, its weight data is stored in the following random order:

[0118] w40, w42, w43, w41, w50, w52, w53, w51, w60, w62, w63, w61, w70, w72, w73, w71, w84, w86, w87, w85, w94, w96, w97, w95.

[0119] The corresponding order information of each processing node is 0, 3, 1, 2.

[0120] In step S710, the input data x0-x3, weight data w40, w42, w43, w41 and sequence information of processing node 4 are obtained and stored in the first to third registers A0-A2 respectively. Figure 9 shown.

[0121] Then, in step S720, the instruction processing method 600 is used to calculate the multiplication and accumulation result of the input data x0-x3 and the weight data w40, w42, w43, and w41 through the multiplication and accumulation instruction. The input data x0-x3 is equivalent to the first source data in the method 600, and the weight data w40, w42, w43, and w41 is equivalent to the second source data in the method 600.

[0122] The multiply-accumulate instruction specifies the fourth register A3, and the select flag acc is 1. The bias b4 for processing node 4 is stored in the fourth register A3. After obtaining the multiplication and accumulation result (result) of the input data and the weight data, the multiplication and accumulation result is added to the bias b4 in the fourth register A3 to obtain the addition result y4, which is then stored in the fourth register A3. y4 is the output data of processing node 4.

[0123] In the embodiment described above, the instruction processing device implements protection of the second source data through multiplication and accumulation instructions. In other embodiments, the multiplication and accumulation instructions may not be used, and the protection of the second source data may be implemented by methods such as command lines and program statements. Those skilled in the art will understand that the protection principle of the second source data by other methods such as command lines and program statements is the same as that of the multiplication and accumulation instructions, that is, the second source data is stored in a random order in the memory. When the second source data performs multiplication and accumulation operations, the order of execution of the multiplication operation of the second source data is determined by sequence information to achieve simple and efficient protection of the second source data.

[0124] Figure 10 FIG1 shows a schematic diagram of a data processing method 1000 according to an embodiment of the present invention. The data processing method 1000 is suitable for Figure 1 、 Figure 3 、 Figure 12-14 The described instruction processing device, processor core, computing device and system on chip can be executed to achieve simple and efficient protection of the second source data, and can be implemented in various ways such as multiply-accumulate instructions, command lines, program statements, etc., and is not limited to multiply-accumulate instructions.

[0125] like Figure 10 As shown, method 1000 begins at step S1010 .

[0126] In step S1010 , a plurality of first source data, a plurality of second source data, and sequence information indicating a sequence for executing multiplication operations on the plurality of second source data are acquired.

[0127] Then, in step S1020, the corresponding relationship between the first source data and the second source data is determined according to the sequence information.

[0128] Then, in step S1030 , the plurality of first source data are multiplied by the corresponding second source data to obtain a plurality of multiplication results, and the plurality of multiplication results are added together to obtain a multiplication-accumulation result.

[0129] Specifically, multiple multipliers may be used to implement multiplication operations of multiple first source data with corresponding second source data to obtain multiple multiplication results. Subsequently, an adder is used to add the multiplication results of the multipliers to obtain a multiplication-accumulation result.

[0130] The data processing method 1000 can be applied to the forward computation of a neural network to protect the neural network weight data. Specifically, the present invention also provides a data processing method 1100 suitable for performing forward computation on a neural network, wherein the neural network includes multiple processing nodes, and the weight data of the neural network is stored in an order different from the order in which the processing nodes are arranged.

[0131] Figure 11 FIG. 1 is a schematic diagram showing a data processing method 1100 according to an embodiment of the present invention. Figure 11 As shown, method 1100 begins at step S1110 .

[0132] In step S1110 , a plurality of input data, a plurality of weight data and sequence information of a processing node are acquired, where the sequence information is suitable for indicating the execution order of multiplication operations of the plurality of weight data.

[0133] Then, in step S1120 , the correspondence between the input data and the weight data is determined according to the sequence information.

[0134] Then, in step S1130 , a weighted sum is performed on the plurality of input data according to the weight data corresponding to each input data to determine the output data of the processing node.

[0135] Specifically, multiple multipliers can be used to multiply the multiple input data with corresponding weight data to obtain multiple multiplication results; adders can be used to add the multiple multiplication results to obtain multiplication-accumulation results, and the output data of the processing node can be determined based on the multiplication-accumulation results.

[0136] As described above, the instruction processing device according to the present invention can be implemented as a processor core, and the instruction processing method can be executed in the processor core. The processor core can be implemented in different processors in different ways. For example, the processor core can be implemented as a general-purpose in-order core for general-purpose computing, a high-performance general-purpose out-of-order core for general-purpose computing, and a dedicated core for graphics and / or scientific (throughput) computing. The processor can be implemented as a CPU (central processing unit) and / or a coprocessor, wherein the CPU may include one or more general-purpose in-order cores and / or one or more general-purpose out-of-order cores, and the coprocessor may include one or more dedicated cores. Such a combination of different processors can lead to different computer system architectures. In one computer system architecture, the coprocessor is on a chip separate from the CPU. In another computer system architecture, the coprocessor is in the same package as the CPU but on a separate die. In another computer system architecture, the coprocessor is on the same die as the CPU (in this case, such a coprocessor is sometimes referred to as dedicated logic such as integrated graphics and / or scientific (throughput) logic, or as a dedicated core). In a computer system architecture also known as a system on a chip, the described CPU (sometimes called an application core or application processor), the coprocessor described above, and additional functionality may be included on the same die.

[0137] Figure 12 FIG. 1 shows a schematic diagram of a processor 1200 according to an embodiment of the present invention. Figure 12As shown in the solid line box in FIG, according to one embodiment, the processor 1200 includes a single core 1210-1, a system agent unit 1220, and a bus controller unit 1230. Figure 12 As shown in the dotted box in , according to another embodiment of the present invention, the processor 1200 may further include multiple cores 1210 -N, an integrated memory controller unit 1222 in the system agent unit 1220 , and dedicated logic 1240 .

[0138] According to one embodiment, the processor 1200 can be implemented as a central processing unit (CPU), wherein the dedicated logic 1240 is integrated graphics and / or scientific (throughput) logic (which may include one or more cores), and the cores 1210-1 to 1210-N are one or more general-purpose cores (e.g., general-purpose in-order cores, general-purpose out-of-order cores, or a combination of the two). According to another embodiment, the processor 1200 can be implemented as a coprocessor, wherein the cores 1210-1 to 1210-N are multiple dedicated cores for graphics and / or science (throughput). According to yet another embodiment, the processor 1200 can be implemented as a coprocessor, wherein the cores 1210-1 to 1210-N are multiple general-purpose in-order cores. Therefore, the processor 1200 can be a general-purpose processor, a coprocessor, or a dedicated processor, such as, for example, a network or communication processor, a compression engine, a graphics processor, a GPGPU (general-purpose graphics processing unit), a high-throughput integrated many-core (MIC) coprocessor (including 30 or more cores), or an embedded processor. The processor can be implemented on one or more chips. Processor 1200 may be part of and / or implemented on one or more substrates using any of a number of processing technologies such as, for example, BiCMOS, CMOS, or NMOS.

[0139] The memory hierarchy includes one or more levels of cache within each core, one or more shared cache units 1250, and external memory (not shown) coupled to the integrated memory controller unit 1222. The shared cache unit 1250 may include one or more intermediate level caches, such as a level 2 (L2), level 3 (L3), level 4 (L4), or other level of cache, a last level cache (LLC), and / or a combination thereof. Although in one embodiment, a ring-based interconnect unit 1252 interconnects the integrated graphics logic 1240, the shared cache unit 1250, and the system agent unit 1220 / integrated memory controller unit 1222, the present invention is not limited thereto, and any number of well-known techniques may be used to interconnect these units.

[0140] System agent unit 1220 includes components that coordinate and operate cores 1210-1 to 1210-N. System agent unit 1220 may include, for example, a power control unit (PCU) and a display unit. The PCU may include logic and components required to adjust the power state of cores 1210-1 to 1210-N and integrated graphics logic 1240. The display unit is used to drive one or more externally connected displays.

[0141] The cores 1210-1 to 1210-N may have the above reference Figure 1 The core architectures described herein may be homogeneous or heterogeneous in terms of the architecture instruction set. That is, two or more of the cores 1210 - 1 to 1210 -N may be capable of executing the same instruction set, while other cores may be capable of executing only a subset of the instruction set or a different instruction set.

[0142] Figure 13 A schematic diagram of a computing device 1300 according to one embodiment of the present invention is shown. Figure 13 The computer device 1300 shown can be a laptop, desktop, handheld PC, personal digital assistant, engineering workstation, server, network device, network hub, switch, embedded processor, digital signal processor (DSP), graphics device, video game device, set-top box, microcontroller, cellular phone, portable media player, handheld device, and various other electronic devices. The present invention is not limited to this, and all systems that can incorporate the processor and / or other execution logic disclosed in this specification are within the scope of protection of the present invention.

[0143] like Figure 13 As shown, system 1300 may include one or more processors 1310, 1315. These processors are coupled to a controller hub 1320. In one embodiment, controller hub 1320 includes a graphics memory controller hub (GMCH) 1390 and an input / output hub (IOH) 1350 (which may be on separate chips). GMCH 1390 includes a memory controller and a graphics controller coupled to memory 1340 and coprocessor 1345. IOH 1350 couples input / output (I / O) devices 1360 to GMCH 1390. Alternatively, the memory controller and graphics controller are integrated into the processor, such that memory 1340 and coprocessor 1345 are directly coupled to processor 1310, in which case controller hub 1320 includes only IOH 1350.

[0144] The optional nature of the additional processor 1315 is indicated by dashed lines in Figure 13 Each processor 1310 , 1315 may include one or more of the processing cores described herein, and may be a version of the processor 1200 described above.

[0145] The memory 1340 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of the two. For at least one embodiment, the controller hub 1320 communicates with the processors 1310, 1315 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a Quick Path Interconnect (QPI), or similar connection 1395.

[0146] In one embodiment, the coprocessor 1345 is a special purpose processor such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, or an embedded processor, etc. In one embodiment, the controller hub 1320 may include an integrated graphics accelerator.

[0147] In one embodiment, processor 1310 executes instructions that control general types of data processing operations. Embedded within these instructions may be coprocessor instructions. Processor 1310 recognizes these coprocessor instructions as being of a type that should be executed by attached coprocessor 1345. Accordingly, processor 1310 issues these coprocessor instructions (or control signals representing coprocessor instructions) to coprocessor 1345 over a coprocessor bus or other interconnect. Coprocessor 1345 accepts and executes the received coprocessor instructions.

[0148] Figure 14 FIG. 1 is a schematic diagram of a system on chip (SoC) 1400 according to an embodiment of the present invention. Figure 14 The system-on-chip shown contains Figure 12 The processor 1200 shown is therefore Figure 12 Similar parts have the same reference numerals. Figure 14 As shown, the interconnect unit 1420 is coupled to the application processor 1410, the system agent unit 1220, the bus controller unit 1230, the integrated memory controller unit 1222, one or more coprocessors 530, a static random access memory (SRAM) unit 1440, a direct memory access (DMA) unit 1450, and a display unit 1460 for coupling to one or more external displays. The application processor 1410 includes a set of one or more cores 1210-1 to 1210-N and a shared cache unit 1250. The coprocessor 1430 includes integrated graphics logic, an image processor, an audio processor, and a video processor. In one embodiment, the coprocessor 1430 includes a special-purpose processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0149] In addition, the system on chip described above can be included in a smart device to implement corresponding functions in the smart device, including but not limited to executing relevant control programs, performing data analysis, calculation and processing, network communication, and controlling peripherals in the smart device.

[0150] Such intelligent devices include specialized intelligent devices, such as mobile terminals and personal digital assistants, etc. These devices include one or more systems on a chip according to the present invention to process data or control peripherals in the devices.

[0151] Such smart devices also include specialized devices designed to implement specific functions, such as smart speakers and smart displays. These devices include a system-on-chip (SoC) according to the present invention to control the speakers and displays, thereby giving them additional communication, perception, and data processing capabilities.

[0152] Such smart devices also include various IoT and AIoT devices. These devices include a system-on-chip according to the present invention to perform data processing, such as AI computing, data communication and transmission, thereby achieving a denser and more intelligent device distribution.

[0153] Such smart devices can also be used in vehicles, for example, they can be implemented as vehicle-mounted devices, or can be embedded in vehicles, thereby providing data processing capabilities for the vehicle's intelligent driving.

[0154] Such smart devices can also be used in the home and entertainment fields, for example, they can be implemented as smart speakers, smart air conditioners, smart refrigerators, smart display devices, etc. These devices include the system-on-chip according to the present invention to perform data processing and peripheral control, thereby realizing the intelligence of home and entertainment devices.

[0155] In addition, such intelligent devices can also be used in the industrial field, for example, they can be implemented as industrial control equipment, sensing equipment, IoT equipment, AIoT equipment, and braking equipment. These devices include the system-on-chip according to the present invention to perform data processing and peripheral control, thereby realizing the intelligence of industrial equipment.

[0156] The above description of the smart device is merely illustrative, and the smart device according to the present invention is not limited thereto. All smart devices that can utilize the system on chip according to the present invention to perform data processing are within the protection scope of the present invention.

[0157] The various embodiments of the mechanisms disclosed herein can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of the present invention can be implemented as a computer program or program code executed on a programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0158] It should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0159] Those skilled in the art will appreciate that the modules, units, or components of the devices in the examples disclosed herein may be arranged in the device described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or further divided into multiple submodules.

[0160] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0161] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0162] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions described. Thus, a processor having the necessary instructions for implementing the method or method element forms a device for implementing the method or method element. Furthermore, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0163] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved and are not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0164] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. An instruction processing device, comprising: A first register, adapted to store a plurality of first source data; a second register adapted to store a plurality of second source data; a third register adapted to store sequence information, wherein the sequence information is adapted to indicate an execution order of the multiplication operation of the second source data; A decoder is adapted to receive and decode a multiply-accumulate instruction, wherein the multiply-accumulate instruction indicates: The data stored in the first register is used as the first operand, The data stored in the second register is used as the second operand, The data stored in the third register is used as a third operand; An execution unit is coupled to the first, second, and third registers and the decoder, respectively, and is suitable for executing the decoded multiplication-accumulation instruction to obtain the multiple first source data from the first register, obtain the multiple second source data from the second register, obtain the sequence information from the third register, multiply the multiple first source data with the corresponding second source data indicated by the sequence information to obtain multiple multiplication results, and add the multiple multiplication results to obtain a multiplication-accumulation result.

2. The instruction processing apparatus according to claim 1, wherein: The sequence information includes a plurality of sequence identifiers corresponding to the plurality of first source data respectively, and the sequence identifiers are suitable for indicating a storage order of the second source data corresponding to the corresponding first source data in the second register.

3. The instruction processing apparatus according to claim 1 , further comprising a fourth register adapted to store the multiplication-accumulation result; The multiply-accumulate instruction further indicates the data stored in the fourth register as a fourth operand; The execution unit is further adapted to store the multiplication-accumulation result into the fourth register.

4. The instruction processing apparatus according to claim 3, wherein: The fourth register stores accumulated data; The execution unit is further adapted to add the multiplication-accumulation result to the accumulated data to obtain an addition result, and store the addition result in the fourth register to replace the accumulated data.

5. The instruction processing apparatus according to claim 1, wherein: The length of the multiplication-accumulation result is the sum of the lengths of the multiple first source data.

6. The instruction processing apparatus according to claim 1, wherein: The plurality of first source data are divided into a plurality of data groups, The execution unit is adapted to add the multiplication results in each data group to obtain a plurality of multiplication-accumulation results.

7. The instruction processing apparatus according to claim 1, wherein: The first source data is input data of a processing node in a neural network, and the second source data is weight data of the processing node.

8. A method for processing an instruction, comprising: receiving a multiply-accumulate instruction and decoding the multiply-accumulate instruction, wherein the multiply-accumulate instruction indicates that data stored in the first register is a first operand, data stored in the second register is a second operand, and data stored in the third register is a third operand; Execute the decoded multiplication-accumulation instruction to obtain multiple first source data from the first register, obtain multiple second source data from the second register, obtain sequence information indicating the execution order of the multiplication operation of the second source data from the third register, multiply the multiple first source data with the corresponding second source data indicated by the sequence information to obtain multiple multiplication results, and add the multiple multiplication results to obtain a multiplication-accumulation result.

9. The instruction processing method according to claim 8, wherein: The sequence information includes a plurality of sequence identifiers corresponding to the plurality of first source data respectively, and the sequence identifiers are suitable for indicating a storage order of the second source data corresponding to the corresponding first source data in the second register.

10. The instruction processing method according to claim 8, wherein: The multiply-accumulate instruction further indicates data stored in a fourth register as a fourth operand, and the method further includes: The multiplication and accumulation result is stored in the fourth register.

11. The instruction processing method according to claim 10, wherein: The fourth register stores accumulated data, and the method further includes: The multiplication-accumulation result is added to the accumulated data to obtain an addition result, and the addition result is stored in the fourth register to replace the accumulated data.

12. The instruction processing method according to claim 8, wherein: The length of the multiplication-accumulation result is the sum of the lengths of the multiple first source data.

13. The instruction processing method according to claim 8, wherein: The plurality of first source data are divided into a plurality of data groups, and the step of adding the plurality of multiplication results to obtain a multiplication-accumulation result includes: The multiplication results within each data group are added together to obtain multiple multiplication-accumulation results.

14. The instruction processing method according to claim 8, wherein: The first source data is input data of a processing node in a neural network, and the second source data is weight data of the processing node.

15. A data processing method suitable for performing forward computation on a neural network, wherein the neural network comprises a plurality of processing nodes, wherein weight data of the neural network is stored in an order different from the order in which the processing nodes are arranged, the method comprising: storing a plurality of input data, a plurality of weight data, and sequence information of the processing node into a first register, a second register, and a third register, respectively, wherein the sequence information is adapted to indicate an execution order of multiplication operations of the plurality of weight data; The instruction processing method according to any one of claims 8 to 14 is used to calculate the multiplication and accumulation results of the multiple input data and the multiple weight data to obtain the output data of the processing node.

16. The data processing method according to claim 15, wherein: The neural network includes a plurality of processing layers, each processing layer includes at least one processing node; The weight data of processing nodes located in the same processing layer are stored in the same order, and the weight data of processing nodes located in different processing layers are stored in different orders; Accordingly, processing nodes located in the same processing layer use the same sequence information, and processing nodes located in different processing layers use different sequence information.

17. A data processing method comprising: Acquire a plurality of first source data, a plurality of second source data, and sequence information for indicating an execution order of a multiplication operation of the plurality of second source data; determining a correspondence between the first source data and the second source data according to the sequence information; The plurality of first source data are multiplied by the corresponding second source data respectively to obtain a plurality of multiplication results, and the plurality of multiplication results are added together to obtain a multiplication-accumulation result.

18. A data processing method suitable for performing forward computation on a neural network, wherein the neural network comprises a plurality of processing nodes, wherein weight data of the neural network is stored in an order different from the order in which the processing nodes are arranged, the method comprising: Acquire a plurality of input data, a plurality of weight data, and sequence information of a processing node, wherein the sequence information is adapted to indicate an execution order of multiplication operations of the plurality of weight data; Determine the corresponding relationship between the input data and the weight data according to the sequence information; The plurality of input data are weighted and summed according to the weight data corresponding to each input data to determine the output data of the processing node.

19. A computing device comprising: at least one processor and memory storing program instructions; When the program instructions are read and executed by the processor, the computing device is caused to perform the method according to any one of claims 8 to 18.

20. A readable storage medium storing program instructions, wherein when the program instructions are read and executed by a computing device, the computing device is caused to execute the method according to any one of claims 8 to 18.

21. A system on chip, comprising the instruction processing device according to any one of claims 1 to 7.

22. An intelligent device comprising the system on chip according to claim 21.

Images