Data processing method and device, equipment and storage medium

[0046] In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

[0048] As mentioned above, in the related art, the feature vector processing module is currently mainly implemented by DSP or a customized unit designed for vector calculation in neural networks. In the design of DSP, there are still many idle computing resources when it is used for vector computing, which leads to waste of computing resources. And its design structure is also more complex, resulting in poor vector calculation efficiency of DSP. The design of customized units is often limited by its application scenarios, lacks flexibility, and easily wastes computing resources.

[0049] In order to solve at least one technical problem existing in the related art, the present disclosure provides a data processing method, apparatus, device and storage medium.

[0050] The core idea of ​​the above technical solution is to abstract various types of vector computing subtasks in the neural network, and implement different types of vector computing subtasks through a plurality of vector computing units. On this basis, based on the input instruction, the target computing unit in the multiple vector computing units obtains the computing task information to be executed, and based on the computing task information, the target computing unit is set with personalized network parameters, so as to obtain through the target computing unit According to the processing results of the feature data, different types of vector computing subtasks in the neural network are implemented. It can be seen that in this scheme, various types of vector computations in the neural network are abstracted into vector computation units suitable for various vector computation subtasks in the neural network, which greatly improves the versatility of the vector computation unit and also improves the vector computation. The flexibility of the unit in processing various vector computing subtasks effectively reduces the waste of computing resources. In addition, through the multiplexing of the vector computing unit, the applicable scenarios of the vector computing unit can be expanded, and the versatility of the vector computing unit can be improved.

[0051] In the embodiments of the present disclosure, first, a feature vector processing module is proposed, which is used for processing vector-type feature data related to computing tasks. The feature data related to the computing task may be image data involved in the field of image processing, corpus data involved in the field of audio processing, text data involved in the field of search, or data from other technical fields. The technical solution provided by the present disclosure is mainly used for processing feature data formed in the form of vectors, and of course, it can also be used for processing feature vectors obtained after processing other feature data, which is not limited by the present disclosure.

[0052] In practical applications, the feature vector processing module is set in a computing device for processing neural networks. Computing devices include but are not limited to: terminal devices, servers, and other various types of electronic devices. The implementation form of the feature vector processing module can be a chip, a circuit board, and other various types of devices or devices, and can also be implemented by program codes mounted in the above-mentioned devices or devices.

[0053] figure 1 A schematic diagram of a feature vector processing module provided by an embodiment of the present disclosure, such as figure 1 As shown, the feature vector processing module includes a plurality of vector calculation units for vector calculation processing, such as figure 1 The vector calculation unit 1, the vector calculation unit 2, . . . , and the vector calculation unit N shown in .

[0054] It should be noted that: first, the vector computing unit 1 and the vector computing unit 2 may be the same type of vector computing units, or may be different types of vector computing units, and this is only an example. Here, the same type of vector unit refers to a unit used to implement the same type of vector calculation subtask. For example, vector calculation unit 1 and vector calculation unit 2 are both vector multiplication (Vector mul) units; different types of vector calculation units refer to Units for implementing different types of vector calculation subtasks, for example, vector calculation unit 1 is a vector multiplication (Vector mul) unit, and vector calculation unit 2 is a vector point-by-point calculation (Element wise) unit. Second, in different computing tasks, the types, calling sequences and numbers of vector computing units used are different. For distinction, the vector computing unit used in the computing task is referred to as a target computing unit in this disclosure.

[0055] In the embodiments provided by the present disclosure, each vector calculation unit is used to implement corresponding vector calculation. In practical applications, multiple vector computing units include, but are not limited to: any one or more of the Vector mul unit, the Element wise unit, the Vector add unit, the Normalize unit, and the Nonlinear unit. indivual. The combination of various vector computing units can meet the rich vector computing needs in different application scenarios, improve the utilization of vector computing resources, and avoid wasting computing resources.

[0056] In practical applications, there are often many services that need to use the vector computing capabilities provided by the feature vector processing module, such as video encoding and decoding services, search services, speech recognition services, and so on. Each service will continuously generate many computing tasks. For example, in the search service, search tasks will be generated in response to search operations triggered by many users. For example, in the speech recognition service, speech recognition tasks will be triggered in response to the continuous input of the user's voice. In order to process a large number of computing tasks triggered by each service, for each service, multiple vector computing units can be triggered simultaneously in the feature vector processing module to process many vector computing tasks corresponding to the service in parallel. computing resources for implementing these vector computing tasks, optionally, such as figure 1 As shown, the present disclosure also provides an instruction decoding unit and a memory access unit. Wherein, the instruction decoding unit is used to obtain the corresponding computing resource allocation information (for example, the target computing unit to be used) from the first instruction for the feature vector processing module, and use the corresponding target computing unit to complete the corresponding computing task. The storage access unit is used to obtain the characteristic data to be calculated and the predetermined network parameters in different types of neural networks from the storage space.

[0057] Specifically, figure 2 A schematic structural diagram of a feature vector processing module provided by an embodiment of the present disclosure, such as figure 2 As shown, the feature vector processing module includes an instruction decoding unit 101 , a storage access unit 102 , a plurality of vector calculation units 103 to 107 for vector calculation processing, and a write-back unit 108 . The instruction decoding unit 101 is connected to one or more of the vector calculation units 103 to 107 , and the memory access unit 102 is connected to one or more of the vector calculation units 103 to 107 .

[0058] In an alternative embodiment, as figure 2 As shown, the instruction decoding unit 101 includes an instruction decoding module (Instruction Decoder) and a distribution module (Dispatch). Wherein, the instruction decoding module can decode the first instruction according to the preset instruction decoding rules, so as to obtain the information for calling the target computing unit (that is, one or more of the vector computing units 103 to 107 ) and the storage access Scheduling instructions for unit 102 . For distinction, in the present disclosure, the scheduling instruction for calling the storage access unit is referred to as the first calling instruction, and the second calling instruction for calling the target computing unit.

[0059] The storage access unit 102 includes a feature data acquisition module (a three-dimensional stride loading module, ie, a 3D Stride LoadInput module) and a network parameter acquisition module (a parameter loading module, ie, a Load Param module). Specifically, assuming that the first call instruction is denoted as instruction 1 (or may also be denoted as ld_op), the storage access unit 102 obtains, from the corresponding storage space, the characteristic data related to the computing task and the network parameters, and output the acquired feature data and network parameters to the target computing unit. The characteristic data includes characteristic data of the neural network or intermediate results obtained by other processing units in the neural network. The network parameters refer to the parameters obtained by training the neural network for configuring the vector computing unit. For a neural network that implements the same computing task, the network parameters can be fixed values ​​or dynamic values ​​that are dynamically adjusted according to feedback obtained during the execution of the computing task.

[0060] The target computing units in the vector computing units 103 to 107, according to the second call instruction issued by the instruction decoding module and the characteristic data from the storage access unit 102 or the vector computing unit of the previous stage, execute the corresponding vector computing subtask, and output The calculation result is used as the processing result of the calculation task information. Among them, the vector calculation units 103 to 107 can be implemented by their corresponding operation modules, and the specific structures are as follows image 3 shown.

[0061] In practical applications, the second call instruction may be one or more, for example figure 2 The second invocation instruction includes instructions 2 to 7 for invoking the vector computing units 103 to 107 . According to the vector calculation subtasks implemented by the vector calculation units 103 to 107 , for example, instructions 2 to 7 may be respectively implemented as vector multiplication instructions (mul_op) for calling the vector multiplication unit (Vector mul), for invoking vector point-by-point calculation The vector point-by-point calculation instruction (ele_op) of the unit (Element wise), the vector addition instruction (add_op) for calling the vector addition unit (Vector add), the normalization calculation instruction for calling the normalization unit (Normalize) ( bn_op), a nonlinear activation calculation instruction (act_op) for calling the linear calculation unit (Nonlinear), and a storage instruction (st_op) for calling the 3D Stride Store module.

[0062] In practical applications, for specific application scenarios, such as video encoding and decoding scenarios, search service scenarios, speech recognition service scenarios, etc., the calling sequence of the target computing units to be called in the vector computing units 103-107 can be adjusted. For example, the vector calculation unit 103 is called multiple times through the first instruction, so that the corresponding vector calculation in the neural network is implemented by multiplexing the vector calculation unit 103 . For example, the vector computing unit 103 , the vector computing unit 105 , the vector computing unit 106 , and the vector computing unit 107 are called through the first instruction, so that the corresponding vector computing subtasks in the neural network are implemented by selectively calling the vector computing units.

[0063] from image 3As can be seen from the shown structure, the vector calculation method in the vector calculation unit 103 can be implemented as vector multiplication, and the vector calculation unit 103 can implement the corresponding vector calculation function according to instruction 2 (eg, vector multiplication instruction, denoted as mul_op). The specific calculation process can be implemented as: if mul_op=BYPASS, then y(t)=x(t); if mul_op=MUL, then y(t)=a(t)*x(t). Among them, t refers to the time, x(t) refers to the feature data, a(t) refers to the input network parameters (such as coef), and y(t) refers to the output calculation result.

[0064] from image 3 It can be seen that the vector calculation method in the vector calculation unit 104 can be implemented as vector point-by-point calculation, and the vector calculation unit 104 can implement the corresponding vector calculation function according to instruction 3 (eg, vector point-by-point calculation instruction, denoted as ele_op). The vector calculation unit 104 includes vector calculation methods 1 to 3, and the specific calculation process can be implemented as follows: if ele_op=BYPASS, then y(t)=x(t); if ele_op=ADD, then y(t)=x (t)+x(t-1); if ele_op=MUL, then y(t)=x(t)*x(t-1); if ele_op=MAX, then y(t)=max(x(t ), x(t-1)). Among them, t refers to the time, x(t) refers to the feature data, and y(t) refers to the output calculation result.

[0065] exist image 3 , the preprocessing can be delayed processing. For example, the specific implementation manner of preprocessing may be to delay one clock cycle, that is, D-1. Correspondingly, the feature data is x(t-1). In conjunction with the function of loading different tensors in the parity cycle in the storage access module 102, the vector calculation unit module 104 can complete the ADD/MUL/MAX function between two input tensors. At this time, a valid execution result is output every two clock cycles.

[0066] from image 3 It can be known that the vector calculation method in the vector calculation unit 105 can be implemented as vector addition, and the vector calculation unit 105 can implement the corresponding vector calculation function according to instruction 4 (eg, vector addition instruction, denoted as add_op). The specific calculation process can be implemented as: if mul_op=BYPASS, then y(t)=x(t); if mul_op=ADD, then y(t)=x(t)+b(t). Among them, t refers to the time, x(t) refers to the feature data, b(t) refers to the input network parameters (such as coef), and y(t) refers to the output calculation result.

[0067] from image 3 It can be seen that the vector calculation method in the vector calculation unit 106 can be implemented as normalized calculation, and the vector calculation unit 106 can implement the corresponding vector calculation function according to instruction 5 (eg, normalized calculation instruction, denoted as bn_op). The specific calculation process can be implemented as: if mul_op=BYPASS, then y(t)=x(t); if mul_op=MADD, then y(t)=a(t)*x(t)+b(t). Among them, t refers to the time, x(t) refers to the feature data, a(t) and b(t) refer to the input network parameters (such as coef), and y(t) refers to the output calculation result. The vector calculation unit 106 is generally designed for normalization calculations in neural networks.

[0068] from image 3 It can be seen that the vector calculation method in the vector calculation unit 107 can be implemented as a nonlinear activation calculation in the neural network, and the vector calculation unit 107 can realize the corresponding vector calculation function according to the instruction 6 (such as the nonlinear activation calculation instruction (act_op). The vector calculation unit 107 includes vector calculation methods 4 to 5, and the specific calculation process can be implemented as: if act_op=BYPASS, y(t)=x(t); if act_op=RELU, then y(t)=MAX(0 , x(t)); if act_op=ADD, then y(t)=Lookup(x(t)). Wherein, t refers to the time, x(t) refers to the feature data, and y(t) refers to the output Calculation results. In image 3 Among them, LookUp refers to the lookup table, which uses x(t) as the characteristic data to complete the function of nonlinear mapping.

[0069] In this embodiment, a write-back unit 108 (for example, implemented as a three-dimensional stride storage module, namely 3DStride Store) is also provided, such as figure 2 As shown, the module is configured to write the processing result back to the sending end of the first instruction, such as the memory, according to the instruction issued by the instruction decoding module.

[0070] Among them, the 3D Stride Load Input module and the 3D Stride Store module mentioned above both adopt an addressing mode with a stride. Generally speaking, loading refers to reading a piece of continuous data (such as the network parameters and/or characteristic data described above) with a set length (such as L) from the base address. Optionally, the Load Param module in the storage access module 102 can be used to implement this function. In order to adapt to the multi-dimensional tensors (Tensor) existing in the neural network, the 3D Stride Load Input/Store module can adopt the addressing access mode with stride, which can be described in pseudocode as follows:

[0071]

[0072]

[0073] Among them, BADDR is the base address for continuous data reading, D1_STRIDE, D2_STRIDE, D3_STRIDE refer to the stride of different dimensions, D1_NUM, D2_NUM, D3_NUM refer to the addressing times of different dimensions. The flexible addressing of 3D Tensor can be accomplished by using the 3D StrideLoad Input/Store module, such as Figure 4 shown.

[0074] Assume that the 3D Tensor is stored in the order of storing row data first, then storing columns, and finally storing channels. For example a memory channel can be implemented as Figure 4 Memory channel C shown. Load if required Figure 4 Assuming D1_NUM=2, D2_NUM=2, D3_NUM=3, then, according to D1_STRIDE=Height*Width, D2_STRIDE=2*Height, D3_STRIDE=2 can be calculated. exist Figure 4 , the height is H and the width is W. Of course, in practical applications, this addressing method can also be extended from three dimensions to multiple dimensions to adapt to different application scenarios.

[0075] The memory access module 102 has two sets of modules for performing 3D Stride address generation, thus, addressing in parity clock cycles can be performed. Through the above two different tensor access functions, in cooperation with the element wise module 104, the function of vector point-by-point calculation (ie element wise add/mul) can be completed.

[0076] Through the feature vector processing module introduced above, the memory access unit and various vector computing units (ie, target computing units) can be controlled by the instruction decoding unit, so as to more targetedly realize various vector computing sub-units in the neural network. tasks, improve the efficiency of vector computing, improve the utilization of vector computing resources, and avoid wasting computing resources.

[0077] The detailed vector processing process of the neural network-based feature data processing scheme will be specifically described below with reference to the following embodiments.

[0078] Based on the foregoing, the neural network-based feature data processing solution provided by the present disclosure can be applied to the foregoing feature vector processing module, which is provided in a computing device for processing neural networks. The specific introduction can refer to the above, and will not be repeated here.

[0079] Figure 5 A flowchart of a data processing method provided by an embodiment of the present disclosure, such as Figure 5 As shown, the method may include the following steps:

[0080] 501. In response to the first instruction directed to the feature vector processing module, acquire information of the computing task of the target computing unit under the first instruction. Wherein, the target calculation unit includes one or more of the multiple vector calculation units included in the feature vector processing module, and the multiple vector calculation units are respectively used to implement respective corresponding vector calculation subtasks.

[0081] 505. Acquire feature data related to the computing task and network parameters for configuring the target computing unit according to the computing task.

[0082] 503. The target computing unit obtained through the network parameter configuration performs vector computing processing on the feature data, to obtain a processing result of computing task information.

[0083] The data processing method provided in this embodiment may be executed by one or more vector computing units (ie, target computing units) in the multiple feature vector processing modules described above. For example, by figure 1 The eigenvector processing modules 1-3 illustrated in the scheme are implemented. Again, by figure 2 The instruction decoding unit 101, the memory access unit 102, the vector calculation units 103-107, and the write-back unit 108 shown in the figure are executed cooperatively.

[0084] The vector calculation process of the feature vector processing module is carried out for each vector calculation unit, that is, the calculation resources of each vector calculation unit are dynamically allocated according to the type of calculation that needs to be performed. It can be performed between vector computing units, or it can be performed between multiple vector computing units of different types. Therefore, by allocating vector computing resources on demand, the problem of wasting computing resources in the feature vector processing module is avoided, and the vector computing efficiency is improved.

[0085] In this embodiment, in the process of using the feature vector processing module to perform vector calculation in the neural network, it is assumed that vector calculation processing is currently performed on the target calculation unit, and the target calculation unit is one or more of multiple vector calculation units, When performing vector computing processing on the target computing unit, it is necessary to obtain and use the target computing unit to realize the corresponding computing task.

[0086] In the embodiment of the present disclosure, it is assumed that the vector calculation is performed in a periodic manner, that is, the vector calculation unit may periodically execute the vector calculation process according to a set scheduling period. The scheduling period can be in microseconds, seconds, or minutes, such as 1 microsecond, 1 second, or 1 minute. For example, assuming that the feature vector processing module can complete the calculation of one element within one clock cycle, then, for a vector with a total number of elements T, the feature vector processing module needs T clock cycles to complete the calculation.

[0087] Of course, in practical applications, in addition to the above periodic methods, in order to speed up the processing speed of computing tasks, optionally, multiple vector computing units can be called in parallel through instructions, or a more efficient vector computing structure can be designed and used structure to call the corresponding vector computing unit to improve the execution efficiency of the computing task and shorten the processing time of the computing task.

[0088] The following is combined with specific examples to introduce Figure 5 The specific execution steps of the data processing method shown.

[0089] First, in step 501, the computing task information of the target computing unit under the first instruction is obtained, which can be implemented as:

[0090] Obtain the target calculation type contained in the first instruction through the instruction decoding unit; determine the target calculation unit that matches the target calculation type in the plurality of vector calculation units; generate a task for indicating calculation task information based on the target calculation type and the target calculation unit Execute the instruction.

[0091]In the above steps, the target computing type intended to be called in the first instruction and the matching target computing unit can be split by the instruction decoding unit, so that the calling operation to the target computing unit is triggered by the task execution instruction, which is a combination of vector computing units and Calls provide the basis to improve the efficiency of vector computations.

[0092] Assuming that the computing task requester sends an instruction stream (Instruction Stream) to the feature vector processing module, the instruction decoding unit in the feature vector processing module can use the pre-acquired decoding rules to decode each instruction in the instruction stream. code, for example, assuming that the instruction stream contains the first instruction, the first instruction can be decoded. In the decoding rule, the type identifiers corresponding to various vector calculations can be pre-specified. For example, the calculation type identifier contained in the first instruction can be read from a prescribed field, so that the first instruction can be determined based on the calculation type identifier. The type of target computation implemented.

[0093] Furthermore, from the plurality of vector computing units included in the feature vector processing module, a target computing unit matching the computing type identifier is selected. E.g, figure 2 In the feature vector processing module shown, corresponding vector calculation units 103-107 are selected according to the calculation type identifier.

[0094] Finally, after determining the target computing type and the target computing unit, in step 501, a task execution instruction for indicating computing task information is generated based on the target computing type and the target computing unit, which can be implemented as:

[0095] Obtain the storage space address contained in the first instruction through the instruction decoding unit, where the storage space address is used to indicate the storage space containing characteristic data and/or network parameters; based on the storage space address and the target calculation type, the storage space address is generated and used in the calculation task information. to call the first call instruction of the storage access unit, and send the first call instruction to the storage access unit; generate a second call instruction for calling the target computing unit based on the target computing type, and send the second call instruction to the target computing unit.

[0096] In the above steps, the first calling instruction for calling the storage access unit and the second calling instruction for calling the target computing unit are generated by the instruction decoding unit, thereby providing a basis for the combination and calling of the vector computing unit, and improving the efficiency of vector computing. .

[0097] In the above steps, it is assumed that the storage space is memory, and the feature vector processing module is assumed to be figure 2 The eigenvector processing module shown. Then, the memory address contained in the first instruction is acquired by the instruction decoding unit, and based on the memory address and the target calculation type, the ld_op (ie, the first call instruction) used to call the storage access unit 102 in the current calculation task information is generated. , and send the ld_op to the storage access unit 102, so that the storage access unit 102 accesses the corresponding storage space based on the memory address, and then obtains characteristic data and network parameters related to the computing task from the storage space. At the same time, instructions 2 to 7 (ie, second call instructions) corresponding to each of the target computing units 103 to 107 can also be generated based on the target computing type, such as mul_op, ele_op, add_op, bn_op, and act_op, respectively, and these Instructions are sent to target computing units 103 to 107 .

[0098] In addition, in order to return the processing result obtained by the target computing unit to the computing task requester (that is, the sender that issued the first instruction), an instruction 7 (st_op) can also be generated and sent to the write-back unit 108 for controlling the write-back The unit 108 returns the final processing result of the current computing task to the memory. For example, the processing result is written back to memory according to the vector address contained in st_op to complete a service request.

[0099] Further, in step 502, the characteristic data related to the computing task and the network parameters used to configure the target computing unit are obtained according to the computing task, which can be implemented as:

[0100] The storage space address is obtained from the first calling instruction through the storage access unit; characteristic data and network parameters related to the computing task are obtained from the storage space indicated by the storage space address.

[0101] In the above steps, the characteristic data and network parameters related to the computing task are acquired through the storage access unit, which provides a data basis for the setting of the vector computing unit and the vector computing, and improves the efficiency of the vector computing.

[0102] Suppose the storage access unit is figure 2 If the storage access unit 102 is shown, then the storage access unit 102 can obtain the memory address from ld_op, and respectively read the characteristic data related to the computing task and the network parameters used to configure the target computing unit from the storage space corresponding to the memory address. For example, in figure 2 , the network parameters a1, a2, a3, b3 are obtained from the storage space by the Load Param module, and input to the vector calculation units 103, 105, and 106 respectively, so that these vector calculation units conform to the current neural network for vector calculation subtasks demand. The write-back unit 108 (assuming a 3DStride Load Input module) acquires feature data from the storage space, and outputs the feature data to the target computing unit to complete the vector computing subtask in the neural network.

[0103] Among them, different neural networks have different network parameters preset for the same type of target computing unit, so that different network parameters can meet the vector computing requirements in different application scenarios, improve the utilization rate of vector computing resources, and avoid computing resource waste. For example, the network parameters for the vector calculation process in the convolutional neural network are different from the network parameters for the vector calculation process in the recurrent neural network. The values ​​of network parameters are determined by the neural network training process. Optionally, network parameters that need to be set for the vector computing unit can be selected according to the type of neural network.

[0104] Further, in step 503, the target computing unit performs vector computing processing on the feature data to obtain the processing result of the computing task information, which can be implemented as:

[0105] The target computing unit is set by network parameters; the computing method corresponding to the target computing type is obtained from the second calling instruction; the target computing unit uses the computing method to perform vector computing processing on the feature data to obtain the target feature data as the processing result.

[0106] In the above steps, the target computing unit is set by network parameters, so that the target computing unit can be more suitable for the current application scenario, so that the vector computing processing of the feature data is completed through the set target computing unit and the calculation method obtained by the second call instruction. Obtaining the final target feature data greatly improves the vector computing efficiency and the utilization of vector computing resources in the current application scenario.

[0107] Suppose the target computing unit is figure 2 The vector calculation units 103 to 107 shown, then, set by the network parameters a1, a2, a3, b3, and from instructions 2 to 7 (ie the second call instruction, respectively denoted as mul_op, ele_op, add_op, bn_op, act_op) The calculation methods corresponding to the vector calculation units 103 to 107 are obtained respectively in . Taking the vector computing unit 103 as an example, the set vector computing unit 103 can perform the vector multiplication calculation in the neural network, and transmit the calculation result to the next-level vector computing unit as the feature data of the latter-level vector computing unit.

[0108] It can be seen that between each vector calculation unit in the feature vector processing module, vector calculation can be implemented in a fine-grained pipeline parallel way. The calculation unit can start the vector calculation of the next link. In this way, the processing efficiency of the vector computing subtask in the neural network can be greatly improved.

[0109] In practical applications, if there are multiple target computing units and there is a cascade relationship between the multiple target computing units, the feature data includes the initial feature data from the storage space and/or the target feature from the upper-level target computing unit. data. The processing result is the target feature data of the last-level target computing unit. For example, the calculation result of the vector calculation unit 107 can be used as the processing result.

[0110] In this embodiment, on the basis of abstracting various types of vector calculations in the neural network into multiple vector calculation units, the calculation tasks to be executed are obtained for the target calculation units in the multiple vector calculation units based on the input instructions, and based on the input instructions The computing task performs personalized network parameter settings on the target computing unit, thereby obtaining processing results for the feature data through the target computing unit, and implementing different types of vector computing subtasks in the neural network. Therefore, the vector computing unit can be applied to the vector computing subtasks in various neural networks, and the multiplexing of the vector computing unit greatly improves the versatility of the vector computing unit, and also improves the vector computing unit in processing various vector computing. Time flexibility, effectively reducing the waste of computing resources.

[0111] For ease of understanding, optional embodiments provided by the present disclosure are described below in combination with specific application scenarios. In this embodiment, it is assumed that the feature vector processing module used to perform the vector calculation in the following neural network is figure 2 The eigenvector processing module shown.

[0112] Take Long Short Term Memory (LSTM), which is widely used in the field of speech processing, as an example to introduce Figure 5 Data processing method shown. Among them, a variant of LSTM, M-ReluGRU, can be expressed by the following formula, specifically:

[0113] Zt=Sigmoid(WzXt+Uz·Ht_1+bz)

[0114] Gt=ReLU(BN(WhXt+Uh·Ht_1+bh))

[0115] Ht=Zt·Ht_1+(1-Zt)·Gt=Zt·(Ht_1-Gt)+Gt

[0116] Among them, sigmoid(·) refers to the nonlinear activation function, and BN(·) refers to batch normalization (BatchNormalization), that is, the calculation of f(x)=bn_a*x+bn_b is performed. WzXt refers to the result of matrix multiplication and is a vector. Uz·Ht_1 and Uh·Ht_1 are the point multiplication calculation of the vector, and the calculation can be completed in the vector calculation unit 104 . Zt, Gt, Ht, bz, and bh are all vector data, and Uz, Uh, bz, and bh are network parameters.

[0117] Assuming that each vector is stored continuously in the storage space, it can be obtained by direct addressing. Assuming that the vector calculation process for M-ReluGRU is controlled by five instructions (that is, the first instruction introduced above), based on this, the vector calculation process for M-ReluGRU is as follows:

[0118] First, in step 501, the instruction decoding unit 101 reads the first instruction from the instruction stream, and parses the following parameters from the first instruction: mul_op is MUL, ele_op is ADD, add_op is ADD, bn_op is BYPASS, act_op is Lookup. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0119] The instruction decoding unit 101 distributes ld_op (ie, the first call instruction) to the storage access unit 102 , and distributes a plurality of second call instructions obtained based on the first instruction to the vector computing units 103 to 107 respectively.

[0120]In step 502, the memory access unit 102 addresses the vectors Wzxt and Ht_1 respectively through the parity clock cycle, ie x1(2t)=Wzxt(t), x1(2t+1)=Ht_1(t). And, the network parameters are obtained through the Load Param module addressing and the corresponding vector calculation unit is output. Suppose the network parameters are a1(2t)=1, a1(2t+1)=Uz(t), a2(2t)=0, a2(2t+1)=bz(t).

[0121] In step 503, the corresponding vector calculation processing flow is performed by the vector calculation units 103 to 107, namely:

[0122] The vector calculation unit 103 performs the following vector calculations according to mul_op: x2(2t)=1·Wzxt(t), x2(2t+1)=Uz(t)·Ht_1(t). And send the calculation result to the vector calculation unit 104 .

[0123] The vector calculation unit 104 performs the following vector calculation according to ele_o: px3(2t+1)=1·Wzxt(t)+Uz(t)·Ht_1(t). And transmit the calculation result to the vector calculation unit 105 .

[0124] The vector calculation unit 105 performs the following vector calculation according to add_op: x4(2t+1)=1·Wzxt(t)+Uz(t)·Ht_1(t)+bz(t). And send the calculation result to the vector calculation unit 106 .

[0125] The vector calculation unit 106 performs the following vector calculation according to bn_op: x5(2t+1)=x4(2t+1). And send the calculation result to the vector calculation unit 107 .

[0126] The vector calculation unit 107 performs the following vector calculation according to act_op: x6(2t+1)=Sigmoid(1·Wzxt(t)+Uz·Ht_1+bz(t)). The calculation result is passed to the write-back unit 108 .

[0127] The write-back unit 108 writes all x6(2t+1) back to the specified position in the storage space according to st_op, and obtains the calculation result of Zt.

[0128] Further, in step 501, the instruction decoding unit 101 reads the second instruction from the instruction stream, and parses the following parameters from the second instruction: mul_op is MUL, ele_op is ADD, add_op is ADD, bn_op is BN, act_op. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0129] In step 502, the memory access unit 102 addresses the vectors Wzxt and Ht_1 respectively through the parity clock cycle, ie x1(2t)=WhXt, x1(2t+1)=Ht_1(t). And, the network parameters are obtained through the Load Param module addressing and the corresponding vector calculation unit is output. Suppose the network parameters are a1(2t)=1, a1(2t+1)=Uh(t); a2(2t)=0; a2(2t+1)=bz(t), a3(2t)=0, a3 (2t+1)=bn_a(t), b3(2t)=0; b3(2t+1)=bn_b(t).

[0130] In step 503, the corresponding vector calculation processing flow is performed by the vector calculation units 103 to 107, namely:

[0131] The vector calculation unit 103 performs the following vector calculations according to mul_op: x2(2t)=1·WhXt(t), x2(2t+1)=Uh(t)·Ht_1(t). And send the calculation result to the vector calculation unit 104 .

[0132] The vector calculation unit 104 performs the following vector calculation according to ele_o: x3(2t+1)=1·WhXt(t)+Uh(t)·Ht_1(t). And transmit the calculation result to the vector calculation unit 105 .

[0133] The vector calculation unit 105 performs the following vector calculation according to add_op: x4(2t+1)=1·WhXt(t)+Uh(t)·Ht_1(t)+bh(t). And send the calculation result to the vector calculation unit 106 .

[0134] The vector calculation unit 106 performs the following vector calculation according to bn_op: x5(2t+1)=BN(WhXt(t)+Uh(t)·Ht_1(t)+bh(t)). And send the calculation result to the vector calculation unit 107 .

[0135] The vector calculation unit 107 performs the following vector calculation according to act_op: x6(2t+1)=RELU(x5(2t+1)). The calculation result is passed to the write-back unit 108 .

[0136] The write-back unit 108 writes all x6(2t+1) back to the specified location in the storage space according to st_op, and obtains the calculation result of Gt.

[0137] Further, in step 501, the instruction decoding unit 101 reads the third instruction from the instruction stream, and parses the following parameters from the third instruction: mul_op is MUL, ele_op is ADD, add_op is BYPASS, bn_op is BYPASS, act_op is BYPASS. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0138] In step 502, the memory access unit 102 addresses the vectors Ht_1 and Gt respectively through parity clock cycles, ie x1(2t)=Ht_1(t), x1(2t+1)=Gt(t). And, the network parameters are obtained through the Load Param module addressing and the corresponding vector calculation unit is output. Suppose the network parameters are a1(2t)=1, a1(2t+1)=-1.

[0139] In step 503, the corresponding vector calculation process is performed by the vector calculation units 103 to 107, namely:

[0140] The vector calculation unit 103 performs the following vector calculations according to mul_op: x2(2t)=1·Ht_1(t), x2(2t+1)=−1·Gt(t). And send the calculation result to the vector calculation unit 104 .

[0141] The vector calculation unit 104 performs the following vector calculation according to ele_o: x3(2t+1)=Ht_1(t)-Gt(t). And transmit the calculation result to the vector calculation unit 105 .

[0142] Since add_op=bn_op=act_op=BYPASS, the vector calculation units 105-107 perform the following vector calculation: x6(2t+1)=x5(2t+1)=x4(2t+1)=x3(2t+1)=Ht_1( t)-Gt(t). The calculation result is passed to the write-back unit 108 .

[0143] The write-back unit 108 writes all x6(2t+1) back to the specified location in the storage space according to st_op, and obtains the calculation result of Ht_1(t)-Gt(t).

[0144] Further, in step 501, the instruction decoding unit 101 reads the fourth instruction from the instruction stream, and parses the following parameters from the fourth instruction: mul_op=BYPASS, ele_op=ADD, add_op=BYPASS, bn_op=BYPASS, act_op=BYPASS. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0145] In step 502, the memory access unit 102 addresses the vectors Ht_1-Gt and Zt respectively through parity clock cycles, ie x1(2t)=Ht_1(t)-Gt(t), x1(2t+1)=Zt(t) .

[0146] In step 503, the corresponding vector calculation process is performed by the vector calculation units 103 to 107, namely:

[0147] The vector calculation unit 103 performs the following vector calculation according to mul_op: x2(t)=x1(t). And send the calculation result to the vector calculation unit 104 .

[0148] The vector calculation unit 104 performs the following vector calculation according to ele_o: x3(2t+1)=Zt(t)·(Ht_1(t)−Gt(t)). And transmit the calculation result to the vector calculation unit 105 .

[0149] Since add_op=bn_op=act_op=BYPASS, the vector calculation units 105-107 perform the following vector calculation: x6(2t+1)=x5(2t+1)=x4(2t+1)=x3(2t+1)=Zt( t)·(Ht_1(t)-Gt(t)). The calculation result is passed to the write-back unit 108 .

[0150] The write-back unit 108 writes all x6(2t+1) back to the specified position in the storage space according to st_op, and obtains the calculation result of Zt·(Ht_1-Gt).

[0151] Finally, in step 501, the instruction decoding unit 101 reads the fifth instruction from the instruction stream, and parses the following parameters from the fifth instruction: mul_op=BYPASS, ele_op=MUL, add_op=BYPASS, bn_op=BYPASS, act_op=BYPASS. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0152] In step 502, the memory access unit 102 addresses the vectors Zt·(Ht_1-Gt) and Zt respectively through the parity clock cycle, that is, x1(2t)=Zt(t)·(Ht_1(t)-Gt(t)), x1(2t+1)=Gt(t).

[0153] In step 503, the corresponding vector calculation process is performed by the vector calculation units 103 to 107, namely:

[0154] The vector calculation unit 103 performs the following vector calculation according to mul_op: x2(t)=x1(t). And send the calculation result to the vector calculation unit 104 .

[0155] The vector calculation unit 104 performs the following vector calculation according to ele_o: x3(2t+1)=Zt(t)·(Ht_1(t)−Gt(t))+Gt(t). The calculation result is passed to the write-back unit 108 .

[0156] Since add_op=bn_op=act_op=BYPASS, the vector calculation units 105-107 perform the following vector calculation: x6(2t+1)=x5(2t+1)=x4(2t+1)=x3(2t+1)=Zt( t)·(Ht_1(t)-Gt(t))+Gt(t). And transmit the calculation result to the vector calculation unit 105 .

[0157] The write-back unit 108 writes all x6(2t+1) back to the specified position in the storage space according to st_op, and obtains the calculation result of Ht=Zt·(Ht_1−Gt)+Gt.

[0158]In this embodiment, the vector calculation in the M-ReluGRU can be completed through an instruction stream formed by five instructions, and the final target feature data Zt, Gt, and Ht (ie, the processing results described above) can be output. Therefore, the feature vector processing module provided by the present disclosure is a pipeline design, which can complete multiple vector calculation subtasks at one time, and can also complete various vector calculation subtasks by splicing multiple instructions.

[0159] In this embodiment, it is assumed that the feature vector processing module used to perform the vector calculation in the following neural network is figure 2 The eigenvector processing module shown. Taking the pooling layer (Pooling) and the linear rectification function (Rectified Linear Unit, ReLU) involved in the field of computer vision as an example, the introduction Figure 5 Data processing method shown.

[0160] The Pooling layer cascades Relu, which is a computational structure often involved in neural networks in the field of computer vision. Suppose the size of the Pooling layer is 2*2, the Stride is 2, and its type is max pooing. It is assumed that the feature data comes from a three-dimensional Tensor, denoted as F, where the three-dimensional dimensions of the three-dimensional Tensor are denoted as C, H, and W, respectively. Then, the pooling level concatenated RELU can be expressed by the following formula: Fout(i, j, k)=RELU(Max(F(i, 2*j, 2*k), F(i, 2*j+1, 2* k), F(i, 2*j, 2*k+1), F(i, 2*j+1, 2*k+1))). It can also be equivalent to the following formula:

[0161] Ftemp(i,j,k)=Max(F(i,j,2*k),F(i,j,2*k+1));

[0162] Fout(i,j,k)=RELU(Max(Ftemp(i,2*j,k),F(i,2*j+1,k))).

[0163] Based on the above formula, it is assumed that the starting storage address of the feature data in F is src0, that is, the address of the point F(i, j, k) is src0+i*H*W+j*W+k. Then, the 3D Stride addressing mode introduced above can address the vectors F(i, j, 2*k) and F(i, j, 2*k+1) respectively in the parity period. In the vector calculation process, two instructions are needed to realize the calculation of the Pooling level cascaded Relu. The specific execution steps are as follows:

[0164] In step 501, the instruction decoding unit 101 reads the first instruction from the instruction stream, and parses the following parameters from the first instruction: mul_op=BYPASS, ele_op=MAX, add_op=BYPASS, bn_op=BYPASS, act_op= BYPASS. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0165] The instruction decoding unit 101 distributes ld_op (ie, the first call instruction) to the storage access unit 102 , and distributes a plurality of second call instructions obtained based on the first instruction to the vector computing units 103 to 107 respectively.

[0166] In step 502, the memory access unit 102 addresses F(i, j, 2*k) and F(i, j, 2*k+1) respectively through parity clock cycles.

[0167] In step 503, the corresponding vector calculation process is performed by the vector calculation units 103 to 107, namely:

[0168] The vector calculation unit 103 performs the following vector calculation according to mul_op: x2(t)=x1(t). And transmit the calculation result to the vector calculation unit 104 .

[0169] The vector calculation unit 104 performs the following vector calculation according to ele_o: x3(2t+1)=MAX(x2(2t), x2(2t+1), and transmits the calculation result to the vector calculation unit 105 .

[0170] Since add_op=bn_op=act_op=BYPASS, the vector calculation units 105-107 perform the following vector calculation: x6(2t+1)=x5(2t+1)=x4(2t+1)=x3(2t+1). The calculation result is passed to the write-back unit 108 . The write-back unit 108 writes all x6(2t+1) back to the specified position in the storage space according to st_op, and obtains the calculation result of Ftemp(i, j, k).

[0171] Further, in step 501, the instruction decoding unit 101 reads the second instruction from the instruction stream, and parses the following parameters from the second instruction: mul_op=BYPASS, ele_op=MAX, add_op=BYPASS, bn_op=BYPASS, act_op=RELU. Therefore, the target computing unit to be called is determined according to the above parameters, that is, figure 2 Vector calculation units 103 to 107 are shown.

[0172] In step 502, the memory access unit 102 addresses Ftemp(i, 2*j, k) and F(i, 2*j+1, k) through parity clock cycles, respectively.

[0173] In step 503, the corresponding vector calculation process is performed by the vector calculation units 103 to 107, namely:

[0174] The vector calculation unit 103 performs the following vector calculation according to mul_op: x2(t)=x1(t). And transmit the calculation result to the vector calculation unit 104 .

[0175] The vector calculation unit 104 performs the following vector calculation according to ele_o: x3(2t+1)=MAX(x2(2t), x2(2t+1), and transmits the calculation result to the vector calculation unit 105 .

[0176] Since add_op=bn_op=BYPASS, the vector calculation units 105-106 perform the following vector calculation: x5(2t+1)=x4(2t+1)=x3(2t+1). And send the calculation result to the vector calculation unit 107 .

[0177] The vector calculation unit 107 performs the following vector calculation according to act_op: x6(2t+1)=RELU(x6(2t+1)). The calculation result is passed to the write-back unit 108 .

[0178] The write-back unit 108 writes all x6(2t+1) back to the specified location in the storage space according to st_op, and obtains the calculation result of Fout.

[0179] In this embodiment, it can be seen that the vector calculation corresponding to the Pooling level concatenated Relu structure can be completed by splicing two first instructions, which shows that the feature vector processing module provided by the present disclosure has strong versatility.

[0180] The feature vector processing module of one or more embodiments of the present disclosure will be described in detail below. Those skilled in the art can understand that these feature vector processing modules can be configured by using commercially available hardware components through the steps taught in this solution.

[0181] Image 6 A schematic structural diagram of a feature vector processing module provided in an embodiment of the present disclosure, the feature vector processing module is mounted in a computing device for running a neural network, such as Image 6 As shown, the feature vector processing module includes: an instruction decoding unit 11 , a storage access unit 12 , and a target calculation unit 13 .

[0182] The instruction decoding unit 11 is configured to, in response to the first instruction directed to the feature vector processing module, acquire the computing task information of the target computing unit under the first instruction, where the target computing unit includes a plurality of vector computations contained in the feature vector processing module Any one or more of the units, and a plurality of vector calculation units are respectively used to realize their corresponding vector calculation subtasks;

[0183] The storage access unit 12 is configured to obtain feature data related to the computing task and network parameters for configuring the target computing unit according to the computing task;

[0184] The target computing unit 13 obtained through the configuration of the network parameters is configured to perform vector computing processing on the feature data to obtain a processing result of computing task information.

[0185] Optionally, the feature vector processing module 11 is an instruction decoding unit connected to one or more vector computing units in the multiple vector computing units; the instruction decoding unit 11 obtains the computing task information of the target computing unit under the first instruction. , is configured as:

[0186] Obtaining the target computing type contained in the first instruction; determining a target computing unit matching the target computing type among the multiple vector computing units; generating a task execution instruction for indicating computing task information based on the target computing type and the target computing unit.

[0187] Wherein, the instruction decoding unit 11, when generating a task execution instruction for indicating computing task information based on the target computing type and the target computing unit, is configured as:

[0188] Obtain the storage space address contained in the first instruction, and the storage space address is used to indicate the storage space containing characteristic data and/or network parameters; based on the storage space address and the target calculation type, the information for calling the storage access unit is generated in the calculation task information. generating a first calling instruction, and sending the first calling instruction to the storage access unit; generating a second calling instruction for calling the target computing unit based on the target computing type, and sending the second calling instruction to the target computing unit.

[0189] Optionally, the storage access unit 12 is connected with one or more vector calculation units in the multiple vector calculation units; when the storage access unit 12 obtains the characteristic data related to the calculation task and the network parameters for configuring the target calculation unit according to the calculation task , is configured as:

[0190] Obtain the storage space address from the first calling instruction; obtain characteristic data and network parameters from the storage space indicated by the storage space address.

[0191] Wherein, optionally, the network parameters preset for the same type of target computing units in different neural networks are different.

[0192] Optionally, when the target computing unit 13 performs vector computing processing on the feature data, and obtains the processing result of computing task information, it is configured as:

[0193] The target computing unit is set through network parameters; the computing method corresponding to the target computing type is obtained from the second calling instruction; the vector computing processing is performed on the feature data by the computing method to obtain the target feature data used as the processing result.

[0194] Wherein, optionally, if there are multiple target computing units and there is a cascade relationship between the multiple target computing units, the feature data includes initial feature data from the storage space and/or data from a target computing unit at a higher level Target feature data; the processing result is the target feature data of the last level target computing unit.

[0195] Optionally, the plurality of vector calculation units include any one or more of the following: a vector multiplication unit, a vector point-by-point calculation unit, a vector addition unit, a normalization unit, and a nonlinear calculation unit.

[0196] Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

[0197]The device embodiments described above are only illustrative, wherein the various modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0198] In one possible design, the above Image 6 The illustrated structure of the feature vector processing module can be implemented as an electronic device. like Figure 7 As shown, the electronic device may include: a processor 21 and a memory 22 . The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, at least the processor 21 can implement the image color matching method provided in the foregoing embodiments.

[0199] The structure of the electronic device may further include a communication interface 23 for communicating with other devices or a communication network.

[0200] In addition, the present disclosure also provides a computer-readable storage medium including instructions, on which executable code is stored, and when the executable code is executed by a processor of a wireless router, causes the processor to execute the aforementioned Each embodiment provides a feature data processing method based on a neural network. Alternatively, the computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

[0201] In an exemplary embodiment, a computer program product is also provided, including a computer program/instruction, when the computer program/instruction is executed by a processor, the method for processing feature data based on a neural network provided in the foregoing embodiments is implemented. The computer program/instruction is implemented by a program running on a terminal or server.

[0202] Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0203] It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.