Data processing method and device, readable medium, electronic equipment and program product

Abstract

The invention relates to a data processing method and device, a readable medium, electronic equipment and a program product, and aims to realize combination operation of different bits and avoid additional computing resources and computing time consumption, the data processing method comprises the following steps: determining parameter data and input data of a network layer in a neural network model, the parameter data and / or the input data are / is multi-bit data; bit decomposition operation is conducted on multi-bit data in the parameter data and the input data, multiple pieces of single-bit data are obtained, and the number of the single-bit data is equal to the number of bits corresponding to the multi-bit data; performing multiplication operation on the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain a target multiplication operation result; and performing bit combination operation according to the target multiplication operation result to obtain output data of the network layer.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data processing method, apparatus, readable medium, electronic device, and program product. Background Technology

[0002] The Graphics Processing Unit (GPU) provides computing cores with different precisions, such as INT8, INT4, and INT1, and can support combinations of operations such as INT8 and INT8, INT4 and INT4, and INT1 and INT1.

[0003] Neural network layers require matrix multiplication of the weight matrix and the input activation matrix to obtain the output data. When the precision of the weight matrix and the input activation matrix differs—for example, the weight matrix is ​​4 bits and the input activation matrix is ​​8 bits—GPUs cannot support the combined operations of INT4 and INT8 data. Therefore, the weight matrix needs to be padded to 8 bits before processing. Higher precision requires more computational resources and time, thus increasing computational overhead. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the subsequent detailed description section. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, this disclosure provides a data processing method, the data processing method comprising: Determine the parameter data and input data of the network layers in the neural network model, wherein the parameter data and / or the input data are multi-bit data; A bit decomposition operation is performed on the multi-bit data in the parameter data and the input data to obtain multiple single-bit data, wherein the number of single-bit data is equal to the number of bits corresponding to the multi-bit data; Multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result; The output data of the network layer is obtained by performing bit combination operations based on the result of the target multiplication operation.

[0006] Secondly, this disclosure provides a data processing apparatus, the data processing apparatus comprising: A determination module is used to determine the parameter data and input data of the network layers in the neural network model, wherein the parameter data and / or the input data are multi-bit data; The bit decomposition module is used to perform bit decomposition operations on the multi-bit data in the parameter data and the input data to obtain multiple single-bit data, wherein the number of single-bit data is equal to the number of bits corresponding to the multi-bit data; The multiplication module is used to multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result. The bit combination module is used to perform bit combination operations based on the target multiplication result to obtain the output data of the network layer.

[0007] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in any of the first aspects above.

[0008] Fourthly, this disclosure provides an electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method described in any of the first aspects above.

[0009] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0010] The above technical solution decomposes multi-bit data in parameter and input data into single-bit data, enabling single-bit multiplication. The results of these multiplications are then combined to obtain the network layer's output data. This method decomposes data of different bit lengths into single-bit data, facilitating subsequent processing using single-bit computational cores. Multiplication results from multi-bit data can be obtained from single-bit operations, allowing for highly flexible combination operations of different bit lengths. Furthermore, it eliminates the need to supplement different bit lengths into fixed-bit data, such as replacing 8-bit and 4-bit data with 8-bit data, thus avoiding additional computational resources and time consumption.

[0011] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure; Figure 3 This is a structural block diagram of a data processing apparatus according to an exemplary embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0015] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] Neural network models typically use 32-bit or higher precision floating-point values ​​to represent model parameters (such as weights and biases) and intermediate computation results (such as activations, i.e., feature maps). Neural network models can be quantized, reducing the precision of the numerical representation, which facilitates deployment on resource-constrained devices or improves model processing speed. However, considering the need to maintain a certain level of model processing performance, quantization is usually performed to 8 bits, 16 bits, etc.

[0025] In related technologies, the computing cores provided by GPUs can support accelerated operations in fixed combinations such as INT8 and INT8, INT4 and INT4, and INT1 and INT1. For example, INT8 and INT8 indicate that they can support multiplication of 8-bit data with 8-bit data. Taking the matrix multiplication of the weight matrix and the input activation matrix in a neural network model as an example, when the precision of the weight matrix and the input activation matrix is ​​different, the data in the matrix needs to be supplemented to the precision supported by the computing core before the operation can be performed. The higher the precision, the more computing resources and time are required for the operation, thus increasing the additional computing resources and time consumption.

[0026] For example, data can be quantified based on the following formula: Q=clip(round(S / R)+Z,Qmin,Qmax)

[0027]

[0028] Where R represents the actual floating-point value, Q represents the quantized fixed-point value, Z represents the quantized fixed-point value corresponding to the floating-point value 0, and S represents the smallest scale that can be represented after quantization (floating-point scaling factor). Q represents the floating-point value after Q conversion. min and Q max These represent the preset lower limit and preset upper limit of Q, respectively. This represents the quantization error, and Z= , where R min and R max These represent the preset lower limit and preset upper limit of R, respectively.

[0029] Suppose that the weight matrix W of a certain layer in a neural network model is quantized to p bits, and the input activation matrix X is quantized to q bits. On a GPU, the corresponding computing core can be invoked for accelerated computation only if p = q = 8, 4, or 1. Arbitrary quantization combinations cannot be used for direct acceleration. However, quantization of neural network models is a trade-off between accuracy and speed. Beyond W8x8, W4x4, and W1x1 quantization combinations, there are quantization combinations that maintain accuracy better. For example, W4x8 typically offers better accuracy than W4x4. However, since there is no dedicated computing core for W4x8, W4x8 requires the use of the W8x8 computing core for accelerated computation, which introduces additional memory usage and computation time.

[0030] In view of the above, this disclosure provides a data processing method, apparatus, readable medium, electronic device, and program product to solve the aforementioned technical problems.

[0031] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0032] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 1 The data processing method includes: S101: Determine the parameter data and input data of the network layers in the neural network model.

[0033] The parameter data and / or input data are multi-bit data. The parameter data can be the weight matrix of the network layer in the neural network model, and the input data can be the input activation value matrix of the network layer in the neural network model. The specific values ​​can be determined according to the actual application scenario, and this disclosure does not impose any restrictions on them.

[0034] It is worth noting that neural network models can be models that require computational processing based on high-precision numerical representations. They can be deep learning models trained on massive amounts of text data, such as text generation models and text recognition models, or deep learning models trained on massive amounts of image data, such as image generation models and image recognition models. The specific model can be determined according to requirements, and this disclosure does not impose any limitations. A neural network model can include multiple network layers, each of which includes input data and parameter data.

[0035] For example, the parameter data and input data can be the weight matrix W and the input activation value matrix X of a certain layer in the neural network model. The model is quantized, for example, the matrix elements of the weight matrix and the input activation value matrix are both 32-bit data. The matrix elements in the weight matrix are quantized into p-bit data, and the matrix elements in the input activation value matrix are quantized into q-bit data. Matrix multiplication of W and X is performed to obtain a 32-bit output Y=WX.

[0036] S102: Perform bit decomposition on the multi-bit data in the parameter data and input data to obtain multiple single-bit data.

[0037] The number of single-bit data is equal to the number of bits corresponding to multi-bit data.

[0038] It should be understood that this disclosure does not limit the representation of parameter data and input data. For example, the data can be represented in binary, decimal, etc. Taking decimal data as an example, the decimal data can be processed into binary data and then bit decomposition operation can be performed.

[0039] S103: Multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result.

[0040] In one possible approach, multiplying the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result can include: using a single-bit matrix multiplication block in the graphics processor to multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result.

[0041] As can be seen from the above-mentioned related technologies, the computing core provided by the GPU can support INT1 and INT2, that is, accelerated operation of single-bit data. Therefore, after decomposing multi-bit data into single-bit data, the single-bit matrix multiply-accumulate (BMMA) hardware in the GPU can be used to perform multiplication of single-bit data, obtain the target multiplication result, and realize accelerated operation of arbitrary quantization combination based on BMMA.

[0042] S104: Perform bit combination operations based on the result of the target multiplication operation to obtain the output data of the network layer.

[0043] It should be understood that, taking the parameter data as the weight matrix of the network layer and the input data as the input activation value matrix of the network layer as an example, in a multi-layer network, the input activation value matrix is ​​the output data of the previous layer. The convolution operation of the network layer can be realized through the above process. For details, please refer to the processing process of the neural network model. This disclosure will not elaborate further here.

[0044] Using the above method, data of different bit positions can be decomposed into single-bit data, which facilitates subsequent processing by computing cores that support single bits. In this way, multiplication results of multi-bit data can be obtained based on single-bit data operations, thereby enabling combined operations of different bit positions. This method is highly flexible and eliminates the need to supplement data of different bit positions into data of fixed bit positions. For example, it eliminates the need to supplement two sets of data to be operated on, 8 bits and 4 bits, into 8 bits and 8 bits respectively, thus avoiding additional computing resources and computation time.

[0045] It should be noted that the parameter data and input data can be numerical data, matrix data, etc., and this disclosure does not impose any restrictions on them. For example, the multi-bit data "01" can be decomposed into two 1-bit scalar data "0" and "1".

[0046] In one possible approach, the parameter data and input data are matrix data, wherein the number of bits corresponding to each matrix element in the parameter data is equal, and the number of bits corresponding to each matrix element in the input data is equal. Bit decomposition operations on the multi-bit data in the parameter data and input data can include: when the matrix elements in the parameter data are multi-bit data, treating the data at the same bit position of each matrix element in the parameter data as a single-bit data, with the single-bit data being a matrix of the same type as the parameter data; when the matrix elements in the input data are multi-bit data, treating the data at the same bit position of each matrix element in the input data as a single-bit data, with the single-bit data being a matrix of the same type as the input data.

[0047] It should be noted that a scalar number at any position in the matrix can be decomposed into a set of 1-bit scalar data. Taking a 2-bit input activation value x as an example, the 2-bit x can be represented as x = x 1 x 0 If x ∈ int1, for example, if x is the data "01", then x 1 Corresponding scalar data "0", x 0 The corresponding scalar data is "1".

[0048] For example, taking an M×N multi-bit matrix with Q-bit elements as an example, the q-th bit of all elements in the multi-bit matrix can be extracted to determine the q-th single-bit submatrix. Here, the single-bit submatrix is ​​an M×N matrix, and the element in the m-th row and n-th column of the q-th single-bit submatrix corresponds to the q-th bit of the element in the m-th row and n-th column of the multi-bit matrix, where m is less than or equal to M, n is less than or equal to N, and q is less than or equal to Q.

[0049] For example, taking a 2×2 matrix {{01,10},{10,11}} as an example of a multi-bit matrix, by treating the data of each matrix element at the same bit position as a single bit, we can obtain single-bit matrices {{0,1},{1,1}} and {{1,0},{0,1}}. This decomposes the multi-bit data into single-bit data, facilitating subsequent combination operations.

[0050] For example, x can also be decomposed based on the following formula to obtain x. 1 and x 0 :

[0051] In other words, for multi-bit data, each bit is labeled from right to left, starting with 0. Following the formula above, for the data x corresponding to bit i... iAfter shifting the multi-bit data right by i bits and performing a logical AND operation with 1, the decomposed single-bit data x is obtained. i For example, shifting 0 and 1 one bit to the right results in 0, so 0 & 1 = 0, which gives x. 1 The corresponding scalar data is 0.

[0052] Accordingly, bit decomposition can be performed on each element in the matrix using the above calculation formula. Taking data where each element in the matrix is ​​2 bits as an example, for the element x in the i-th row and j-th column of the matrix... i,j It can be broken down into Thus, the corresponding decomposition matrix X is obtained. 1 and X 0 .

[0053] It should be noted that single-bit data can be encoded using different values. In a neural network model, a 1-bit weight matrix is ​​encoded using -1 and 1, rather than 0 and 1, to improve the accuracy of the neural network model. In this case, data 0 is encoded as -1, and data 1 is encoded as 1. To support the diversity of encoded data, an adaptive operator selection strategy is introduced, which allows for the processing of different encoded data through different bit operations XOR (logical exclusive OR operation) and AND (logical AND operation).

[0054] In possible ways, multiplying multiple single-bit data to obtain the target multiplication result can include: when the multiple single-bit data are all data encoded using 0 and 1, multiplying the single-bit data corresponding to the parameter data with the single-bit data corresponding to the input data based on a logical AND operation to obtain the target multiplication result; or, when the multiple single-bit data include data encoded using -1 and 1, multiplying the single-bit data corresponding to the parameter data with the single-bit data corresponding to the input data based on a logical XOR operation or a logical AND operation to obtain the target multiplication result.

[0055] For example, different operators are used to perform logical operations based on different encoding methods, thereby realizing the operation and processing of data with different encodings.

[0056] It should be understood that single-bit numerical data can be multiplied directly, but matrix multiplication requires multiplying both row and column data to obtain the resulting matrix. Refer to the following matrix multiplication process:

[0057] Therefore, taking the matrix multiplication of matrix A and matrix B to obtain matrix C as an example, each element in matrix C is obtained by multiplying the corresponding elements of the row data of matrix A and the column data of matrix B and then summing them up.

[0058] The following example illustrates the process of multiplying single-bit data using matrix data as an example. Since the multiplication of single-bit numerical data can be regarded as the multiplication of a first-order matrix, the process of multiplying single-bit numerical data can also refer to the process of multiplying matrix data. This disclosure will not elaborate further on this.

[0059] In one possible approach, based on a logical AND operation, multiplying the single-bit data corresponding to the parameter data with the single-bit data corresponding to the input data to obtain the target multiplication result may include: performing a logical AND operation on each single-bit data corresponding to the parameter data and each single-bit data corresponding to the input data to obtain the AND operation result, and determining the number of values ​​of 1 in each AND operation result; correspondingly, the target multiplication result includes the number of values ​​of 1 in each AND operation result.

[0060] For example, when multiple single-bit data are encoded using 0s and 1s, such as when the weight matrix W and the input activation matrix X after bit decomposition are both encoded using 0s and 1s, multiplying matrices W and X yields the matrix WX corresponding to the target multiplication result. The element in the i-th row and j-th element of matrix WX is obtained by multiplying the data in the i-th row of matrix W with the data in the j-th row of matrix X.

[0061] It should be understood that, since the values ​​of the elements in the single-bit matrices W and X are either 0 or 1, the value of the j-th element in the i-th row of matrix WX is equivalent to the number of times the value of 1 is accumulated after multiplying the corresponding elements in the i-th row of matrix W and the j-th row of matrix X.

[0062] For example, taking the multiplication of data in the i-th row of matrix W (w=[0,1]) and data in the j-th row of matrix X (x=[1,1]) as an example, then wx=popc(AND([0,1],[1,1]))=popc([0,1])=1, that is, performing a bitwise AND operation on w and x to obtain the AND result, and then using the popc(POPulation Count) instruction to count the number of values ​​of 1 in the AND result, this number corresponds to the value of the j-th element in the i-th row of matrix WX, and matrix WX is the result of the target multiplication operation. Thus, the combined operation on data encoded with 0s and 1s is realized.

[0063] It should be understood that when multiple single-bit data include data encoded with -1 and 1, the two sets of data involved in the combination operation can both be encoded with -1 and 1, or one set can be encoded with -1 and 1 and the other set can be encoded with 0 and 1.

[0064] For example, when single-bit data includes data encoded with -1 and 1, it is necessary to process the elements with a value of -1 in the matrix, changing their value from -1 to 0 before calculating the target multiplication result. This can be achieved by mapping the elements with a value of -1 to elements with a value of 0, resulting in a processed matrix. Alternatively, a linear transformation can be used. Taking matrix F=[-1,1] as an example, the following formula can be used to map the elements with a value of -1 to elements with a value of 0, resulting in a processed matrix. :

[0065] In this case, matrix J and matrix F are matrices of the same type, that is, the number of rows and columns of matrix J are equal to the number of rows and columns of matrix F, and the value of each element in matrix J is 1.

[0066] In one possible approach, when multiple single-bit data include data encoded with -1 and 1, a multiplication operation is performed between the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data, based on a logical XOR operation or a logical AND operation, to obtain the target multiplication result. This can include: when multiple single-bit data are all data encoded with -1 and 1, processing the data encoded with -1 among the multiple single-bit data into data encoded with 0, to obtain multiple target single-bit data; performing a logical XOR operation between each target single-bit data corresponding to the parameter data and each target single-bit data corresponding to the input data, to obtain the XOR operation result; determining the product result after multiplying the number of values ​​of 1 in each XOR operation result by 2, and determining the difference between the vector length corresponding to the single-bit data and each product result; correspondingly, the target multiplication result includes the difference between the vector length corresponding to the single-bit data and each product result.

[0067] For example, continuing with the single-bit weight matrix W and input activation value matrix X after bit decomposition, both matrices W and X are encoded using -1 and 1 respectively. After processing, matrices W and X are obtained as matrices encoded using 0 and 1 respectively. sum matrix Furthermore, the matrix WX corresponding to the target multiplication result of matrix W and matrix X can be obtained through the logical XOR operation.

[0068] For example, taking the multiplication operation between the data in the i-th row of matrix W (w=[-1,1]) and the data in the j-th row of matrix X (x=[1,1]) as an example, after processing w=[-1,1] to [0,1], wx=n-2×popc(XOR([0, 1], [1, 1]))=n-2×popc([1,0])=1, that is, performing a bitwise XOR operation between the processed data of w and x to obtain the XOR result, and then using the popc instruction to count the number of values ​​of 1 in the XOR result, multiplying this number by 2 to obtain the product result. The vector length corresponding to a single bit of data can be the number of rows in matrix W or the number of columns in matrix X. Determine the difference between the vector length and the product result, then this difference is the value corresponding to the j-th element in the i-th row of matrix WX, and matrix WX is the target multiplication result. Thus, the combined operation of data encoded with -1 and 1 is realized.

[0069] In one possible approach, when multiple single-bit data include data encoded with -1 and 1, a multiplication operation is performed between the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data, based on a logical XOR operation or a logical AND operation, to obtain the target multiplication result. This can include: when multiple single-bit data include data encoded with -1 and 1, processing the data encoded with -1 in the multiple single-bit data to data encoded with 0, obtaining processed single-bit data; performing a logical AND operation between each processed single-bit data and each unprocessed single-bit data in the multiple single-bit data to obtain a first operation result; for each first operation result, performing a logical AND operation between the unprocessed single-bit data corresponding to the first operation result and preset data to obtain a second operation result corresponding to the first operation result, and determining the difference between the first operation result and the corresponding second operation result, where the preset data represents data with all values ​​of 1 corresponding to the processed single-bit data; correspondingly, the target multiplication result includes the difference between each first operation result and the corresponding second operation result.

[0070] For example, continuing with the single-bit weight matrix W and input activation matrix X after bit decomposition, matrix W is encoded using -1 and 1, and matrix X is encoded using 0 and 1. In practical applications, neural network models with a 1-bit weight matrix and a q-bit input activation matrix where q is greater than 1 often use this encoding method. In this case, since the matrix contains three values—-1, 1, and 0—XOR or AND multiplication cannot be directly used. We can first process matrix W to obtain a matrix encoded using 0 and 1. For example, matrix W can be processed using the linear transformation described above. This will not be elaborated further in this publication. Such a matrix... The elements in matrix X are all data encoded using 0 and 1.

[0071] The aforementioned preset data is a matrix. Or a matrix J of the same type as matrix W, with all elements equal to 1. For each matrix... Multiplication operations are performed with matrix X, matrix X and matrix J. For details, please refer to the multiplication process of data that uses 0 and 1 encoding as described above. This disclosure will not repeat the process here.

[0072] For example, wx can be obtained based on the following formula:

[0073] Where j represents the preset data. If w=[-1,1], then j=[1,1], that is, j is a matrix of the same type as w, and the value of each element in j is 1. The operation process of x and jx can be referred to the multiplication operation process of data that uses 0 and 1 encoding as described above, and will not be repeated here.

[0074] Taking the multiplication of the i-th row data w=[-1,1] in matrix W and the j-th row data x=[0,1] in matrix X as an example, w=[-1,1] is transformed into =[0,1], then =2×1-1=1, the calculation result represents the value corresponding to the j-th element in the i-th row of matrix WX, where matrix WX is the result of the target multiplication operation. This enables combined operations on data encoded with -1 and 1 and data encoded with 0 and 1.

[0075] It is worth noting that in related technologies, taking a 1-bit weight w and a 2-bit input activation value x as an example, they are usually represented by a 4-bit integer, and then the INT4 computation primitive is used on the computing core, resulting in additional memory and computational consumption. However, in this disclosure, after performing bit decomposition on multi-bit data to obtain single-bit data, the INT1 computation primitive, namely the aforementioned BMMA, can be used on the computing core to perform the aforementioned multiplication operation process, obtaining a multi-bit output matrix representing the result of the target multiplication operation, such as 32 bits. The specific number of bits in the output matrix can be determined according to the requirements, and this disclosure does not impose any restrictions on this.

[0076] For example, continuing with the 1-bit weight w and 2-bit input activation value x mentioned above, the decomposition of the 2-bit input activation value x yields x. 1 and x 0 Then wx = w × (x 1 x 0 )=OP(w, x 1 )×2+OP(w, x 0OP(a, b) represents a calculation operation that multiplies 1-bit data into 32-bit data.

[0077] Therefore, if the above single-bit multiplication process is performed on the INT1 computation primitive of the computation core for a 1-bit weight matrix W and a 2-bit input activation matrix X, it can be represented as Y 1 =BMMA(W, X) 1 ), Y 0 =BMMA(W, X) 0 ), X 1 and X 0 Y represents the single-bit matrix obtained by decomposing matrix X. 1 Represent matrices W and X 1 The matrix corresponding to the target product obtained by performing the multiplication operation, Y 0 Represent matrices W and X 0 The matrix corresponding to the target product obtained by performing multiplication, scalar , , Let Y and Y represent matrices respectively. 1 Matrix Y 0 The scalar element in the i-th row and j-th column, and Extending this to the entire matrix operation, we get Y=Y 1 ×2+Y.

[0078] Furthermore, for any matrix multiplication combination of p bits and q bits, there are multiple target multiplication results. In one possible approach, bit combination operations are performed based on the target multiplication results to obtain the network layer's output data. This can include: for each target multiplication result, multiplying the target multiplication result by its corresponding target value to obtain the target multiplication result. The target multiplication result is obtained by multiplying the s-th single-bit data corresponding to the parameter data and the t-th single-bit data corresponding to the input data. The target value corresponding to the target multiplication result is 2 raised to the power of (s+t), where s and t are integers greater than or equal to 0, and s is less than or equal to the number of bits corresponding to the parameter data, and t is less than or equal to the number of bits corresponding to the input data. The multiple target multiplication results are then accumulated to obtain the network layer's output data.

[0079] For example, refer to Figure 2 Taking a p-bit weight matrix W and a q-bit input activation matrix X as an example, we first decompose the p-bit weight matrix W to obtain a 1-bit matrix Wi. s s∈{0,1,…,p-1} and matrix X t t∈{0,1,…,q-1}, matrix Ws elements in Matrix X t elements in Then, the BMMA operation is called p×q times to obtain a series of 1-bit matrix multiplication components Y. s,t =BMMA(W s X t ).

[0080] Furthermore, based on the result of the target multiplication operation, a bit combination operation is performed to accumulate all 1-bit matrix multiplication components to obtain a multi-bit output matrix. .

[0081] Using the above method, a novel computational kernel is provided. This kernel can decompose the operations of arbitrary quantization combinations of a neural network model into a series of 1-bit computational primitive operations. Specifically, it can decompose the operations of arbitrary quantization combinations of a neural network model into the superposition of 1-bit matrix multiplication operations, and call the 1-bit computational primitive BMMA in the computational kernel to implement the multiplication process. This enables operations combining different bits, offering high flexibility and avoiding additional computational resources and time consumption. Furthermore, a data-adaptive operator selection strategy is introduced, which can flexibly convert the numerical range of the input data according to its encoding method. This allows the computational kernel to automatically support various input data with different encoding methods (e.g., -1 / +1 and 0 / 1) through bitwise XOR and AND logical operations.

[0082] It should be understood that the above calculation formula is only an example. In practical applications, the calculation formula can be modified, and this disclosure does not limit it.

[0083] Based on the same concept, embodiments of this disclosure also provide a data processing apparatus, such as... Figure 3 As shown, the data processing device 300 may include: The determining module 301 is used to determine the parameter data and input data of the network layer in the neural network model, wherein the parameter data and / or the input data are multi-bit data; Bit decomposition module 302 is used to perform bit decomposition operation on multi-bit data in the parameter data and the input data to obtain multiple single-bit data, wherein the number of single-bit data is equal to the number of bits corresponding to the multi-bit data; The multiplication module 303 is used to perform a multiplication operation on the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result. Bit combination module 304 is used to perform bit combination operation based on the target multiplication result to obtain the output data of the network layer.

[0084] Using the above-mentioned device, data of different bit positions can be decomposed into single-bit data, which makes it easier to perform subsequent calculations using a computing core that supports single bits. In this way, the result of multiplication of multi-bit data can be obtained based on the calculation of single-bit data, thereby realizing the combination of different bit positions. This is highly flexible and eliminates the need to supplement the data of different bit positions into data of fixed bit positions. For example, it eliminates the need to supplement the two sets of data to be calculated, 8 bits and 4 bits, into 8 bits and 8 bits respectively, thus avoiding additional computing resources and computation time.

[0085] Optionally, the parameter data and the input data are matrix data, wherein the number of bits corresponding to each matrix element in the parameter data is equal, and the number of bits corresponding to each matrix element in the input data is equal. The bit decomposition module 302 is used for: When the matrix elements in the parameter data are multi-bit data, the data of each matrix element in the parameter data that are in the same bit position are regarded as a single bit data, and the single bit data is a matrix of the same type as the parameter data; When the matrix elements in the input data are multi-bit data, the data where each matrix element in the input data is in the same bit position is treated as a single-bit data, and the single-bit data is a matrix of the same type as the input data.

[0086] Optionally, the multiplication module 303 includes: The first arithmetic module is configured to, when all the multiple single-bit data are encoded using 0 and 1, perform a logical AND operation to multiply the single-bit data corresponding to the parameter data with the single-bit data corresponding to the input data to obtain the target multiplication result; or, The second operation module is used to perform a multiplication operation on the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data based on a logical XOR operation or a logical AND operation when the multiple single-bit data include data encoded with -1 and 1, so as to obtain the target multiplication result.

[0087] Optionally, the first arithmetic module is used for: Perform a logical AND operation between each single bit of the parameter data and each single bit of the input data to obtain the AND operation result, and determine the number of 1s in each AND operation result; Accordingly, the target multiplication result includes the number of times the value is 1 in each AND operation result.

[0088] Optionally, the second arithmetic module is used for: When all the single-bit data are obtained by encoding with -1 and 1, the data encoded with -1 in the multiple single-bit data are processed into data encoded with 0 to obtain multiple target single-bit data; Perform a logical XOR operation between each target single-bit data corresponding to the parameter data and each target single-bit data corresponding to the input data to obtain the XOR operation result. Determine the product of the number of 1s in each XOR operation result multiplied by 2, and determine the difference between the vector length corresponding to the single bit data and each product result; Accordingly, the result of the target multiplication operation includes the difference between the vector length corresponding to the single-bit data and each of the product results.

[0089] Optionally, the second arithmetic module is used for: When the plurality of single-bit data includes data encoded with -1 and 1, the data encoded with -1 in the plurality of single-bit data is processed into data encoded with 0 to obtain the processed single-bit data; Perform a logical AND operation between each processed single bit data and each unprocessed single bit data in the plurality of single bit data to obtain the first operation result; For each of the first operation results, perform a logical AND operation on the unprocessed single-bit data corresponding to the first operation result and the preset data to obtain the second operation result corresponding to the first operation result, and determine the difference between the first operation result and the corresponding second operation result. The preset data represents the data with all values ​​of 1 corresponding to the processed single-bit data. Accordingly, the target multiplication result includes the difference between each first operation result and the corresponding second operation result.

[0090] Optionally, the target multiplication result is multiple, and the bit combination module 304 is used for: For each target multiplication result, the target multiplication result is multiplied by the target value corresponding to the target multiplication result to obtain the target multiplication result corresponding to the target multiplication result. The target multiplication result is obtained by multiplying the s-th single bit data corresponding to the parameter data and the t-th single bit data corresponding to the input data. The target value corresponding to the target multiplication result is 2 raised to the power of (s+t), where s and t are integers greater than or equal to 0, and s is less than or equal to the number of bits corresponding to the parameter data, and t is less than or equal to the number of bits corresponding to the input data. The results of multiplying multiple targets are accumulated to obtain the output data of the network layer.

[0091] Optionally, the multiplication module 303 is used for: The single-bit matrix multiplication operation block in the graphics processor is used to multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result.

[0092] Optionally, the parameter data is the weight matrix of the network layer in the neural network model, and the input data is the input activation value matrix of the network layer in the neural network model.

[0093] Based on the same concept, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any of the above data processing methods.

[0094] Based on the same concept, this disclosure also provides an electronic device that may include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of any of the above data processing methods.

[0095] Based on the same concept, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above data processing methods.

[0096] The following is for reference. Figure 4 This diagram illustrates a structural schematic of an electronic device 400 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0097] like Figure 4As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. Processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0098] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0099] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0100] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0101] In some implementations, communication can be conducted using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0102] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0103] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine parameter data and input data of a network layer in a neural network model, wherein the parameter data and / or the input data are multi-bit data; perform bit decomposition operations on the multi-bit data in the parameter data and the input data to obtain multiple single-bit data, wherein the number of single-bit data is equal to the number of bits corresponding to the multi-bit data; perform multiplication operations on the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain a target multiplication result; and perform bit combination operations based on the target multiplication result to obtain the output data of the network layer.

[0104] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0107] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0110] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0111] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A data processing method, characterized in that, The data processing method includes: Determine the parameter data and input data of the network layers in the neural network model, wherein the parameter data and / or the input data are multi-bit data; A bit decomposition operation is performed on the multi-bit data in the parameter data and the input data to obtain multiple single-bit data, wherein the number of single-bit data is equal to the number of bits corresponding to the multi-bit data; Multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result; The output data of the network layer is obtained by performing bit combination operations based on the result of the target multiplication operation.

2. The data processing method according to claim 1, characterized in that, The parameter data and the input data are matrix data, wherein the number of bits corresponding to each matrix element in the parameter data is equal, and the number of bits corresponding to each matrix element in the input data is equal. The bit decomposition operation on the multi-bit data in the parameter data and the input data includes: When the matrix elements in the parameter data are multi-bit data, the data of each matrix element in the parameter data that are in the same bit position are regarded as a single bit data, and the single bit data is a matrix of the same type as the parameter data; When the matrix elements in the input data are multi-bit data, the data where each matrix element in the input data is in the same bit position is treated as a single-bit data, and the single-bit data is a matrix of the same type as the input data.

3. The data processing method according to claim 1, characterized in that, The step of performing a multiplication operation based on the multiple single-bit data to obtain the target multiplication result includes: When all the single-bit data are encoded using 0 and 1, a logical AND operation is performed to multiply the single-bit data corresponding to the parameter data with the single-bit data corresponding to the input data to obtain the target multiplication result; or, When the plurality of single-bit data includes data encoded with -1 and 1, the single-bit data corresponding to the parameter data is multiplied by the single-bit data corresponding to the input data based on logical XOR or logical AND operations to obtain the target multiplication result.

4. The data processing method according to claim 3, characterized in that, The step of multiplying the single-bit data corresponding to the parameter data with the single-bit data corresponding to the input data based on the logical AND operation to obtain the target multiplication result includes: Perform a logical AND operation between each single bit of the parameter data and each single bit of the input data to obtain the AND operation result, and determine the number of 1s in each AND operation result; Accordingly, the target multiplication result includes the number of times the value is 1 in each AND operation result.

5. The data processing method according to claim 3, characterized in that, When the plurality of single-bit data includes data encoded using -1 and 1, based on logical XOR or logical AND operations, the single-bit data corresponding to the parameter data is multiplied by the single-bit data corresponding to the input data to obtain the target multiplication result, including: When all the single-bit data are obtained by encoding with -1 and 1, the data encoded with -1 in the multiple single-bit data are processed into data encoded with 0 to obtain multiple target single-bit data; Perform a logical XOR operation between each target single-bit data corresponding to the parameter data and each target single-bit data corresponding to the input data to obtain the XOR operation result. Determine the product of the number of 1s in each XOR operation result multiplied by 2, and determine the difference between the vector length corresponding to the single bit data and each product result; Accordingly, the result of the target multiplication operation includes the difference between the vector length corresponding to the single-bit data and each of the product results.

6. The data processing method according to claim 3, characterized in that, When the plurality of single-bit data includes data encoded using -1 and 1, based on logical XOR or logical AND operations, the single-bit data corresponding to the parameter data is multiplied by the single-bit data corresponding to the input data to obtain the target multiplication result, including: When the plurality of single-bit data includes data encoded with -1 and 1, the data encoded with -1 in the plurality of single-bit data is processed into data encoded with 0 to obtain the processed single-bit data; Perform a logical AND operation between each processed single bit data and each unprocessed single bit data in the plurality of single bit data to obtain the first operation result; For each of the first operation results, perform a logical AND operation on the unprocessed single-bit data corresponding to the first operation result and the preset data to obtain the second operation result corresponding to the first operation result, and determine the difference between the first operation result and the corresponding second operation result. The preset data represents the data with all values ​​of 1 corresponding to the processed single-bit data. Accordingly, the target multiplication result includes the difference between each first operation result and the corresponding second operation result.

7. The data processing method according to any one of claims 1-6, characterized in that, The target multiplication operation results in multiple steps. The step of performing bit combination operations based on the target multiplication operation results to obtain the output data of the network layer includes: For each target multiplication result, the target multiplication result is multiplied by the target value corresponding to the target multiplication result to obtain the target multiplication result corresponding to the target multiplication result. The target multiplication result is obtained by multiplying the s-th single bit data corresponding to the parameter data and the t-th single bit data corresponding to the input data. The target value corresponding to the target multiplication result is 2 raised to the power of (s+t), where s and t are integers greater than or equal to 0, and s is less than or equal to the number of bits corresponding to the parameter data, and t is less than or equal to the number of bits corresponding to the input data. The results of multiplying multiple targets are accumulated to obtain the output data of the network layer.

8. The data processing method according to any one of claims 1-6, characterized in that, The step of multiplying the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result includes: The single-bit matrix multiplication operation block in the graphics processor is used to multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result.

9. The data processing method according to any one of claims 1-6, characterized in that, The parameter data is the weight matrix of the network layer in the neural network model, and the input data is the input activation value matrix of the network layer in the neural network model.

10. A data processing apparatus, characterized in that, The data processing device includes: A determination module is used to determine the parameter data and input data of the network layers in the neural network model, wherein the parameter data and / or the input data are multi-bit data; The bit decomposition module is used to perform bit decomposition operations on the multi-bit data in the parameter data and the input data to obtain multiple single-bit data, wherein the number of single-bit data is equal to the number of bits corresponding to the multi-bit data; The multiplication module is used to multiply the single-bit data corresponding to the parameter data and the single-bit data corresponding to the input data to obtain the target multiplication result. The bit combination module is used to perform bit combination operations based on the target multiplication result to obtain the output data of the network layer.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the computer program performs the steps of the method according to any one of claims 1-9.

12. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-9.

Images