Data processing method based on reconstruction model, electronic device and storage medium

Abstract

The application provides a data processing method based on a reconstruction model, an electronic device and a storage medium. The method comprises the following steps: analyzing the network structure of a model deployed in an integrated circuit; replacing a matrix multiplication operator in a first structure block with a preset replacement operator to obtain an improved first structure block; constructing a plurality of replacement neurons based on the improved first structure block and a second structure block; obtaining an improved network structure based on the plurality of replacement neurons and a third structure block, wherein the improved network structure comprises a data processing structure and a data application structure; in response to a processing request for input data, calling the data processing structure to perform Zernike matrix transformation on the input data to obtain a one-dimensional tensor corresponding to the input data, and constructing a two-dimensional tensor based on the one-dimensional tensor; and calling the data application structure to process the two-dimensional tensor and output a processing result corresponding to the input data. The above method can reduce the hardware cost during model operation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a data processing method, electronic device and storage medium based on a reconstruction model. Background Technology

[0002] Different task types of datasets have different dimensional features. For example, one-dimensional time series / tabular data is suitable for feedforward neural networks, two-dimensional / three-dimensional image pixels are suitable for convolutional neural networks, and two-dimensional sequence and word embedding features are suitable for generative Transformers. Therefore, it is necessary to design neural networks with corresponding configurations to efficiently extract features. However, the computational core of existing neural network architectures all rely on matrix multiplication operations. The hardware implementation of matrix multiplication requires a large number of multiplication instructions and dedicated multiplication circuits. This not only leads to a large number of gate circuits and a significant increase in integrated circuit area, but also introduces serious hardware overhead problems such as limited clock frequency and excessive power consumption. Summary of the Invention

[0003] This application discloses a data processing method, electronic device, and storage medium based on a reconstruction model, which solves the technical problem that existing neural network models rely on matrix multiplication operations, resulting in high operating costs.

[0004] This application provides a data processing method based on a reconstruction model, applied to an electronic device including an integrated circuit. The method includes: parsing the network structure of a model deployed in the integrated circuit, the network structure including a first structural block, a second structural block, and a third structural block, wherein the first structural block includes a matrix multiplication operator; replacing the matrix multiplication operator in the first structural block with a preset replacement operator to obtain an improved first structural block; constructing multiple replacement neurons based on the improved first structural block and the second structural block; obtaining an improved network structure based on the multiple replacement neurons and the third structural block, the improved network structure including a data processing structure and a data application structure; responding to a processing request for input data, calling the data processing structure to perform a Zernike moment transformation on the input data to obtain a one-dimensional tensor corresponding to the input data, and constructing a two-dimensional tensor based on the one-dimensional tensor; calling the data application structure to process the two-dimensional tensor and outputting the processing result corresponding to the input data.

[0005] In some embodiments of this application, the replacement operator includes a Pythagorean operator or a harmonic operator, and the second structural block includes unconnected zscore operator layers and activation function operators. The construction of multiple replacement neurons based on the improved first structural block and the second structural block includes: deleting the activation function operator; and connecting the zscore operator layer with the Pythagorean operator in the improved first structural block, using the output of the zscore operator layer as the input of the Pythagorean operator in the improved first structural block to construct at least one Pythagorean neuron, which is then used as the replacement neuron; or, connecting the zscore operator layer with the harmonic operator in the improved first structural block, using the output of the zscore operator layer as the input of the harmonic operator in the improved first structural block to construct at least one harmonic neuron, which is then used as the replacement neuron.

[0006] In some embodiments of this application, the replacement operator includes a Pythagorean operator or a harmonic operator, and the second structural block includes a zscore operator layer and an activation function layer that are not connected to each other. The step of constructing multiple replacement neurons based on the improved first structural block and the second structural block includes: connecting the Pythagorean operator in the improved first structural block to the zscore operator layer and the activation function layer respectively to construct at least one Pythagorean neuron, and using the Pythagorean neuron as the replacement neuron; wherein the input of the Pythagorean operator in the improved first structural block is the output of the zscore operator layer, and the output of the Pythagorean operator in the improved first structural block is the input of the activation function layer; or, connecting the harmonic operator in the improved first structural block to the zscore operator layer and the activation function layer respectively to construct at least one harmonic neuron, and using the harmonic neuron as the replacement neuron; wherein the input of the harmonic operator in the improved first structural block is the output of the zscore operator layer, and the output of the harmonic operator in the improved first structural block is the input of the activation function layer.

[0007] In some embodiments of this application, the step of calling the data application structure to process the two-dimensional tensor and outputting the processing result corresponding to the input data includes: processing the two-dimensional tensor based on the temporal logic of the data application structure to obtain an intermediate result; if the intermediate result is the input of any replacement neuron in the data application structure, calling one or more of the adder, shifter, logic unit, register, and XOR gate associated with the replacement neuron, wherein one or more of the adder, shifter, logic unit, register, and XOR gate are integrated on the integrated circuit; inputting the intermediate result into the replacement neuron, processing the intermediate result using one or more of the adder, shifter, logic unit, register, and XOR gate to obtain the processing result of the replacement neuron; and determining the processing result corresponding to the input data based on the processing result of the replacement neuron.

[0008] In some embodiments of this application, the step of calling the data processing structure to perform a Zernike moment transform on the input data to obtain a one-dimensional tensor corresponding to the input data includes: when the input data is image data, segmenting the image data based on a preset sliding window to obtain multiple window regions; determining an order selection strategy based on the number of the multiple window regions; generating a first transform order corresponding to the Zernike moment transform based on the order selection strategy; and performing multiple Zernike moment transforms on the multiple window regions based on the first transform order to generate a one-dimensional tensor corresponding to the image data.

[0009] In some embodiments of this application, the step of performing multiple Zernike moment transformations on the plurality of window regions based on the first transformation order to generate a one-dimensional tensor corresponding to the image data includes: performing a Zernike moment transformation on each window region based on the first transformation order to generate a one-dimensional tensor corresponding to each window region; generating a first intermediate tensor based on the one-dimensional tensor corresponding to each window region; performing a Zernike moment transformation on each slice of the first intermediate tensor based on the first transformation order to generate a one-dimensional tensor corresponding to each slice; generating a second intermediate tensor based on the one-dimensional tensor corresponding to each slice; and performing a Zernike moment transformation on the second intermediate tensor based on the first transformation order to generate a one-dimensional tensor corresponding to the image data.

[0010] In some embodiments of this application, the step of performing multiple Zernike moment transformations on the plurality of window regions based on the first transformation order to generate a one-dimensional tensor corresponding to the image data includes: when the input data is text data, performing word segmentation and tensor transformation on the text data to generate a sequence tensor; determining an order selection strategy based on the number of segments corresponding to the sequence tensor; generating a second transformation order corresponding to the Zernike moment transformation based on the order selection strategy, wherein the second transformation order includes a first order and a second order, and the first order is greater than the second order; performing a Zernike moment transformation on the sequence tensor based on the first order to generate a transformed tensor; and performing a Zernike moment transformation on the transformed tensor based on the second order to generate a one-dimensional tensor corresponding to the text data.

[0011] In some embodiments of this application, the order selection strategy includes one or more of the following: optical wavefront aberration analysis strategy, image rectangular description strategy, adaptive optics real-time correction strategy, and progressive verification strategy.

[0012] This application also provides an electronic device, which includes an integrated circuit and a memory. The integrated circuit includes a control chip, and the memory stores a computer program. When the control chip executes the computer program, it implements the above-described data processing method based on the reconstruction model.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data processing method based on the reconstruction model.

[0014] In the data processing method based on the reconstruction model provided in this application, the network structure of the model deployed in the integrated circuit of an electronic device is analyzed so that the network structure can be improved subsequently, thereby reducing the operating cost of the model. The matrix multiplication operator in the first structural block is replaced with a preset replacement operator to obtain an improved first structural block. Based on the improved first and second structural blocks, multiple replacement neurons are constructed to improve the network structure. Based on the multiple replacement neurons and the third structural block, an improved network structure is obtained, which includes a data processing structure and a data application structure. In response to the processing request for input data, the data processing structure is invoked to perform a Zernike moment transformation on the input data, thereby handling input data with different dimensional features. A one-dimensional tensor obtained after the Zernike moment transformation is obtained, and a two-dimensional tensor is constructed based on the one-dimensional tensor. The data application structure is invoked to process the two-dimensional tensor, and the processing structure corresponding to the input data is output, thereby improving the model's processing efficiency for input data while reducing hardware costs. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0016] Figure 2 This is a flowchart of a data processing method based on a reconstruction model provided in an embodiment of this application.

[0017] Figure 3 This is a comparative schematic diagram of neurons provided in the embodiments of this application.

[0018] Figure 4 This is a schematic diagram of the Pythagorean neuron and the Pythagorean neuron provided in the embodiments of this application.

[0019] Figure 5 This is a partial schematic diagram of the improved network structure provided in the embodiments of this application.

[0020] Figure 6 This is a comparison diagram of the network structure before and after the improvement provided in the embodiments of this application.

[0021] Figure 7 This is a schematic diagram of the calculation result of the matrix multiplication operator provided in the embodiments of this application.

[0022] Figure 8 This is a schematic diagram of the calculation results of the Pythagorean operator provided in the embodiments of this application.

[0023] Figure 9 This is a schematic diagram of the calculation results of the harmonic operator provided in the embodiments of this application. Detailed Implementation

[0024] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0025] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0026] Different task types of datasets have different dimensional features. For example, one-dimensional time series / tabular data is suitable for feedforward neural networks, two-dimensional / three-dimensional image pixels are suitable for convolutional neural networks, and two-dimensional sequence and word embedding features are suitable for generative Transformers. Therefore, it is necessary to design neural networks with corresponding configurations to efficiently extract features. However, the computational core of existing neural network architectures all rely on matrix multiplication operations. The hardware implementation of matrix multiplication requires a large number of multiplication instructions and dedicated multiplication circuits. This not only leads to a large number of gate circuits and a significant increase in integrated circuit area, but also introduces serious hardware overhead problems such as limited clock frequency and excessive power consumption.

[0027] To avoid problems such as large number of gate circuits, large integrated circuit area, limited clock frequency, and high power consumption caused by relying on matrix multiplication operations, this application proposes a data processing method, electronic device, and storage medium based on a reconstruction model. It can improve the network structure of the model by using Pythagorean operators, so that the improved network structure does not need to rely on a large number of multiplication instructions and dedicated multiplication circuits corresponding to matrix multiplication operations during operation.

[0028] The data processing method based on the reconstruction model provided in this application can be applied to electronic devices. The structure of the electronic device is described below.

[0029] Figure 1 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 1 As shown, the electronic device 10 can be a mobile phone, tablet computer, smart wearable device, augmented reality (AR) / virtual reality (VR) device, laptop computer, netbook, etc. This application embodiment does not impose any restrictions on the specific type of electronic device 10.

[0030] like Figure 1 As shown, the electronic device 10 may include an integrated circuit 100, a display device 110, a communication module 120, a memory 130, a processor 140, an input / output (I / O) interface 150, and a bus 160. The processor 140 is coupled to the integrated circuit 100, the display device 110, the communication module 120, the memory 130, and the I / O interface 150 via the bus 160.

[0031] Integrated circuit 100 includes, but is not limited to, control chip 1001, adder 1002, shifter 1003, logic unit 1004, register 1005, XOR gate 1006, etc. Processor 140 can deploy the received model on integrated circuit 100 to improve the network structure of the model and process the input data using the improved network structure. Control chip 1001 provides computational and control capabilities, used to invoke adder 1002, shifter 1003, logic unit 1004, register 1005, and XOR gate 1006 to perform operations of Pythagorean theorem units composed of Pythagorean operators.

[0032] Display device 110 may be a touch screen, specifically a touch-sensitive liquid crystal display device. Alternatively, display device 110 may be a non-touch screen. Display device 110 is used to display the network structure of a model, and / or an improved network structure, and / or the processing results of input data, etc.

[0033] The communication module 120 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR). The memory 130 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 140, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0034] Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.

[0035] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 140. Non-volatile memory can include disk storage devices and flash memory.

[0036] The memory 130 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 140 or the control chip 1001. Exemplarily, the one or more computer programs include multiple instructions that, when executed by the control chip 1001, can implement a data processing method based on a reconfiguration model that is executed on the electronic device 10.

[0037] In other embodiments, the electronic device 10 also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10.

[0038] Processor 140 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0039] The processor 140 provides computing and control capabilities. For example, the processor 140 is used to execute computer programs stored in the memory 130 to assist the control chip 1001 in implementing the above-described data processing method based on the reconstructed model. For example, the processor 140 is used to deploy the user-input model on the integrated circuit 100 and trigger the control chip 1001 to parse the deployed model, improve the network structure, and process the model.

[0040] I / O interface 150 is used to provide a channel for user input or output. For example, I / O interface 150 can be used to connect various input and output devices, such as mouse, keyboard, touch device, display screen, etc., so that users can enter information or visualize information.

[0041] Bus 160 is used at least to provide a channel for communication between integrated circuit 100, display device 110, communication module 120, memory 130, processor 140 and I / O interface 150 in electronic device 10.

[0042] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0043] Figure 2 This is a flowchart of a data processing method based on a reconstruction model provided in an embodiment of this application, applied in electronic devices (e.g., Figure 1 In electronic device 10). Depending on different needs, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0044] Step S201: Analyze the network structure of the model deployed in the integrated circuit.

[0045] In some embodiments of this application, the electronic device includes an integrated circuit configured to analyze the network structure, improve the network structure, and operate the improved network structure, etc.

[0046] Users can input the model file corresponding to the model to be parsed on an electronic device. This model file contains not only weight values ​​but also a complete schematic diagram of the network structure. The electronic device inputs the received model file into an integrated circuit, which parses the network structure of the model. Specifically, it can parse the overall design blueprint of each layer from the input layer to the output layer, including the type, arrangement, connection method, and parameter configuration of each layer (such as dimension, kernel size, etc.).

[0047] Taking an image recognition model as an example, the parsed network structure can include one or more of the following: input layer (such as a 224×224×3 image), convolutional layer, pooling layer, batch normalization layer, activation layer (ReLU), fully connected layer, softmax output layer, and skip connections (such as residual blocks) and their connection relationships.

[0048] Taking a text recognition model as an example, the parsed network structure can include one or more of the following: input layer (such as index sequence), word embedding layer, recurrent layer (LSTM / GRU) or self-attention layer (Transformer), position encoding, layer normalization, feedforward network, and output layer (such as fully connected + Softmax mapping to vocabulary) and their connection relationships.

[0049] In some embodiments of this application, the parsed network structure is divided into a first structural block, a second structural block, and a third structural block.

[0050] The first structural block includes matrix multiplication operators. For example, it identifies each layer in the network structure and the computational units within each layer. If an identified computational unit is constructed at least by matrix multiplication operators, then that computational unit is recorded as the first structural block. For instance, if a computational unit in a fully connected layer is identified as being constructed at least by matrix multiplication operators, then the computational unit of the fully connected layer is referred to as the first structural block.

[0051] In another example, the first structural block may include convolution operators (which are further subdivided into standard convolution, dilated convolution (also known as extended convolution), grouped convolution, and deconvolution). Since 2D and above nD convolution operations can be equivalently converted into matrix multiplication operations by the im2col algorithm, the computational unit is recorded as the first structural block when it is determined that the computational unit is at least constructed from convolution operators and associated with the im2col algorithm.

[0052] The first structural block is connected to the second structural block. For example, the level or other unit connected to the computation unit containing the matrix multiplication operator is called the second structural block. That is, the level or other unit connected to the first structural block is called the second structural block.

[0053] The first and third structural blocks are not connected. For example, any level or other unit in the network structure that is not connected to the first structural block is referred to as a third structural block. At least some third structural blocks can be connected to the second structural block.

[0054] Step S202: Replace the matrix multiplication operator in the first structural block with a preset replacement operator to obtain the improved first structural block.

[0055] In some embodiments of this application, since matrix multiplication operators are indivisible in terms of processor timing, or even if matrix multiplication operators are forcibly serially segmented into timing, a lot of intermediate data needs to be stored and transmitted over long distances. Therefore, in order to reduce the hardware cost of integrated circuits and improve operating efficiency, matrix multiplication operators can be replaced by substitution operators. The substitution operators can be Pythagorean operators or harmonic operators.

[0056] Pythagorean operators can be nonlinear operators with built-in activation and compression properties. Pythagorean operators include the summation and square root operation and the summation and square root reciprocal operation. The summation and square root operation can be implemented using the CORDIC algorithm, and the summation and square root reciprocal operation can be implemented using Carmack's magic number fast algorithm.

[0057] In one example, under chip design cost constraints (such as the sum of length, width, and height ≤ 13), the matrix multiplication operator uses a fixed parallel 8×8 multiplication array, completing the calculation in one clock cycle. The sum of its area (8+8) and delay (1) is 17, which does not meet the constraints. However, the CORDIC operator, through iterative addition and shifting, can reduce the hardware unit to 4×4 and reuse it over 4 clock cycles, thereby adjusting the trade-off between area and delay to 4+4+4=12, which meets the constraints. By using the CORDIC algorithm, an additional dimension of margin that can be divided, folded, reused, and compromised is added, that is, flexible trade-offs can be made between area and time, providing greater design redundancy space for overcoming the computational wall, bandwidth wall, and memory wall.

[0058] Harmonic operators can also be nonlinear operators with inherent activation and compression properties. Compared to Pythagorean operators, the high-complexity parts of harmonic operators can be implemented using ln operations. Harmonic operators include summation followed by natural logarithm operations, summation followed by reciprocal operations, and ln operations. The ln operation can be implemented using the CORDIC algorithm. Compared to Pythagorean operators, harmonic operators have advantages in large-scale computations; for example, harmonic operators can reuse CORDIC PE circuits during forward and backward propagation chain differentiation.

[0059] The first structural block, with the matrix multiplication operator replaced, is identified as the improved first structural block. Alternatively, if the second structural block connected to the first structural block contains a separate activation function operator (such as sigmoid) used in conjunction with the matrix multiplication operator, the matrix multiplication operator in the first structural block can be replaced with a Pythagorean theorem or a harmonic operator, and the second structural block containing the separate activation function operator (such as sigmoid) can be deleted to obtain the improved first structural block. Or, if the first structural block contains a separate activation function operator (such as sigmoid) used in conjunction with the matrix multiplication operator, the matrix multiplication operator and sigmoid in the first structural block can be replaced with a Pythagorean theorem or a harmonic operator to obtain the improved first structural block.

[0060] Step S203: Based on the improved first and second structural blocks, construct multiple replacement neurons.

[0061] In some embodiments of this application, the first structural block is connected to the improved second structural block. Therefore, the improved first structural block is also connected to the second structural block, thereby constructing multiple replacement neurons. The number of replacement neurons can be determined based on the number of matrix multiplication operators.

[0062] In some embodiments of this application, if the second structural block includes unconnected zscore operator layers and activation function operators, and the zscore operator layer is connected to the matrix multiplication operator of the first structural block before improvement, then the zscore operator layer can be connected to the replacement operator (such as the Pythagorean operator or the harmonic operator) of the first structural block after improvement. Furthermore, since the matrix multiplication operator needs to be used in conjunction with a separate explicit activation function operator (such as sigmoid) during operation, this separate explicit activation function operator (such as sigmoid) can also be deleted when the matrix multiplication operator is replaced. The zscore operator layer includes one or more redistribution operators from Batch Normalization (BN), Layer Normalization (LN), Instance Normalization (IN), and Group Normalization (GN), depending on the slice dimension and grouping method.

[0063] Combination Figure 3 As shown in (a), the output of the zscore operator layer is the input of the matrix multiplication operator, and the neurons built based on the matrix multiplication operator need to be used in conjunction with separate explicit activation function operators. Figure 3 As shown in (b), after replacing the matrix multiplication operator with the replacement operator, the output of the zscore operator layer becomes the input of the replacement operator. Furthermore, the replacement neuron constructed based on the replacement neuron (such as the Pythagorean neuron or the harmonic neuron) does not need to be used with a separate explicit activation function operator.

[0064] Therefore, the first structural block before the improvement can include Figure 3 The matrix multiplication operator and activation function operator shown in (a) can be improved by including the first structural block. Figure 3 The replacement operator is shown in (b). The unmodified neuron includes... Figure 3 The zscore operator layer, matrix multiplication operator, and activation function operator are shown in (a); the improved replacement neuron includes the zscore operator layer and the replacement operator.

[0065] In some embodiments of this application, to enhance expressive power, cross-layer connections can be made to increase residuals and construct replacement neurons with residual connections and activation functions. For example, the replacement operator in the improved first structural block is connected to the zscore operator layer and the activation function layer (including the lnabsa1 activation function layer) respectively to construct at least one replacement neuron, for example, a replacement neuron with residual connections and activation functions.

[0066] Combination Figure 4 As shown in (a), taking the zscore operator layer as the batch_normalization layer and the replacement operator as the Pythagorean operator as an example, the Pythagorean neuron with residual connections and activation functions includes a batch_normalization layer, a Pythagorean operator, an activation function layer, and an add layer. The output of the batch_normalization layer is the input of the Pythagorean operator, the output of the Pythagorean operator is the input of the lnabsa1 activation function layer, and the output of the lnabsa1 activation function layer and the input of the batch_normalization layer are used together as the input of the add layer, thus constructing a Pythagorean neuron as shown in (a). Figure 4 The Pythagorean neuron shown in (a) has residual connections and activation functions, and is used as a replacement neuron.

[0067] Combination Figure 4 As shown in (b), taking the zscore operator layer as the batch_normalization layer and the replacement operator as the harmonic operator as an example, the harmonic neuron with residual connections and activation functions includes a batch_normalization layer, a harmonic operator, an activation function layer, and an add layer. The output of the batch_normalization layer is the input of the harmonic operator, the output of the harmonic operator is the input of the activation function layer, and the output of the activation function layer and the input of the batch_normalization layer are used together as the input of the add layer, thus constructing a... Figure 4 (b) shows a harmonic neuron with residual connections and activation functions, and this harmonic neuron is used as a replacement neuron.

[0068] Step S204: Based on multiple replacement neurons and the third structural block, an improved network structure is obtained.

[0069] In some embodiments of this application, the third structural block may be a portion of the network structure that is not connected to the first structural block. Based on the logical timing of the network structure in the model, multiple replacement neurons can be connected in series to construct multiple layers of deep hidden layers, for example... Figure 5 The structure shown in (a) is an example. Multiple replacement neurons can also be connected in parallel and then aggregated, i.e., a multi-head mechanism, such as... Figure 5 The structure shown in (b) is further... Figure 5 The sum shown in (b) can be a third structural block.

[0070] By employing the aforementioned substitution matrix multiplication operator, the improved network structure is constructed from multiple substitution neurons and a third structural block, reducing the hardware cost during model runtime. The improved network structure includes a data processing structure and a data application structure. The data processing structure preprocesses the input data to the model, while the data application structure includes at least one substitution neuron for performing data inference. Since the substitution neurons can be Pythagorean neurons and / or harmonic neurons, the data application structure can include at least one Pythagorean neuron and / or at least one harmonic neuron.

[0071] To better understand the patch design of circuits (such as the arrangement and routing of computing units) involved in the operation of the original and improved network structures on electronic devices, the following section combines... Figure 6 (a) and Figure 6 The design of the facet in (b) is described.

[0072] like Figure 6 The patch design structure shown in (a) is configured to run the network structure before the improvement, as follows: Figure 6 The patch design structure shown in (b) is configured to run the improved network structure. From the processor clock cycle axis of the electronic device, compared to... Figure 6 The patch design structure shown in (a) is as follows: Figure 6 (b) corresponds to fewer control circuits and is designed as a more regular serial pipeline structure in terms of hardware spatial structure. The integrated circuit area is significantly reduced, which means that the cost of hardware design is reduced and the power consumption during model operation is lowered.

[0073] Step S205: In response to the request to process the input data, the data processing structure is invoked to perform Zernike moment transformation on the input data to obtain the one-dimensional tensor corresponding to the input data, and a two-dimensional tensor is constructed based on the one-dimensional tensor.

[0074] In some embodiments of this application, a user can input data to be processed on an electronic device. The input data can be one or more of image data, text data, video data, and audio data.

[0075] In response to a request to process input data, an electronic device invokes a data processing structure to preprocess the input data, such as performing a Zernike moment transform on the input data.

[0076] Taking image data as the input data as an example, the image data is segmented using a preset sliding window to obtain multiple window regions. An order selection strategy is determined based on the number of window regions, and a first transformation order corresponding to the Zernike moment transform is generated according to the order selection strategy; for example, n=5. Based on the first transformation order, multiple Zernike moment transforms are performed on the multiple window regions to generate a one-dimensional tensor corresponding to the image data.

[0077] In one example, the first transformation order can be a dynamically changing parameter. Multiple Zernike moment transformations are performed on multiple window regions to generate a one-dimensional tensor corresponding to the image data. For example, the image data includes 28... 28-pixel grayscale image, with 4 The sliding window specification of 4 is divided into Each grid, both Taking the first transformation order n=5 as an example, (4) 4) 7 Each of the 7 tensor matrices is 4 Performing a Zernike transformation of order n=5 using a 4-window method yields... .Will Each tensor matrix Performing a 5th-order Zernike transformation yields 21. 21. (The last part, "21," appears to be a fragment and doesn't translate directly. It's likely A tensor matrix of order n=10 is obtained by performing a transformation on it. That is, a one-dimensional tensor with a dimension length of i_W61.

[0078] In another example, based on a first transform order, multiple Zernike moment transformations are performed on multiple window regions to generate a one-dimensional tensor corresponding to the image data. This includes: performing a Zernike moment transformation on each window region based on the first transform order to generate a one-dimensional tensor corresponding to each window region; generating a first intermediate tensor based on the one-dimensional tensor corresponding to each window region; performing a Zernike moment transformation on each slice of the first intermediate tensor based on the first transform order to generate a one-dimensional tensor corresponding to each slice; generating a second intermediate tensor based on the one-dimensional tensor corresponding to each slice; and performing a Zernike moment transformation on the second intermediate tensor based on the first transform order to generate a one-dimensional tensor corresponding to the image data.

[0079] In one example, the image data includes 28 28-pixel grayscale image, with 4 The 4-inch sliding window specification is divided into 7. 7 grids, i.e. (4 4) 7 7. Perform an n=3 order Zernike moment transform on each window region to generate a dimension of (n+1). The one-dimensional tensor (n+2) / 2=10 is denoted as the one-dimensional tensor corresponding to each window region. The one-dimensional tensors corresponding to each window region are aggregated to obtain the first intermediate tensor, denoted as [4][4]

[10] . A Zernike moment transformation is performed on each slice in the first intermediate tensor, that is, an n=3 order Zernike transformation is performed on each [4][4][1]->[4][4] slice to generate the one-dimensional tensor corresponding to each slice, denoted as (n+1). (n+2) / 2=10.

[0080] The one-dimensional tensors corresponding to each slice are aggregated to generate a second intermediate tensor, denoted as

[10]

[10] . An n=4 order Zernike transform is performed on the second intermediate tensor to generate a one-dimensional tensor corresponding to the image data, denoted as (n+1). (n+2) / 2=15 or

[15] .

[0081] Taking text data as the input data as an example, the text data is segmented and transformed into tensors to generate a sequence tensor. The order selection strategy is determined based on the number of segments corresponding to the sequence tensor. Based on the order selection strategy, a second transformation order corresponding to the Zernike moment transform is generated. The second transformation order includes the first and second orders, with the first order being greater than the second. The sequence tensor is then subjected to a Zernike moment transform based on the first order to generate a transformed tensor. Finally, the transformed tensor is subjected to a Zernike moment transform based on the second order to generate a one-dimensional tensor corresponding to the text data.

[0082] For example, seq is obtained in advance according to methods such as BPE. embedding=1024 A 1024-dimensional token sequence tensor is denoted as the intermediate dataset. Each row of this intermediate dataset represents a sequence number (seq). Based on the sequence tensor, for the m-th token, mask[m : 1024][ : ] is used, retaining the element values ​​of tokens in the range [1 : m - 1][ : ] as model input, and using [m : m][ : ] as model output values ​​to recreate the dataset. This recreated dataset is denoted as the sequence tensor.

[0083] Each row of the sequence tensor represents a token of a sequence number (seq). The input tensor has a size of

[1024]

[1024] , and the output tensor has a size of

[1024] . Perform a Zernike transformation or Zernike grouping transformation on the input tensor (i.e., the sequence tensor)

[1024]

[1024] : (32 32) (32 32), perform two first-order transformations (n1=14) to generate a transformation tensor, denoted as ((n+1)). (n+2) / 2)^2=120 120; Perform another second-order transformation (n² = 63) (n² + 1). (n²+2) / 2=2080, finally obtaining the one-dimensional tensor corresponding to the text data, with a dimension length of 2080.

[0084] Where, for any m For an m-dot lattice region, to ensure fitting accuracy, the theoretically required Zenk transform is proportional to m^2; this is consistent with the complexity of convolution (equivalent to im2col matrix multiplication). However, the Zenk transform order can be flexibly adjusted down to m based on sampling density, signal-to-noise ratio, and lattice physical properties. Therefore, the order of the Zernike transform can be determined using one or more of the following strategies: optical wavefront aberration analysis, image rectangular description, adaptive optics real-time correction, and progressive verification.

[0085] For optical wavefront aberration analysis (such as Zemax) strategies: typically 15 to 20 terms (order n=5 to 6) are needed to cover common low-order aberrations (defocus, coma, astigmatism, etc.); freeform surfaces or high-precision systems can set 25 to 35 terms (n≈7–10); more than 40 to 50 terms are easily dominated by noise and need to be confirmed by residuals or cross-validation.

[0086] For rectangular image representation strategies, orders n=7 to 15 typically balance rotation invariance with detail representation; n>20 has weak anti-noise capabilities and should only be used cautiously in high-resolution / low-noise images.

[0087] For real-time adaptive optics (AO) correction strategies, due to the limited number of actuators, orders less than or equal to 10 to 15 (corresponding to approximately 15 to 30 Zernike terms) are often used; atmospheric turbulence modeling commonly uses n=10 (36 terms) or n=15 (66 terms).

[0088] An incremental verification strategy is employed, starting with n=5 (15 terms) or n=6 (21 terms), gradually increasing until the RMS residuals no longer show a significant decrease (e.g., <5% improvement), or constrained by the physical system's degrees of freedom (e.g., the number of mirror partitions).

[0089] Convert a one-dimensional tensor into a two-dimensional tensor, where the first dimension of the two-dimensional tensor can be a batch.

[0090] Step S206: Call the data application structure to process the two-dimensional tensor and output the processing result corresponding to the input data.

[0091] In some embodiments of this application, the data application structure includes multiple Pythagorean neurons. Based on the temporal logic of the data application structure, the two-dimensional tensor is processed to obtain intermediate results, which serve as the input to any Pythagorean neuron. The control chip of the integrated circuit can call one or more of the adders, shifters, logic units, registers, and XOR gates associated with the Pythagorean neurons to process the intermediate results. Based on the processing result of any Pythagorean neuron, the system continues to operate according to the temporal logic of the data application structure to obtain the processing result of the input data.

[0092] In one example, taking image data as input data and Pythagorean neurons as replacement neurons, the data application structure consists of five layers of Pythagorean neurons connected in series, with the first layer having a weight parameter of w_H. w_W=61 80, the weight parameters for the second, third, and fourth layers are w_H w_W=80 80, the weight parameter of the fifth layer is w_H w_W=80 10. The total number of weight parameters is 24880. Among them, the purpose of w_H and w_W being slightly larger than the one-dimensional tensor length i_W=61 is to redundancy the column rank of the second parameter of the matrix Pythagorean theorem, that is, the column rank of the weight parameter matrix. The redundancy factor can be 2 times, 4 times, etc., to compensate for the loss of accuracy and suppress noise in the one-dimensional tensor processing.

[0093] In one example, taking text data as the input, the data application structure can be composed of 64 layers of Pythagorean neurons connected in series, with the weight parameter w_H of the hidden layers (layers 2 to 63) as follows. w_W=4096 4096. First layer weight parameters w_H w_W=2080 4096, the last layer weight parameter w_H w_W=4096 1024, total weight parameters 1,052,901,376.

[0094] The two-dimensional tensor is input into the first layer of Pythagorean neurons for processing. The control chip of the integrated circuit can call one or more of the adders, shifters, logic units, registers, and XOR gates associated with the Pythagorean neurons to process the two-dimensional tensor, thereby obtaining the processing result corresponding to the input data.

[0095] To better understand the differences between matrix multiplication operators and Pythagorean operators in terms of operations, the following description uses specific formulas as examples.

[0096] Let A be the two-dimensional tensor generated in step S206 above, and let B be the transpose of A. When matrix multiplication operators are used in the operation, their specifications are m respectively. p, p n, , Then the result of the matrix multiplication operator corresponding to A and B is , The derivative of the matrix multiplication operator is: , The result of matrix multiplication is y=f(x,w)=x. w can be represented graphically as follows: Figure 7 The diagram shown is shown in the image.

[0097] When performing operations using the Pythagorean theorem on matrices A and B, the formula can be expressed as: , where q represents a real hyperparameter that controls the exponentiation.

[0098] Calculation results corresponding to the Pythagorean operator Represented as a diagram Figure 8 The diagram shown is shown in the image.

[0099] At this point, the derivative formula for the Pythagorean theorem is: ; ; As an optional implementation, if q=1, then: , ; ; ; ; ; As can be seen from the above formulas and the diagram of the calculation results, the Pythagorean theorem and its derivative operator can be used to perform summation and square root operations in high-complexity computation stages. Summation, square root, and reciprocal operations This can reduce the hardware design cost of integrated circuits.

[0100] When performing operations on matrices A and B using the harmonic operator, the formula is as follows: ; ; ; ; ; ; The computational results of the harmonic operator can be expressed as follows: Figure 9 The diagram shown is shown in the image.

[0101] The harmonic operator utilizes summation followed by natural logarithm calculation in high-complexity computation stages. Summation and reciprocal operations This can reduce the design scale of integrated circuits, reduce hardware costs, and improve the computational efficiency of models.

[0102] Through the above embodiments, the network structure of the model deployed in the integrated circuit of the electronic device is analyzed so that the network structure can be improved in the future, thereby reducing the operating cost of the model. The matrix multiplication operator in the first structural block is replaced by a preset replacement operator to obtain an improved first structural block. Based on the improved first and second structural blocks, multiple replacement neurons are constructed to improve the network structure. Based on the multiple replacement neurons and the third structural block, an improved network structure is obtained, which includes a data processing structure and a data application structure. In response to the processing request of input data, the data processing structure is invoked to perform a Zernike moment transformation on the input data, thereby handling input data with different dimensional features. A one-dimensional tensor obtained after the Zernike moment transformation is obtained, and a two-dimensional tensor is constructed based on the one-dimensional tensor. The data application structure is invoked to process the two-dimensional tensor, and the processing structure corresponding to the input data is output, thereby improving the model's processing efficiency for input data while reducing hardware costs.

[0103] Furthermore, by using Pythagorean operators or harmonic operators for substitution, the network structure of the integrated circuit exhibits time-separable characteristics and circuit structure regularity. For example, Figure 6 As shown in (b), the improved network architecture compromises latency and reduces the design limitations of the electronic device's memory subsystem by using a pipelined architecture or a fusion framework that combines pipelines and parallels. This allows for the allocation of on-chip area and gate circuit design space to the SRAM buffer structure, thereby increasing the clock frequency, bandwidth, reducing the clock cycles occupied by the cache and leakage current overhead, and reducing power consumption in the electronic device.

[0104] The integrated circuit in this application embodiment can implement: a cordic multiplication-free circuit + pipeline + "matrix c = F(a,b) accumulation", denoted as C=G(A, B), where G(A, B) = .

[0105] If the matrix multiplication operator is used, then F(a,b) = a b; If the Pythagorean theorem is used, then If the harmonic operator is used, then It is understood that the embodiments of this application achieve temporal separability of the model through algorithmic accumulation and enable it to run on a regular circuit structure, thereby reducing on-chip area, power consumption, and hardware cost. In contrast, the matrix multiplication operators used in the prior art cannot achieve temporal separability or run on a regular circuit structure.

[0106] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed, the method implemented can refer to the methods in the above embodiments of this application.

[0107] The computer-readable storage medium can be the internal memory of the electronic device described in the above embodiments, such as the hard disk or memory of the electronic device. Alternatively, the computer-readable storage medium can be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the electronic device.

[0108] In some embodiments, a computer-readable storage medium may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of the electronic device, etc.

[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0110] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method based on a reconstruction model, applied to an electronic device, the electronic device including an integrated circuit, characterized in that, The method includes: The network structure of the model deployed in the integrated circuit is analyzed. The network structure includes a first structural block, a second structural block, and a third structural block. The first structural block includes a matrix multiplication operator. The matrix multiplication operator in the first structural block is replaced by a preset replacement operator to obtain an improved first structural block; Based on the improved first and second structural blocks, multiple replacement neurons are constructed; Based on the multiple replacement neurons and the third structural block, an improved network structure is obtained, which includes a data processing structure and a data application structure. In response to a request to process input data, the data processing structure is invoked to perform a Zernike moment transformation on the input data to obtain a one-dimensional tensor corresponding to the input data, and a two-dimensional tensor is constructed based on the one-dimensional tensor. The data application structure is invoked to process the two-dimensional tensor, and the processing result corresponding to the input data is output.

2. The data processing method based on the reconstruction model according to claim 1, characterized in that, The replacement operator includes a Pythagorean operator or a harmonic operator. The second structural block includes a layer of unconnected z-score operators and activation function operators. Based on the improved first and second structural blocks, multiple replacement neurons are constructed, including: Delete the activation function operator; and Connect the zscore operator layer to the Pythagorean operator in the improved first structural block, and use the output of the zscore operator layer as the input of the Pythagorean operator in the improved first structural block to construct at least one Pythagorean neuron, which is then used as the replacement neuron; or... Connect the zscore operator layer to the harmonic operator in the improved first structural block, and use the output of the zscore operator layer as the input of the harmonic operator in the improved first structural block to construct at least one harmonic neuron, which is then used as the replacement neuron.

3. The data processing method based on the reconstruction model according to claim 1, characterized in that, The replacement operator includes a Pythagorean operator or a harmonic operator. The second structural block includes a non-connected z-score operator layer and an activation function layer. Based on the improved first structural block and the second structural block, multiple replacement neurons are constructed, including: The Pythagorean operator in the improved first structural block is connected to the zscore operator layer and the activation function layer respectively to construct at least one Pythagorean neuron, which is used as the replacement neuron; wherein the input of the Pythagorean operator in the improved first structural block is the output of the zscore operator layer, and the output of the Pythagorean operator in the improved first structural block is the input of the activation function layer; or... The harmonic operator in the improved first structural block is connected to the zscore operator layer and the activation function layer respectively to construct at least one harmonic neuron, and the adjustment neuron is used as the replacement neuron; wherein, the input of the harmonic operator in the improved first structural block is the output of the zscore operator layer, and the output of the harmonic operator in the improved first structural block is the input of the activation function layer.

4. The data processing method based on the reconstruction model according to claim 1, characterized in that, The process of calling the data application structure to process the two-dimensional tensor and outputting the processing result corresponding to the input data includes: Based on the temporal logic of the data application structure, the two-dimensional tensor is processed to obtain intermediate results; If the intermediate result is the input of any replacement neuron in the data application structure, one or more of the adder, shifter, logic unit, register, and XOR gate associated with the replacement neuron are invoked, and one or more of the adder, shifter, logic unit, register, and XOR gate are integrated on the integrated circuit. The intermediate result is input into any of the replacement neurons, and one or more of the adder, the shifter, the logic unit, the register, and the XOR gate are used to process the intermediate result to obtain the processing result of any of the replacement neurons; Based on the processing result of any of the replaced neurons, the processing result corresponding to the input data is determined.

5. The data processing method based on the reconstruction model according to claim 1, characterized in that, The step of calling the data processing structure to perform a Zernike moment transform on the input data to obtain a one-dimensional tensor corresponding to the input data includes: When the input data is image data, the image data is segmented based on a preset sliding window to obtain multiple window regions; The order selection strategy is determined based on the number of the multiple window regions; The first transformation order corresponding to the Zernike moment transform is generated based on the order selection strategy; Based on the first transformation order, multiple Zernike moment transformations are performed on the multiple window regions to generate a one-dimensional tensor corresponding to the image data.

6. The data processing method based on the reconstruction model according to claim 5, characterized in that, The step of performing multiple Zernike moment transformations on the multiple window regions based on the first transformation order to generate a one-dimensional tensor corresponding to the image data includes: Based on the first transformation order, perform Zernike moment transformation on each window region to generate a one-dimensional tensor corresponding to each window region; Based on the one-dimensional tensor corresponding to each window region, a first intermediate tensor is generated; Based on the first transformation order, perform Zernike moment transformation on each slice of the first intermediate tensor to generate a one-dimensional tensor corresponding to each slice; Based on the one-dimensional tensor corresponding to each slice, a second intermediate tensor is generated; Based on the first transformation order, the second intermediate tensor is subjected to a Zernike moment transformation to generate a one-dimensional tensor corresponding to the image data.

7. The data processing method based on the reconstruction model according to claim 1, characterized in that, The step of performing multiple Zernike moment transformations on the multiple window regions based on the first transformation order to generate a one-dimensional tensor corresponding to the image data includes: When the input data is text data, the text data is segmented and transformed into tensors to generate sequence tensors; The order selection strategy is determined based on the number of segments corresponding to the sequence tensor. The second transformation order corresponding to the Zernike moment transform is generated based on the order selection strategy. The second transformation order includes a first order and a second order, wherein the first order is greater than the second order. Based on the first order, a Zernike moment transform is performed on the sequence tensor to generate a transformed tensor; Based on the second order, a Zernike moment transform is performed on the transformation tensor to generate a one-dimensional tensor corresponding to the text data.

8. The data processing method based on the reconstruction model according to claim 5 or 7, characterized in that, The order selection strategy includes one or more of the following: optical wavefront aberration analysis strategy, image rectangular description strategy, adaptive optics real-time correction strategy, and progressive verification strategy.

9. An electronic device, characterized in that, The electronic device includes an integrated circuit and a memory. The integrated circuit includes a control chip. The memory stores a computer program. When the control chip executes the computer program, it implements the data processing method based on the reconstruction model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the data processing method based on the reconstruction model as described in any one of claims 1 to 8.

Images