Model inference acceleration method, system, electronic device, and storage medium

Abstract

The present disclosure provides a model inference acceleration method, system, electronic device and storage medium, relating to the fields of hardware acceleration, optoelectronic hybrid computing and model inference. The method can include: obtaining an input sequence, and dividing the input sequence into input blocks of a predetermined size; for each input block, the following processing is performed respectively: generating a tunable metasurface control state corresponding to the input block, adjusting the optical response characteristics of a tunable metasurface optical convolution module according to the tunable metasurface control state, and generating a first modulated light field carrying input block data; generating a second modulated light field using the adjusted tunable metasurface optical convolution module according to the first modulated light field, generating a model inference output corresponding to the input block based on the second modulated light field; and combining the model inference outputs corresponding to each input block to generate a target inference result corresponding to the input sequence. The present disclosure can improve the model inference speed, etc.

Description

Technical Field

[0001] This disclosure relates to the fields of hardware acceleration, optoelectronic hybrid computing and model inference, and in particular to model inference acceleration methods, systems, electronic devices and storage media. Background Technology

[0002] In current graphics processing units (GPUs) and artificial intelligence (AI) accelerators, the bottleneck of input-dependent state-space models, represented by Mamba, lies not only in multiply-accumulate operations, but also in data transfer between high-bandwidth memory (HBM) and on-chip static random access memory (SRAM) / cache. A large amount of data transfer can severely affect the inference speed of the model. Summary of the Invention

[0003] This disclosure provides methods, systems, electronic devices, and storage media for accelerating model inference.

[0004] A method for accelerating model inference, comprising:

[0005] Obtain the input sequence and divide the input sequence into input blocks of a predetermined size; For each input block, the following processing is performed: generating a tunable metasurface control state corresponding to the input block; adjusting the optical response characteristics of the tunable metasurface optical convolution module according to the tunable metasurface control state; and generating a first modulation light field carrying the input block data; generating a second modulation light field using the adjusted tunable metasurface optical convolution module based on the first modulation light field; and generating the model inference output corresponding to the input block using an input-dependent state space model based on the second modulation light field. The target inference result corresponding to the input sequence is generated by combining the model inference output corresponding to each input block.

[0006] A model inference acceleration system includes: a digital subsystem, an optical subsystem, a back-end subsystem, and a spatial light modulator; The digital subsystem is used to divide the input sequence into input blocks of a predetermined size, and for each input block, perform the following processing: generate an adjustable metasurface control state corresponding to the input block, adjust the optical response characteristics of the adjustable metasurface optical convolution module according to the adjustable metasurface control state, and send the input block to the spatial light modulator; The spatial light modulator is used to generate a first modulated light field carrying the input block data; The optical subsystem is used to generate a second modulated light field based on the first modulated light field using the adjusted tunable metasurface optical convolution module, wherein the tunable metasurface optical convolution module is located in the optical subsystem. The back-end subsystem is used to generate the model inference output corresponding to the input block based on the second modulated optical field using the input-dependent state space model, and to generate the target inference result corresponding to the input sequence by combining the model inference output corresponding to each input block.

[0007] An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of an embodiment of the model inference acceleration method described in this disclosure; Figure 2 This is a flowchart of an embodiment of the method for generating tunable metasurface control states as described in this disclosure; Figure 3 This is a flowchart of an embodiment of the method for generating model inference output corresponding to any input block as described in this disclosure; Figure 4 This is a schematic diagram of the composition structure of Embodiment 400 of the model inference acceleration system described in this disclosure; Figure 5 This is a schematic diagram of the composition structure of the digital subsystem 401 described in this disclosure; Figure 6 This is a schematic diagram of the composition and structure of the optical subsystem 402 described in this disclosure; Figure 7 This is a schematic diagram of the composition of the rear terminal system 403 described in this disclosure; Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0013] Figure 1 This is a flowchart illustrating an embodiment of the model inference acceleration method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0014] In step 101, the input sequence is obtained and divided into input blocks of a predetermined size.

[0015] In step 102, for each input block, the following processing is performed: generating the tunable metasurface control state corresponding to the input block based on the input block; adjusting the optical response characteristics of the tunable metasurface optical convolution module according to the tunable metasurface control state; and generating a first modulation light field carrying the input block data; generating a second modulation light field using the adjusted tunable metasurface optical convolution module based on the first modulation light field; and generating the model inference output corresponding to the input block using the input-dependent state space model based on the second modulation light field.

[0016] In step 103, the target inference result corresponding to the input sequence is generated by combining the model inference output corresponding to each input block.

[0017] By employing the scheme described in the above method embodiments, the long-range convolution (long-range state space convolution) computation of the input-dependent state space model can be migrated from the digital domain to the optical domain. Utilizing a dynamically adjustable metasurface for each input block, convolution operations are performed in parallel in the optical domain through physical propagation and frequency modulation. This allows for the parallel convolution of all tokens within the same input block with a fixed optical response, effectively bypassing the reliance on repeated read-write operations to the HBM for time-step recursion of long sequences in purely digital implementations. The computation mode is transformed from "repeated access to the HBM by the digital arithmetic logic unit (ALU)" to "optical propagation + metasurface modulation + analog-to-digital converter (ADC) readout," significantly reducing memory access bandwidth requirements and the computational load of digital multiply-accumulate (MAC) operations, thereby improving model inference speed and energy efficiency.

[0018] Considering the hardware limitations of real tunable metasurfaces, such as refresh rate and control precision, the solution described in this disclosure adopts a block-level input-dependent processing approach. That is, the input sequence is divided into several input blocks of a predetermined size. The specific value of the predetermined size can be determined according to actual needs, such as 128-1024 tokens. Each input block can use on-chip SRAM as its storage medium; block reads and writes are performed on-chip and do not occupy HBM.

[0019] Compared to the traditional method of token-by-token iteration, the processing method disclosed herein can significantly reduce the number of HBM accesses.

[0020] Furthermore, each input block can be processed sequentially according to its appearance order in the input sequence. For each input block, a corresponding tunable metasurface control state can be generated first, and the optical response characteristics of the tunable metasurface optical convolution module can be adjusted based on the tunable metasurface control state.

[0021] In some embodiments of this disclosure, the method for generating the tunable metasurface control state corresponding to the input block based on the input block may include: generating State Space Model (SSM) parameters corresponding to the input block using a learnable network module, generating a dynamic convolution kernel based on the SSM parameters, generating a frequency domain transfer function based on the dynamic convolution kernel, and then generating the tunable metasurface control state based on the frequency domain transfer function. Correspondingly, the tunable metasurface control state may also be converted into an analog driving signal, and the analog driving signal may be used to drive the optical response characteristics of the tunable metasurface optical convolution module for adjustment.

[0022] The tunable metasurface control state refers to the set of physical state parameters used to configure each metasurface unit in the tunable metasurface optical convolution module. The physical state parameters may include at least one of the following: metasurface unit phase, metasurface unit amplitude, polarization modulation state, transmittance, reflectance, phase change material state, liquid crystal orientation state, micro-electro-mechanical system (MEMS) microstructure displacement, electro-optic modulation voltage, and combination state between multilayer metasurfaces.

[0023] Adjusting the optical response characteristics of the tunable metasurface optical convolution module according to the control state of the tunable metasurface specifically includes: converting the control state of the tunable metasurface into an analog driving signal applied to each metasurface unit of the tunable metasurface; the analog driving signal changes the physical state of each metasurface unit (including unit phase and / or unit amplitude) so that the transfer function of the tunable metasurface optical convolution module in the frequency domain plane matches the frequency domain transfer function corresponding to the current input block.

[0024] In the scheme described in this disclosure, the learnable network module in the electronic control module can generate the corresponding SSM parameters based on the current input block through a preset mapping function. The mapping function can be implemented by a lightweight neural network, which includes at least linear layers and nonlinear activation layers, and its network parameters can be pre-trained.

[0025] For example, a lightweight neural network can generate a set of SSM parameters for any input block Xb, which may include: Δb, Bb, and Cb, i.e.: Δb, Bb and Cb=Ψ(Xb); (1) Where Δb represents the discretization step size parameter, Bb represents the input projection matrix, Cb represents the output projection matrix, and Ψ represents the mapping function. Δb, Bb, and Cb all have input-dependent characteristics and are dynamically generated according to different input blocks.

[0026] Alternatively, the learnable network module can also be implemented using a single-layer linear projection, i.e.: [Δb; Bb; Cb]=W·Xb'+b; (2) Where Xb' represents the feature vector after global average pooling of the input block Xb, and W and b are trainable parameters.

[0027] For scenarios with extremely limited hardware resources, the learnable network module can also be implemented using a lookup table approach: multiple sets of typical inputs' corresponding SSM parameters are pre-stored, and during inference, interpolation is performed based on the similarity between the current input block and each typical input to determine the SSM parameters corresponding to the current input block. This method eliminates the need for online neural network computation, further reducing the power consumption and latency of parameter generation.

[0028] Dynamic convolutional kernels can be generated based on the SSM parameters. In some embodiments of this disclosure, the SSM parameters can be locally shared or subjected to low-rank transformation processing, and then dynamic convolutional kernels can be generated based on the processing results.

[0029] Local sharing processing can refer to sharing Δb as a single value for all time steps within the same input block, or sharing Bb and Cb of adjacent time steps within the same input block by channel grouping.

[0030] Low-rank transformation processing can refer to decomposing Bb and Cb into the product of two smaller matrices to reduce the degrees of freedom of the output parameters.

[0031] When generating a dynamic convolution kernel, the discrete state transition matrix can be obtained by discretization calculation based on the three SSM parameters (Δb, Bb and Cb) corresponding to the current input block. This matrix can be obtained by multiplying Δb with a pre-set fixed state transition matrix and taking the exponent.

[0032] Then, for the length L of the current input block, the dynamic convolution kernel sequence can be generated as follows: the i-th coefficient of the dynamic convolution kernel is obtained by multiplying Cb, the i-th power of the discrete state transition matrix, and the transformation matrix in sequence, where i takes the values ​​0, 1, ..., L-1. The transformation matrix can be calculated as follows: first, subtract the predetermined identity matrix from the discrete state transition matrix, then multiply it by Δb and Bb to obtain an intermediate result, and finally, multiply this intermediate result by the inverse of the discrete state transition matrix to obtain the transformation matrix.

[0033] The sequence formed by concatenating the 0th coefficient, 1st coefficient, ..., L-1th coefficient of the dynamic convolution kernel can be used as the dynamic convolution kernel corresponding to the current input block.

[0034] Based on the dynamic convolution kernel, a frequency domain transfer function can also be generated. For example, a Fourier transform can be performed on the dynamic convolution kernel to convert the time-domain kernel coefficient sequence into a frequency-domain response, thus obtaining the frequency-domain transfer function corresponding to the input block.

[0035] That is, we can have: Hb(ω)=F[Kb];(3) Where Kb represents the dynamic convolution kernel, Hb(ω) represents the frequency domain transfer function, and F represents the Fourier transform. This frequency domain transfer function is determined by the input block, i.e.: Hb(ω)=Hb(ω,Xb;(4) Where Xb represents the input block.

[0036] To reduce the complexity of metasurface control, the dynamic convolution kernel can be decomposed into a weighted combination of a fixed base kernel and R tunable base kernels, and a frequency domain transfer function can be generated based on the weighted combination, where R is a positive integer.

[0037] Specifically, the dynamic convolution kernel corresponding to the current input block is equal to the sum of the fixed base kernel plus R adjustable base kernels multiplied by their respective combination coefficients. The fixed base kernel is always present in the combination, with a weight of 1. The weights of each adjustable base kernel are dynamically controlled by its corresponding combination coefficients, which are generated based on the current input block.

[0038] Correspondingly, the frequency domain transfer function is equal to the frequency domain response of the fixed base core plus the sum of the frequency domain responses of the R tunable base cores multiplied by their respective combination coefficients. The frequency domain response of the fixed base core is obtained by Fourier transforming the fixed base core, and the frequency domain response of each tunable base core is obtained by Fourier transforming the corresponding tunable base core.

[0039] Through the above decomposition method, the system only needs to generate R combination coefficients to achieve dynamic adjustment of the frequency domain transfer function, without having to generate the complete frequency domain transfer function. This reduces the complexity of parameter generation and metasurface control. At the same time, the combination coefficients are dynamically generated based on the input block, ensuring that the core combination result still has input-dependent characteristics.

[0040] In practical applications, fixed and / or tunable base cores are selected from a pre-constructed frequency domain response codebook. Specifically, the codebook stores M sets of preset base cores and their corresponding frequency domain responses. By selecting one or more base cores from the codebook as fixed and / or tunable base cores, the base cores used in the base core combination are physically realizable and do not require online calibration.

[0041] Furthermore, the frequency domain transfer function can be compiled into a tunable metasurface control state, resulting in: Θb=C(Hb(ω));(5) Θb=Θb(Xb); (6) Where C represents the compilation mapping function, which is used to map the amplitude and / or phase response of the frequency domain transfer function Hb(ω) to the physical control state Θb that can be executed by each unit of the tunable metasurface, where Θb represents the control state of the tunable metasurface.

[0042] As can be seen, the scheme described in this disclosure adopts a block-level dynamic reconstruction method, that is, sharing the metasurface configuration within the same input block, and updating the configuration between blocks according to the input content, thereby reducing the metasurface refresh frequency, reducing the complexity of the control circuit, and adapting to the block-based inference of Mamba-type models while retaining the input dependency selectivity, thus having higher engineering feasibility, etc.

[0043] In some embodiments of this disclosure, a pre-constructed frequency domain response codebook can also be obtained. The frequency domain response codebook may include M sets of metasurface standard control states and reference frequency domain responses corresponding to each metasurface standard control state, where M is a positive integer greater than 1. Then, N reference frequency domain responses that match the frequency domain transfer function can be determined from each reference frequency domain response. When N is 1, the metasurface standard control state corresponding to the reference frequency domain response can be determined as an adjustable metasurface control state. When N is greater than 1, the metasurface standard control states corresponding to each reference frequency domain response can be combined according to the corresponding weights, and the adjustable metasurface control state can be determined based on the combination result.

[0044] The frequency domain response codebook can be pre-constructed, and may include M sets of metasurface standard control states and their corresponding reference frequency domain responses. The value of M can be flexibly set according to actual factors such as the adjustable range of the metasurface, control accuracy, and hardware resources. When an input block arrives, the generated frequency domain transfer function can be matched with each reference frequency domain response in the frequency domain response codebook. For example, the similarity or distance metric between the two can be calculated. Then, N reference frequency domain responses that meet the preset matching conditions are selected, where N is a positive integer less than or equal to M. Meeting the preset conditions can be greater than a predetermined threshold or being in the top N positions after sorting. If N equals 1, the metasurface standard control state corresponding to the reference frequency domain response can be directly determined as the adjustable metasurface control state corresponding to the current input block. This is a hard selection method, suitable for scenarios with limited control bandwidth and requiring fast response. If N is greater than 1, weights can be assigned to the corresponding multiple sets of metasurface standard control states according to the matching degree of each reference frequency domain response, and a weighted combination can be performed to generate the final adjustable metasurface control state. This is a soft combination method, suitable for scenarios with high accuracy requirements and allowing for certain computational overhead.

[0045] By adopting the above processing method, the processing complexity can be reduced and the generation speed of tunable metasurface control states can be improved.

[0046] Based on the above introduction, Figure 2 This is a flowchart illustrating an embodiment of the method for generating tunable metasurface control states as described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0047] In step 201, the SSM parameters corresponding to the input block are generated using the learnable network module.

[0048] In step 202, a dynamic convolution kernel is generated based on the SSM parameters.

[0049] In step 203, a frequency domain transfer function is generated based on the dynamic convolution kernel.

[0050] In step 204, the control state of the tunable metasurface is generated based on the frequency domain transfer function.

[0051] Furthermore, the control state of the tunable metasurface can be converted into an analog drive signal using a digital-to-analog converter (DAC). The optical response characteristics of the tunable metasurface optical convolution module can then be adjusted using the analog drive signal, thereby changing its optical response characteristics.

[0052] Additionally, a first modulated optical field carrying the input block data can be generated. For example, the data of the current input block can be loaded into a spatial light modulator, which then modulates the incident light carrier. Specifically, the spatial light modulator can map each data element in the input block to the corresponding pixel position in the spatial light field. By changing the transmittance or reflectance of each pixel, the spatial amplitude and / or phase distribution of the light field carries the data information of the input block. One spatial axis of the spatial light modulator corresponds to the time dimension of the input sequence, and the other spatial axis corresponds to the channel dimension or feature dimension. Alternatively, multi-channel multiplexing can be achieved through wavelength or polarization dimension, thereby transforming a one-dimensional sequence into a two-dimensional spatial light field distribution.

[0053] Based on the first modulated light field, the second modulated light field can be generated using the adjusted tunable metasurface optical convolution module.

[0054] In some embodiments of this disclosure, the first modulation light field can first be subjected to Fourier transform using the first lens to obtain the first transformation result. Then, the first transformation result can be subjected to optical modulation corresponding to the frequency domain transfer function using the adjusted tunable metasurface optical convolution module to obtain the second transformation result. Furthermore, the second transformation result can be subjected to inverse Fourier transform using the second lens to obtain the third transformation result, and the third transformation result can be determined as the second modulation light field.

[0055] The first transformation result obtained after the first modulated light field passes through the first lens can also be called the frequency domain light field. Then, the tunable metasurface optical convolution module located in the Fourier plane can apply optical modulation corresponding to the frequency domain transfer function to obtain the second transformation result. The second transformation result can be transformed by inverse Fourier transform to obtain the third transformation result, which is the required second modulated light field.

[0056] As can be seen, the scheme described in this disclosure implements input-dependent state-space convolution. Since the frequency domain transfer function changes dynamically with the input block rather than being a fixed constant, the convolution result output by the system is represented in the frequency domain as the product of the input light field spectrum and the dynamic frequency domain transfer function. This process is equivalent to dynamic convolution in the time domain, rather than the traditional convolution operation with a fixed kernel.

[0057] Based on the second modulated light field, the model inference output corresponding to the input block can be generated. In some embodiments of this disclosure, the second modulated light field can be photoelectrically converted using a photodetector array to obtain an analog electrical signal. Then, the analog electrical signal can be converted from analog to digital to obtain a digital representation of the optical convolution result corresponding to the input block. Subsequently, the model inference output can be generated using an input-dependent state-space model based on the digital representation.

[0058] In some embodiments of this disclosure, to further enhance the model's ability to reuse and extract features within the same input block, multiple recurrent convolutions can be performed on the same input block. Specifically, for the current input block, after completing a series of optical convolutions, the analog electrical signal output by the photodetector array can be sampled by an ADC and fed back to the spatial light modulator, re-encoded into a light field, and then sent back to the tunable metasurface optical convolution module to perform a second optical convolution. This process can be repeated multiple times. Each cycle can use the same frequency domain transfer function or a different frequency domain transfer function to simulate the effect of multiple convolution kernel stacking in the digital domain. The recurrent convolution result is sent to subsequent processing after the final photoelectric conversion and ADC sampling. This method is suitable for scenarios that require multiple feature extractions from the same input block, such as lightweight recurrent networks or deep feature reuse. It should be noted that recurrent convolution increases inference latency, and the choice should be made based on the accuracy requirements and hardware resources during implementation.

[0059] Accordingly, Figure 3 This is a flowchart illustrating an embodiment of the method for generating model inference output corresponding to any input block as described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0060] In step 301, a first modulated light field carrying input block data is generated using a spatial light modulator.

[0061] In step 302, the first modulation light field is subjected to Fourier transform using the first lens to obtain the first transform result.

[0062] In step 303, the adjusted tunable metasurface optical convolution module is used to apply optical modulation corresponding to the frequency domain transfer function to the first transformation result to obtain the second transformation result.

[0063] In step 304, the second lens is used to perform an inverse Fourier transform on the second transformation result to obtain the third transformation result, namely the second modulated light field.

[0064] In step 305, the second modulated light field is photoelectrically converted using a photodetector array to obtain an analog electrical signal.

[0065] In step 306, the analog electrical signal is converted from analog to digital to obtain a digital representation of the optical convolution result corresponding to the input block.

[0066] For example, an ADC can be used to perform analog-to-digital conversion on analog electrical signals to obtain the digital representation.

[0067] In step 307, the model inference output corresponding to the input block is generated using the input-dependent state space model based on the numerical representation.

[0068] Following the above method, the model inference output corresponding to each input block can be obtained separately. Accordingly, the target inference result corresponding to the input sequence can be generated by combining the model inference output corresponding to each input block. For example, the model inference output corresponding to each input block can be concatenated sequentially according to the position order of each input block in the input sequence to obtain the target inference result.

[0069] As can be seen, the scheme described in this disclosure establishes an optical computing path through digital-to-analog conversion, tunable metasurfaces, and analog-to-digital conversion: digital input is loaded into the optical field via a spatial light modulator, causing the optical field to carry the input block data. The optical field then enters the optical system, where the tunable metasurface applies optical modulation corresponding to the input block in the frequency domain. Long-range convolution is completed through physical propagation and frequency domain modulation. After photoelectric conversion and analog-to-digital conversion, the data returns to the digital domain for post-processing. Under this architecture, the HBM no longer undertakes the intermediate state storage and repeated transport of the complete long-range convolution, but mainly handles input block reading, metasurface configuration parameter reading or generation, analog-to-digital conversion output result writing back, and a small amount of electronic gating and post-processing data access. Furthermore, the scheme described in this disclosure does not completely eliminate digital storage or HBM entirely, but rather reduces the most expensive digital intermediate state transport in long-range convolution, thus alleviating the dependence of pure digital implementations such as medium-to-long sequence scanning on HBM bandwidth and intermediate state read / write operations.

[0070] Furthermore, taking the Mamba-type model as an example, its basic structure includes input gating and a selective state space module. With the scheme described in this disclosure, it can be divided into two parts: optical and electronic. The optical part is responsible for long-range convolution determined by the input, while the electronic part is responsible for operations such as input gating, residual connection, normalization, and channel mixing. The digital representation output by the analog-to-digital conversion is sent to the electronic post-processing module, and the digital-to-analog conversion is responsible for optical field encoding and metasurface driving. This structure avoids the implementation of complex nonlinearity in optical devices while retaining the core selectivity of the Mamba-type model.

[0071] In addition, each channel in the multi-channel / multi-head structure of the Mamba-type model can be independently mapped to a different wavelength or polarization channel, and each channel can be independently dynamic optically convolved using the frequency domain transfer function corresponding to its own input block, thereby realizing parallel optical acceleration of multi-channel SSM.

[0072] In some embodiments of this disclosure, the SSM parameters corresponding to the input block can be generated using a learnable network module. The network parameters of the learnable network module can be obtained through pre-training. Furthermore, during the training process, the non-ideal response model of the tunable metasurface optical convolution module can be introduced into the network forward computation to make the training objective include task loss and hardware non-ideal constraints. The network parameters of the learnable network module can be iteratively optimized according to the training objective.

[0073] The non-ideal response model may include at least one of the following: phase quantization error, amplitude quantization error, optical noise, inter-channel crosstalk, thermal drift, manufacturing error, and quantization error of digital-to-analog conversion and analog-to-digital conversion of the tunable metasurface.

[0074] Multiple training samples can be constructed, each of which can include an input sequence and a true label. The true label can be the actual task processing result corresponding to the input sequence. The corresponding predicted task processing result can be generated based on the input sequence in the training sample, and the predicted task processing result can be determined as the predicted label. Then, the task loss can be determined based on the true label and the predicted label.

[0075] The specific contents of the hardware non-ideality constraints can be determined according to actual needs, and may include one or any combination of the following: Lo, Lq, Ls and La, where Lo represents the error between the ideal convolution and the actual optical response, Lq represents the metasurface phase or amplitude quantization loss, Ls represents the metasurface switching cost between adjacent input blocks, and La represents the digital-to-analog conversion error constraint.

[0076] Assuming that the hardware non-ideal constraints include Lo, Lq, Ls, and La, then Lo, Lq, Ls, La, and the task loss Ltask can be weighted and summed to obtain the desired training objective. The network parameters of the learnable network modules can then be updated based on the training objective.

[0077] That is, we can have: Lz=Ltask+λ1Lo+λ2Lq+λ3Ls+λ4La; (7) Where λ1, λ2, λ3 and λ4 represent weighting coefficients, and the weighting coefficient of Ltask can be considered to be 1.

[0078] By using the above-mentioned hardware constraint-aware training method, the network parameters after training convergence can automatically compensate for the impact of hardware non-ideal factors on inference accuracy when deployed to a real system, without the need to add an additional compensation step during the inference stage.

[0079] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0080] The above is an introduction to the method embodiments. The following system embodiments will further illustrate the solution described in this disclosure.

[0081] Figure 4 This is a schematic diagram of the structural composition of embodiment 400 of the model inference acceleration system described in this disclosure. Figure 4 As shown, it includes: a digital subsystem 401, an optical subsystem 402, a rear subsystem 403, and a spatial light modulator 404.

[0082] The digital subsystem 401 is used to divide the input sequence into input blocks of a predetermined size, and for each input block, perform the following processing: generate an adjustable metasurface control state corresponding to the input block, adjust the optical response characteristics of the adjustable metasurface optical convolution module according to the adjustable metasurface control state, and send the input block to the spatial light modulator 404.

[0083] Spatial light modulator 404 is used to generate a first modulated light field carrying input block data.

[0084] Optical subsystem 402 is used to generate a second modulated light field based on the first modulated light field using an adjusted tunable metasurface optical convolution module, the tunable metasurface optical convolution module being located in optical subsystem 402.

[0085] The rear subsystem 403 is used to generate the model inference output corresponding to the input block based on the second modulated optical field using the input-dependent state space model, and to generate the target inference result corresponding to the input sequence by combining the model inference output corresponding to each input block.

[0086] Figure 5 This is a schematic diagram of the composition structure of the digital subsystem 401 described in this disclosure. Figure 5 As shown, it includes: an input sequence buffer module 4011, a learnable network module 4012, a compilation module 4013, and a driver module 4014.

[0087] The input sequence buffer module 4011 is used to divide the input sequence into input blocks of a predetermined size and send the input blocks to the learnable network module 4012 and the spatial light modulator 404 respectively.

[0088] The learnable network module 4012 is used to generate the SSM parameters corresponding to the input block.

[0089] Compilation module 4013 is used to generate a dynamic convolution kernel based on the SSM parameters, generate a frequency domain transfer function based on the dynamic convolution kernel, and generate an adjustable metasurface control state based on the frequency domain transfer function.

[0090] The driving module 4014 is used to convert the control state of the tunable metasurface into an analog driving signal, and use the analog driving signal to drive the optical response characteristics of the tunable metasurface optical convolution module to adjust.

[0091] Figure 6 This is a schematic diagram of the composition and structure of the optical subsystem 402 described in this disclosure. Figure 6 As shown, it includes: a first lens 4021, an adjustable metasurface optical convolution module 4022, a second lens 4023, and a photodetector array 4024.

[0092] The first lens 4021 is used to perform a Fourier transform on the first modulated light field to obtain the first transform result.

[0093] The adjusted tunable metasurface optical convolution module 4022 is used to apply optical modulation corresponding to the frequency domain transfer function to the first transformation result to obtain the second transformation result.

[0094] The second lens 4023 is used to perform an inverse Fourier transform on the second transformation result to obtain a third transformation result, and the third transformation result is determined as the second modulation optical field.

[0095] The photodetector array 4024 is used to perform photoelectric conversion on the second modulated light field to obtain an analog electrical signal.

[0096] Figure 7 This is a schematic diagram of the composition of the rear terminal system 403 described in this disclosure. Figure 7 As shown, it includes: sampling module 4031 and electronic post-processing module 4032.

[0097] The sampling module 4031 is used to perform analog-to-digital conversion on the analog electrical signal to obtain a digital representation of the optical convolution result corresponding to the input block.

[0098] The electronic post-processing module 4032 is used to generate model inference output based on the digital representation using the input-dependent state space model, and to generate the target inference result corresponding to the input sequence by combining the model inference output corresponding to each input block.

[0099] The specific workflow of the above system embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0100] The input sequences and task processing results in the embodiments described in this disclosure are not targeted at any specific user and are not intended to reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

[0101] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0102] Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0103] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0104] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0111] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for accelerating model inference, characterized in that, include: Obtain the input sequence and divide the input sequence into input blocks of a predetermined size; For each input block, the following processing is performed: generating a tunable metasurface control state corresponding to the input block; adjusting the optical response characteristics of the tunable metasurface optical convolution module according to the tunable metasurface control state; and generating a first modulation light field carrying the input block data; generating a second modulation light field using the adjusted tunable metasurface optical convolution module based on the first modulation light field; and generating the model inference output corresponding to the input block using an input-dependent state space model based on the second modulation light field. The target inference result corresponding to the input sequence is generated by combining the model inference output corresponding to each input block.

2. The method according to claim 1, characterized in that, The process of generating the adjustable metasurface control state corresponding to the input block based on the input block includes: The state-space model parameters corresponding to the input block are generated using a learnable network module; Generate dynamic convolution kernels based on the state space model parameters; A frequency domain transfer function is generated based on the dynamic convolution kernel; The tunable metasurface control state is generated based on the frequency domain transfer function.

3. The method according to claim 2, characterized in that, The step of generating a dynamic convolution kernel based on the state space model parameters includes: The parameters of the state space model are subjected to local sharing processing or low-rank transformation processing, and the dynamic convolution kernel is generated based on the processing result.

4. The method according to claim 2, characterized in that, The step of generating the frequency domain transfer function based on the dynamic convolution kernel includes: The dynamic convolution kernel is decomposed into a weighted combination of a fixed base kernel and R adjustable base kernels, where R is a positive integer; The frequency domain transfer function is generated based on the weighted combination.

5. The method according to claim 2, characterized in that, The step of generating the tunable metasurface control state based on the frequency domain transfer function includes: Obtain a pre-constructed frequency domain response codebook, which includes M sets of metasurface standard control states and reference frequency domain responses corresponding to each metasurface standard control state, where M is a positive integer greater than 1; N reference frequency domain responses that match the frequency domain transfer function are determined from each reference frequency domain response; When N is 1, the standard control state of the metasurface corresponding to the reference frequency domain response is determined as the adjustable metasurface control state. When N is greater than 1, the standard control states of the metasurface corresponding to each reference frequency domain response are combined according to their respective weights, and the adjustable metasurface control state is determined based on the combination result.

6. The method according to claim 2, characterized in that, The step of generating a second modulated light field using the adjusted tunable metasurface optical convolution module based on the first modulated light field includes: The first modulated light field is subjected to a Fourier transform using the first lens to obtain the first transform result; The first transformation result is obtained by applying optical modulation corresponding to the frequency domain transfer function to the first transformation result using the adjusted tunable metasurface optical convolution module; The second transformation result is subjected to an inverse Fourier transform using a second lens to obtain a third transformation result, which is then determined as the second modulated light field.

7. The method according to claim 1, characterized in that, The step of generating the model inference output corresponding to the input block based on the second modulated light field includes: The second modulated light field is photoelectrically converted using a photodetector array to obtain an analog electrical signal; The analog electrical signal is converted from analog to digital to obtain a digital representation of the optical convolution result corresponding to the input block; Based on the numerical representation, the model inference output is generated using the input-dependent state-space model.

8. The method according to claim 2, characterized in that, The network parameters of the learnable network module are obtained through pre-training. During the training process, the non-ideal response model of the tunable metasurface optical convolution module is introduced into the forward computation of the network, so that the training objective includes task loss and hardware non-ideal constraints. The network parameters of the learnable network module are iteratively optimized according to the training objective.

9. A model inference acceleration system, characterized in that, include: Digital subsystem, optical subsystem, back-end subsystem, and spatial light modulator; The digital subsystem is used to divide the input sequence into input blocks of a predetermined size, and for each input block, perform the following processing: generate an adjustable metasurface control state corresponding to the input block, adjust the optical response characteristics of the adjustable metasurface optical convolution module according to the adjustable metasurface control state, and send the input block to the spatial light modulator. The spatial light modulator is used to generate a first modulated light field carrying the input block data; The optical subsystem is used to generate a second modulated light field based on the first modulated light field using the adjusted tunable metasurface optical convolution module, wherein the tunable metasurface optical convolution module is located in the optical subsystem. The back-end subsystem is used to generate the model inference output corresponding to the input block based on the second modulated optical field using the input-dependent state space model, and to generate the target inference result corresponding to the input sequence by combining the model inference output corresponding to each input block.

10. The system according to claim 9, characterized in that, The digital subsystem includes: an input sequence buffer module, a learnable network module, a compilation module, and a driver module; The input sequence caching module is used to divide the input sequence into input blocks of the predetermined size, and send the input blocks to the learnable network module and the spatial light modulator respectively; The learnable network module is used to generate the state space model parameters corresponding to the input block; The compilation module is used to generate a dynamic convolution kernel based on the state space model parameters, generate a frequency domain transfer function based on the dynamic convolution kernel, and generate the tunable metasurface control state based on the frequency domain transfer function. The driving module is used to convert the control state of the tunable metasurface into an analog driving signal, and use the analog driving signal to drive the optical response characteristics of the tunable metasurface optical convolution module to adjust.

11. The system according to claim 10, characterized in that, The optical subsystem also includes: a first lens, a second lens, and a photodetector array; The first lens is used to perform a Fourier transform on the first modulated light field to obtain a first transform result; The adjusted tunable metasurface optical convolution module is used to apply optical modulation corresponding to the frequency domain transfer function to the first transformation result to obtain the second transformation result; The second lens is used to perform an inverse Fourier transform on the second transformation result to obtain a third transformation result, and the third transformation result is determined as the second modulated light field; The photodetector array is used to perform photoelectric conversion on the second modulated light field to obtain an analog electrical signal.

12. The system according to claim 11, characterized in that, The rear terminal system includes: a sampling module and an electronic post-processing module; The sampling module is used to perform analog-to-digital conversion on the analog electrical signal to obtain a digital representation of the optical convolution result corresponding to the input block; The electronic post-processing module is used to generate the model inference output based on the digital representation using the input-dependent state space model, and to generate the target inference result corresponding to the input sequence by combining the model inference output corresponding to each input block.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images