On-chip model solidification system and on-chip model inference method

By employing a hybrid optoelectronic computing architecture, optical inference computation is performed using a pre-cured photon scattering medium, combined with digital back-end processing. This solves the problems of high energy consumption and high latency in large-scale pre-trained models, enabling low-cost and efficient on-chip model deployment.

CN122693743BActive Publication Date: 2026-11-10SHANGHAI NORTH OCEAN TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611176529.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-05
Publication Date
2026-11-10
Estimated Expiration
2046-08-05

AI Technical Summary

Technical Problem

Large-scale pre-trained models have high inference energy consumption, inference latency and deployment costs, and traditional on-chip optical neural network designs struggle to balance inference efficiency and performance.

Method used

A hybrid optoelectronic computing architecture is constructed. The input data is converted into light field control signals through a digital front-end processing module, optical inference calculations are performed using a pre-cured photon scattering medium, and mathematical processing is performed by a digital back-end processing module, forming an end-to-end on-chip model inference process.

Benefits of technology

It reduces inference energy consumption, improves model inference efficiency, and enables low-cost on-chip deployment of large-scale pre-trained models, balancing the high speed of optical computing and the flexibility of digital computing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122693743B_ABST
    Figure CN122693743B_ABST
Patent Text Reader

Abstract

This application provides an on-chip model solidification system and an on-chip model inference method. The on-chip model solidification system includes a digital front-end processing module, multiple physical foundation model modules, and a digital back-end processing module to construct an optoelectronic hybrid computing architecture for pre-trained model inference, enabling on-chip solidification and inference of pre-trained models. Specifically, the digital front-end processing module converts input data into optical field control signals; the physical foundation model modules receive the optical field control signals, perform optical inference calculations based on a pre-solidified photon scattering medium, and output corresponding electrical signals, wherein the photon scattering medium pre-solidifies the physical structures corresponding to the pre-trained model parameters; and the digital back-end processing module receives the electrical signals output by the physical foundation model modules and performs at least one mathematical process to output the inference result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence inference chips and integrated photonics technology, and in particular to an on-chip model solidification system and an on-chip model inference method. Background Technology

[0002] As the parameter size of Large Language Models (LLMs) continues to increase, their inference energy consumption, inference latency, and deployment costs continue to rise, making it difficult to meet the application requirements of edge devices for low-energy, high-speed inference.

[0003] Currently, some research has attempted to use optical computing technologies such as on-chip optical neural networks to achieve neural network inference operations. However, traditional technologies are mainly geared towards the training and inference design of small-scale optical neural networks. For large-scale pre-trained models, there is still a lack of on-chip model solidification systems that can adapt to their inference processes, making it difficult to balance inference efficiency and performance, and further improvements are urgently needed.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] Based on this, embodiments of this application provide an on-chip model solidification system and an on-chip model inference method to construct an optoelectronic hybrid computing architecture for pre-trained model inference and realize on-chip solidification inference of pre-trained models.

[0006] According to some embodiments, this application provides an on-chip model solidification system, including a digital front-end processing module, multiple physical base model modules, and a digital back-end processing module, wherein... The digital front-end processing module is used to convert input data into light field control signals; The physical foundation model module is used to receive the light field control signal, perform optical inference calculations based on the pre-cured photon scattering medium, and output the corresponding electrical signal. The photon scattering medium is pre-cured with the physical structure corresponding to the pre-trained model parameters. The digital back-end processing module is used to receive the electrical signal output by the physical foundation model module and perform at least one mathematical process to output the inference result.

[0007] In some embodiments, the digital front-end processing module is used to perform word segmentation encoding on the input data, convert the input data into input tokens, perform embedding mapping and position encoding on the input tokens to generate an input digital vector, and perform a conversion between vector and light field encoding on the input digital vector to generate the light field control signal.

[0008] In some embodiments, the physical fundamental model module includes an on-chip phase modulator array, a photon scattering medium with a solidified refractive index distribution during manufacturing, and a photodetector array, wherein, The on-chip phase modulator array is used to perform phase modulation on the input optical field according to the received optical field control signal; The photon scattering medium is used to receive the phase-modulated input light field and perform the optical inference calculation; The photodetector array is used to receive optical inference calculation results and convert the optical inference calculation results into the electrical signal.

[0009] In some embodiments, the on-chip model solidification system further includes a module routing control module; The module routing control module is used to route the light field control signal output by the digital front-end processing module to the corresponding physical foundation model module according to the routing strategy.

[0010] In some embodiments, the module routing control module includes a gated network; The gated network is based on a hybrid expert scheduling strategy, generates routing results based on the input digital vector, and determines the physical fundamental model module corresponding to the optical field control signal based on the routing results.

[0011] In some embodiments, the on-chip model solidification system further includes a post-manufacturing adaptive module, which is used to perform low-rank adaptation during the mathematical processing performed by the digital back-end processing module. The digital back-end processing module outputs the inference result after low-rank adaptation.

[0012] In some embodiments, the mathematical processing includes residual connections, which include short-circuit residual connections or linear projection residual connections; The digital front-end processing module is configured to store a digital copy of the input data before generating the light field control signal; The digital back-end processing module is configured to, upon receiving the electrical signal, perform short-circuit residual addition on the mathematical processing result corresponding to the electrical signal and the digital copy, or perform linear projection residual addition on the result of linear projection of the digital copy, to generate the inference result.

[0013] In some embodiments, the digital back-end processing module is configured to output the inference result it outputs to the next-layer digital front-end processing module as input data for the next-layer digital front-end processing module.

[0014] In some embodiments, the post-manufacturing adaptive module includes a low-rank adaptation matrix configured to perform low-rank adaptation via digital paths during mathematical processing performed by the digital back-end processing module.

[0015] According to some embodiments, this application also provides an on-chip model inference method, which is executed using the on-chip model solidification system provided in the foregoing embodiments, and includes: The input data is converted into light field control signals through a digital front-end processing module; The system receives the light field control signal through multiple physical model modules, performs optical inference calculations based on the pre-cured photon scattering medium, and outputs the corresponding electrical signal. The digital back-end processing module receives the electrical signals output by the multiple physical model modules and performs at least one mathematical process to output the inference result.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0017] The embodiments of this application may have, or at least have, the following advantages: In this embodiment, the digital front-end processing module converts input data into light field control signals, the physical foundation model module performs optical inference calculations based on a photon scattering medium with pre-fixed physical structures corresponding to pre-trained model parameters, and the digital back-end processing module performs mathematical processing on the optical inference calculation results and outputs the inference results. Thus, through a hybrid optoelectronic computing approach, an on-chip model-fixed inference architecture suitable for pre-trained model inference is constructed, forming an end-to-end on-chip model inference process.

[0018] Furthermore, this application utilizes a photon scattering medium with a pre-fixed physical structure corresponding to the pre-trained model parameters to perform optical inference calculations. This eliminates the need for frequent reading and updating of model parameters during the inference process, which is beneficial for fully utilizing the low energy consumption and high speed characteristics of optical computing, thereby improving model inference efficiency and reducing inference costs.

[0019] Furthermore, this application employs a digital front-end processing module and a digital back-end processing module to complete input processing and digital post-processing respectively, enabling optical computing and digital computing to work together. While fully leveraging the advantages of optical computing, it also takes into account the flexibility of digital processing. For example, the optical part undertakes computationally intensive matrix operations, while the digital part is responsible for auxiliary processing that requires high flexibility and has a relatively small computational load, thereby facilitating the on-chip deployment of large-scale pre-trained models.

[0020] Other advantages, objectives, and features of this application will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from practice of this application. The objectives and other advantages of this application can be realized and obtained through the following description. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0022] Figure 1 This is a schematic diagram of the structure of an on-chip model solidification system provided in some embodiments of this application; Figure 2 This application provides a schematic diagram of the system architecture for an on-chip model solidification system in some embodiments. Figure 3 A schematic diagram illustrating the working principle of the module routing control module in the on-chip model solidification system provided in some embodiments of this application; Figure 4 A schematic diagram illustrating the working principle of the post-manufacturing adaptive module in an on-chip model solidification system provided in some embodiments of this application; Figure 5 This is a flowchart illustrating an on-chip model inference method provided in some embodiments of this application.

[0023] Explanation of reference numerals in the attached figures: 100. Digital front-end processing module; 200. Physical basic model module; 300. Digital back-end processing module; 400. Module routing control module; 500. Post-manufacturing adaptive module. Detailed Implementation

[0024] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0026] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0027] As the parameter size of Large Language Models (LLMs) continues to increase, their inference energy consumption, inference latency, and deployment costs continue to rise, making it difficult to meet the application requirements of edge devices for low-energy, high-speed inference.

[0028] Currently, some research has attempted to use optical computing technologies such as on-chip optical neural networks to achieve neural network inference operations. However, traditional technologies are mainly geared towards the training and inference design of small-scale optical neural networks. For large-scale pre-trained models, there is still a lack of on-chip model solidification systems that can adapt to their inference processes, making it difficult to balance inference efficiency and performance, and further improvements are urgently needed.

[0029] Based on this, this application aims to provide a solution that can address the aforementioned technical problems, thereby constructing an optoelectronic hybrid computing architecture for pre-trained model inference and realizing on-chip solidified inference of pre-trained models. Details will be elaborated in subsequent embodiments.

[0030] Please combine Figure 1 and Figure 2 Understood, this application provides an on-chip model solidification system. For example... Figure 1 and Figure 2 As shown, the on-chip model solidification system includes a digital front-end processing module 100, multiple Physical Foundation Model (PFM) modules 200, and a digital back-end processing module 300.

[0031] The digital front-end processing module 100 converts input data into optical field control signals. The physical foundation model module 200 receives the optical field control signals, performs optical inference calculations based on a pre-cured photon scattering medium, and outputs corresponding electrical signals. The photon scattering medium has a pre-cured physical structure corresponding to the pre-trained model parameters. The digital back-end processing module 300 receives the electrical signals output by the physical foundation model module 200 and performs at least one mathematical process to output inference results.

[0032] In the aforementioned on-chip model solidification system, the digital front-end processing module 100 converts input data into optical field control signals, the physical foundation model module 200 performs optical inference calculations based on a photon scattering medium with pre-solidified physical structures corresponding to pre-trained model parameters, and the digital back-end processing module 300 performs mathematical processing on the optical inference calculation results and outputs the inference results. Thus, through a hybrid optoelectronic computing approach, an on-chip model solidification inference architecture suitable for pre-trained model inference is constructed, forming an end-to-end on-chip model inference process.

[0033] Furthermore, the aforementioned on-chip model solidification system utilizes a photon scattering medium with pre-solidified physical structures corresponding to pre-trained model parameters to perform optical inference calculations. This eliminates the need for frequent reading and updating of model parameters during the inference process, which helps to fully utilize the low energy consumption and high speed characteristics of optical computing, improve model inference efficiency, and reduce inference costs.

[0034] Furthermore, the aforementioned on-chip model solidification system employs a digital front-end processing module 100 and a digital back-end processing module 300 to complete input processing and digital post-processing, respectively. This allows optical computing and digital computing to work together, fully leveraging the advantages of optical computing while also taking into account the flexibility of digital processing. For example, the optical part can handle computationally intensive matrix operations, while the digital part can handle auxiliary processing that requires high flexibility but has a relatively small computational load. This helps to achieve on-chip deployment of large-scale pre-trained models.

[0035] The above is the core idea of ​​this application. The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0036] In this embodiment, the digital front-end processing module 100 is used to convert input data into light field control signals so that the input data can be adapted to the physical foundation model module 200 to perform optical inference calculations.

[0037] As an example, the digital front-end processing module 100 can be implemented using CMOS digital circuits, but it is not limited to this.

[0038] In some embodiments, the digital front-end processing module 100 may first perform word segmentation encoding on the input data, converting the input data into input tokens. Subsequently, the digital front-end processing module 100 performs embedding mapping and position encoding on the input tokens to generate an input digital vector. A vector-to-light field encoding conversion is then performed on the input digital vector to generate a light field control signal, which is then input to the physical foundation model module 200.

[0039] As an example, the input numeric vector x can be, for instance, a d-dimensional numeric vector, denoted as: x∈R d Where d represents the dimension of the input numeric vector, and R d Let represent a d-dimensional real vector space.

[0040] For example, the digital front-end processing module 100 can convert the input digital vector into a control signal for the spatial light modulator, which serves as a light field control signal.

[0041] This application embodiment is configured with multiple physical basic model modules 200, each physical basic model module 200 having a photon scattering medium integrated inside, and the photon scattering medium having a physical structure corresponding to the pre-trained model parameters pre-fixed.

[0042] In other words, the physical foundation model module 200 can solidify the parameters of the pre-trained model and the physical structure corresponding to the model in the physical hardware, so that the model parameters can be permanently stored in the physical hardware in the form of physical geometry (such as refractive index distribution). Therefore, there is no need to read or write the model parameters during the inference process. The calculation process is directly completed through physical dynamics (such as the propagation process of light in a photon scattering medium naturally completing matrix-vector multiplication) to perform the inference calculation corresponding to the pre-trained model.

[0043] Therefore, the physical fundamental model module 200 can receive the light field control signal output by the digital front-end processing module 100, perform optical inference calculations based on the pre-cured photon scattering medium, and convert the optical inference results into corresponding electrical signals and output them to the outside.

[0044] As an example, an on-chip model solidification system may include K physical model modules 200, denoted as: {M1, …… , M K} Where K represents the number of physical foundation model modules 200, and each physical foundation model module 200 can have the same or different physical structures.

[0045] In the above embodiments, the K physical foundation model modules 200 may correspond to K different expert sub-networks in the pre-trained model, or to K different downstream task domains. During the manufacturing process, each physical foundation model module 200 solidifies the physical structure of its corresponding expert sub-network to encode different expert knowledge.

[0046] As an example, the photon scattering medium can be fabricated based on an SOI (Silicon-on-Insulator) substrate or a glass substrate, and the refractive index distribution can be solidified during the fabrication process, so that the physical structure corresponding to the pre-trained model parameters is solidified in the photon scattering medium.

[0047] In some embodiments, the physical fundamental model module 200 may specifically include an on-chip phase modulator array, a photon scattering medium with a solidified refractive index distribution during manufacturing, and a photodetector (PD) array.

[0048] The on-chip phase modulator array receives the optical field control signal output by the digital front-end processing module 100 and modulates the input optical field according to the optical field control signal.

[0049] The photon scattering medium receives the phase-modulated input light field and performs optical inference calculations. The physical structure corresponding to the pre-trained model parameters is pre-fixed inside the photon scattering medium. That is, the pre-trained model parameters are fixed into the refractive index distribution of the photon scattering medium through the manufacturing process, so that the scattering and propagation process of the input light field in the photon scattering medium corresponds to the forward inference process of the pre-trained model.

[0050] Subsequently, the photodetector array receives the optical inference calculation results and converts them into corresponding electrical signals for subsequent mathematical processing by the digital back-end processing module 300.

[0051] As an example, an on-chip phase modulator array can be used to provide programmable parameters to support fine-tuning after manufacturing. Exemplarily, the number of programmable parameters can be less than 1% of the total number of parameters in the model, allowing for post-manufacturing fine-tuning of the model while maintaining the physical structure of the photon scattering medium unchanged.

[0052] In this embodiment, the pre-trained model parameters are permanently stored in the physical foundation model module 200 in the form of physical geometry (e.g., refractive index distribution). No parameter reading or writing operations are required during inference, which fundamentally eliminates the energy consumption bottleneck caused by data transfer between storage units and computing units in the von Neumann architecture.

[0053] Furthermore, the computation process is executed directly in the form of physical dynamics. For example, the propagation of light in a photon scattering medium naturally completes matrix-vector multiplication, and its energy consumption is close to the physical limit, limited only by the quantum efficiency of photoelectric conversion and detection processes, which is far lower than the energy consumption of bit-by-bit operations in digital circuits.

[0054] In addition, after the photon scattering medium in the physical foundation model module 200 is pre-cured, the core parameters do not need to be changed frequently. However, it can be combined with the digital front-end processing module 100 and the digital back-end processing module 300, or further combined with the post-manufacturing adaptive module involved in subsequent embodiments to achieve flexible adaptation to multiple tasks, avoiding the limitation of fixed single purpose after one-time manufacturing.

[0055] As an example, the physics foundation model module 200 can be used to implement matrix operations in the Transformer attention mechanism.

[0056] Specifically, the Transformer attention mechanism can be represented as:

[0057] Among them, matrix multiplication This is achieved through the following optical method: the query matrix Q is encoded as the input light field, and the bond matrix K corresponds to the refractive index distribution solidified in the photon scattering medium. The physical fundamental model module 200 outputs the attention weight distribution by utilizing the coherent superposition between the input light field and the solidified scattering medium. Furthermore, the normalization (Softmax) of the attention weights is completed by the digital back-end processing module 300.

[0058] The digital back-end processing module 300 is used to receive the electrical signals output by the physical foundation model module 200 and perform digital processing on the electrical signals to output inference results.

[0059] As an example, the digital back-end processing module 300 receives the intensity vector output by the photodetector array. The intensity vector is an output vector with dimension d. out The vector, denoted as: y∈R dout Where, d out R represents the dimension of the output vector. dout d out A real vector space.

[0060] In some embodiments, the digital back-end processing module 300 can perform mathematical processing on the electrical signals output by the physical foundation model module 200, including normalization, residual connection and / or layer normalization, to generate inference results.

[0061] In some embodiments, the digital back-end processing module 300 may also be configured to output its own inference results to the next-layer digital front-end processing module 100 as input data for the next-layer digital front-end processing module 100.

[0062] Therefore, the digital back-end processing module 300 is used to complete the post-processing of the optical calculation results from the optical part to the digital part, and realizes the synergy between the optical calculation results and the digital calculation through digital processing, thereby completing the inference process of the on-chip model solidification system.

[0063] In some embodiments, the mathematical processing includes residual connections, which include short-circuit residual connections or linear projection residual connections, to form a cascaded residual inference architecture.

[0064] In the above embodiments, the digital front-end processing module 100 can store a digital copy of the input data before generating the light field control signal; after receiving the electrical signal, the digital back-end processing module 300 performs short-circuit residual addition on the mathematical processing result corresponding to the electrical signal and the digital copy, or performs linear projection residual addition on the result after linear projection of the digital copy, to generate the inference result.

[0065] As an example, the short-circuit residual connection can be implemented by a digital adder, such as the digital back-end processing module 300: z=y+x Where x represents the input digital vector stored in the digital front-end processing module 100, y represents the mathematical processing result received by the digital back-end processing module 300, and z represents the output result after short-circuit residual connection.

[0066] As an example, for linear projection residual connections, the digital back-end processing module 300 performs the following: z=y+Wx Where W represents the linear projection matrix.

[0067] For example, the matrix W in the linear projection residual can be implemented to form a cascaded residual inference architecture through a lightweight solidified physical foundation model module.

[0068] Please continue reading. Figure 2 In some embodiments, the on-chip model solidification system may further include a module routing control module 400, which is used to route the light field control signal output by the digital front-end processing module 100 to the corresponding physical basic model module 200 according to the routing strategy, so as to realize the collaborative work between multiple physical basic model modules 200.

[0069] As an example, the module routing control module 400 can be implemented using a programmable logic device, but it is not limited to this.

[0070] In some embodiments, the module routing control module 400 includes a gated network g. θ .

[0071] Specifically, please refer to Figure 3 understand, Figure 3A schematic diagram illustrating the working principle of the module routing control module provided in this embodiment is shown. In this embodiment, the gating network g θ Based on the Mixture of Experts (MoE) scheduling strategy, routing results can be generated according to the input digital vector x generated inside the digital front-end processing module 100, and the physical basic model module 200 corresponding to the optical field control signal can be determined according to the routing results, thereby controlling the input of the optical field control signal to the corresponding physical basic model module 200 to perform optical inference calculation.

[0072] Furthermore, in some embodiments, the gating network generates routing results digitally and can be updated in real time during system operation. Meanwhile, the photon scattering medium in each physical model module 200 remains in a physically solidified state after manufacturing, unchanged by updates from the gating network. The digital routing and the physical model module 200 are independent of each other, thereby decoupling the scale of solidified parameters from the flexibility of the task.

[0073] As an example, routing results can be generated by a lightweight digital gating network, and can be represented as:

[0074] In the above embodiments, the gating network g θ The routing results are dynamically generated based on a hybrid expert scheduling strategy, and the corresponding physical foundation model module 200 is determined according to the routing results. This achieves a one-time manufacturing and solidification of the physical structure, while the task allocation strategy is adjusted in real time digitally, thus decoupling the scale of solidified parameters from the flexibility of tasks.

[0075] Please continue reading. Figure 2 In some embodiments, the on-chip model solidification system may further include a post-manufacturing adaptive module 500. The post-manufacturing adaptive module 500 is used to perform low-rank adaptation during the mathematical processing performed by the digital back-end processing module 300. In this embodiment, the inference result output by the digital back-end processing module 300 is the inference result after low-rank adaptation.

[0076] As an example, the total number of adaptation parameters corresponding to the post-manufacturing adaptive module 500 can not exceed 0.1% of the total number of solidified parameters. In this way, the post-manufacturing adaptive module 500 can achieve fine-grained task adaptation through digital paths without modifying the solidified photonic devices in the physical basis model module 200.

[0077] Please see Figure 4 , Figure 4The diagram illustrates the working principle of a post-manufacturing adaptive module according to some embodiments of this application. In some embodiments, the post-manufacturing adaptive module 500 includes a low-rank adaptation matrix, which is configured to perform low-rank adaptation via digital paths during mathematical processing performed by the digital back-end processing module 300.

[0078] As an example, the post-manufacturing adaptive module 500 includes independently trainable low-rank fitness matrices A and B. Low-rank fitness matrices A and B can be represented as: A k ∈R d×r B k ∈R r×d Where d represents the dimension of the input numeric vector, r represents the rank of the low-rank matrix, and r d.

[0079] Specifically, LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. Its core idea is to freeze the main parameters of the pre-trained model, train only two low-rank matrices A and B, and approximate the parameter update by multiplying the low-rank adaptation matrix A and the low-rank adaptation matrix B, thereby achieving model fine-tuning.

[0080] In the above embodiments, the LoRA concept is applied to the on-chip model solidification system. The solidified photonic devices in the physical foundation model module 200 undertake the basic transformation function corresponding to the main parameters of the pre-trained model, while the low-rank adaptation matrices A and B achieve fine-grained task adaptation through digital paths.

[0081] Please continue reading. Figure 4 To understand some embodiments of this application, such as Figure 4 As shown, the physical fundamental model module 200 is the main path, used to perform optical inference calculations, and its output is denoted as T(x). The optical inference calculations correspond to the fundamental inference calculations performed by the pre-trained model parameters.

[0082] The corresponding branch of the post-manufacturing adaptive module 500 generates a low-rank adaptation (LoRA) result, denoted as B·A·x, through low-rank adaptation matrix A and low-rank adaptation matrix B. Here, low-rank adaptation matrix A and low-rank adaptation matrix B are trainable parameters.

[0083] Subsequently, the basic inference result T(x) output by the main path and the low-rank adaptation result B·A·x output by the branch are added together in the digital back-end processing module 300 to obtain the final output h: h=T(x)+B·A·x Where h represents the inference result after LoRA fine-tuning.

[0084] Furthermore, since the number of parameters corresponding to the low-rank adaptation matrix A and the low-rank adaptation matrix B is much smaller than the number of parameters of the fixed parameters in the physical foundation model module 200, while keeping the physical structure of the photon scattering medium in the physical foundation model module 200 unchanged, only the low-rank adaptation matrix A and the low-rank adaptation matrix B need to be updated to achieve rapid switching between different task configurations without having to remanufacture or modify the physical structure of the photon scattering medium in the physical foundation model module 200.

[0085] For example, in some embodiments, the number of parameters corresponding to the low-rank adaptation matrix A and the low-rank adaptation matrix B is approximately 5 × 10. 5 The number of curing parameters in the physical fundamental model module 200 is approximately 10. 8 ~10 9 Therefore, the parameter sizes corresponding to low-rank adaptation matrices A and B are much smaller than the fixed parameter sizes, which helps to reduce the storage, loading, and switching costs of adaptation parameters.

[0086] The following example, using a low-rank fitness matrix A and a low-rank fitness matrix B with a total number of fitness parameters of 2dr, will be used for further explanation.

[0087] In some embodiments, when r ≤ 64 and d ≤ 4096, the total number of adaptation parameters is approximately 5 × 10⁻⁶. 5 Compared to solid-state photonic devices, which are approximately 10 8 ~10 9 With fixed parameters of a certain scale, the number of parameters corresponding to a low-rank adaptation matrix can be reduced by about 3 to 4 orders of magnitude, thus reducing the storage, loading, and switching costs of adaptation parameters.

[0088] In the aforementioned embodiments, the same physical model module 200 does not need to remanufacture or modify the solidified photonic device; it can switch to different task configurations simply by loading different low-rank adaptation matrices A and B.

[0089] For example, the same solidified physical model module 200 can switch between different task configurations by quickly loading different LoRA parameter sets. Each LoRA parameter set only needs to store about 2MB of data (assuming a single parameter occupies 4 bytes) and can switch to different downstream task configurations in milliseconds without remanufacturing or modifying the solidified photonic devices in the physical model module 200.

[0090] In some embodiments, the on-chip model curing system may further include a temperature compensation module. Specifically, the temperature compensation module may be used to monitor the temperature of the physical basis model module 200 in real time, determine the refractive index shift based on the pre-calibrated thermo-optic coefficient, and compensate for the refractive index shift.

[0091] As an example, the temperature compensation module may specifically include an on-chip temperature sensing array, which can monitor the temperature T of the physical fundamental model module 200 in real time. chip Temperature T of 200 in the physical fundamental model module was collected. chip Afterwards, the temperature compensation module can look up the table based on the pre-calibrated thermo-optic coefficient. n / T, and calculate the refractive index shift caused by temperature change according to the following formula:

[0092] Where δn(T) represents the refractive index shift at temperature T. Let ΔT represent the thermo-optic coefficient, and ΔT represent the temperature change between the temperature of the physical fundamental model module 200 and the reference temperature. Therefore, based on real-time monitoring of the temperature T of the physical fundamental model module 200... chip The calculated refractive index shift can be denoted as δn(T) chip ).

[0093] For example, the temperature compensation module can adjust the bias of the on-chip phase modulator array in the physical basis model module 200 according to the determined refractive index shift to compensate for the refractive index drift caused by temperature changes.

[0094] For example, in some embodiments, by setting a temperature compensation module in the on-chip model curing system, the inference accuracy loss of the on-chip model curing system does not exceed 5% within a temperature variation range of ±40K.

[0095] Based on the same inventive concept, this application also provides an on-chip model inference method, which is executed using the on-chip model solidification system provided in the foregoing embodiments.

[0096] Please see Figure 5 The on-chip model inference method includes the following steps S100~S300: S100: Converts input data into light field control signals through a digital front-end processing module.

[0097] S200: Receives light field control signals through multiple physical model modules, performs optical inference calculations based on a pre-cured photon scattering medium, and outputs corresponding electrical signals.

[0098] S300: Receives electrical signals output from multiple physical model modules 200 via a digital back-end processing module, and performs at least one mathematical process to output inference results.

[0099] It is understood that each step of the above-mentioned on-chip model inference method can be executed by the aforementioned on-chip model solidification system. Therefore, all the technical effects that the aforementioned on-chip model solidification system can achieve can also be achieved by the above-mentioned on-chip model inference method, and will not be repeated here.

[0100] In the description of this specification, references to terms such as "some embodiments," "as an example," "exemplarily," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiment or example.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features of the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An on-chip model solidification system, characterized in that, It includes a digital front-end processing module, multiple physical model modules, a digital back-end processing module, and a post-manufacturing adaptive module, among which, The digital front-end processing module is used to convert input data into light field control signals; The physical foundation model module is used to receive the light field control signal, perform optical inference calculations based on the pre-cured photon scattering medium, and output the corresponding electrical signal. The photon scattering medium is pre-cured with the physical structure corresponding to the pre-trained model parameters. The digital back-end processing module is used to receive the electrical signal output by the physical foundation model module and perform at least one mathematical process to output the inference result; The post-manufacturing adaptive module is used to perform low-rank adaptation during the mathematical processing performed by the digital back-end processing module. The digital back-end processing module outputs the inference result after low-rank adaptation.

2. The on-chip model curing system according to claim 1, characterized in that, The digital front-end processing module is used to perform word segmentation encoding on the input data, convert the input data into input tokens, perform embedding mapping and position encoding on the input tokens to generate an input digital vector, and perform a conversion between vector and light field encoding on the input digital vector to generate the light field control signal.

3. The on-chip model curing system according to claim 1, characterized in that, The physical fundamental model module includes an on-chip phase modulator array, a photon scattering medium with a solidified refractive index distribution during manufacturing, and a photodetector array. The on-chip phase modulator array is used to perform phase modulation on the input optical field according to the received optical field control signal; The photon scattering medium is used to receive the phase-modulated input light field and perform the optical inference calculation; The photodetector array is used to receive optical inference calculation results and convert the optical inference calculation results into the electrical signal.

4. The on-chip model curing system according to claim 3, characterized in that, The on-chip model solidification system also includes a module routing control module; The module routing control module is used to route the light field control signal output by the digital front-end processing module to the corresponding physical foundation model module according to the routing strategy.

5. The on-chip model curing system according to claim 4, characterized in that, The module routing control module includes a gated network; The gated network is based on a hybrid expert scheduling strategy, generates routing results based on the input digital vector, and determines the physical fundamental model module corresponding to the optical field control signal based on the routing results.

6. The on-chip model curing system according to claim 1, characterized in that, The mathematical processing includes residual connections, which include short-circuit residual connections or linear projection residual connections; The digital front-end processing module is configured to store a digital copy of the input data before generating the light field control signal; The digital back-end processing module is configured to, upon receiving the electrical signal, perform short-circuit residual addition on the mathematical processing result corresponding to the electrical signal and the digital copy, or perform linear projection residual addition on the result of linear projection of the digital copy, to generate the inference result.

7. The on-chip model curing system according to claim 1, characterized in that, The digital back-end processing module is configured to output the reasoning result it generates to the next-layer digital front-end processing module as input data for the next-layer digital front-end processing module.

8. The on-chip model curing system according to claim 1, characterized in that, The post-manufacturing adaptive module includes a low-rank adaptation matrix, which is configured to perform low-rank adaptation via digital paths during the mathematical processing performed by the digital back-end processing module.

9. An on-chip model inference method, characterized in that, The on-chip model inference method is executed using the on-chip model solidification system as described in any one of claims 1 to 8, including: The input data is converted into light field control signals through a digital front-end processing module; The system receives the light field control signal through multiple physical model modules, performs optical inference calculations based on the pre-cured photon scattering medium, and outputs the corresponding electrical signal. The digital back-end processing module receives the electrical signals output by the multiple physical foundation model modules and performs at least one mathematical process to output the inference result. The post-manufacturing adaptive module performs low-rank adaptation during the mathematical processing performed by the digital back-end processing module. The digital back-end processing module outputs the inference result after low-rank adaptation.

Citation Information

Patent Citations

  • Devices and methods employing optical-based machine learning using diffractive deep neural networks

    US20210142170A1