Storage and calculation integrated photon calculation system based on phase change metasurface and holographic light field modulation

Abstract

The invention discloses a storage and calculation integrated photon calculation system and method based on a phase change metasurface and holographic light field modulation, and aims to solve the problems that storage and calculation are separated, the dynamic modulation speed is low and the precision is limited in an existing photon calculation framework. The invention provides a hybrid computing architecture which takes a blue-ray disc or a phase change storage wafer as a high-density holographic weight storage medium and is combined with a high-speed spatial light modulator and a photoelectric nonlinear activation circuit. Through an area array parallel reading mechanism, a large-scale weight matrix is read at one time by using beam-expanded coherent light, light velocity matrix multiplication is completed, and interlayer fast switching of the deep neural network is realized in combination with a streaming rotation or light beam scanning technology. According to the invention, the non-volatility and multistage refractive index characteristics of the phase change material are utilized to realize high-density on-chip storage and zero-power-consumption lossless reading of large model weight, and the problems of a video memory wall and an energy consumption wall in large model reasoning are effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of photonic computing, artificial intelligence hardware acceleration, and optical storage technology, and particularly to an in-memory photonic computing system and method based on phase change metasurfaces and holographic light field modulation. Background Technology

[0002] With the explosive growth of deep learning, especially Large Language Models (LLM), the demand for computing power in artificial intelligence is increasing exponentially. However, traditional electronic computer architectures (von Neumann architecture) face severe challenges of the "memory wall" and the "power wall." When performing large model inference, the processor needs to frequently read massive amounts of weight data from memory, resulting in data transfer power consumption far exceeding the power consumption of the computation itself, and bandwidth limits the inference speed.

[0003] Photonic computing, as an emerging computing paradigm, utilizes the wave nature of photons (interference, diffraction) for computation, possessing inherent advantages such as high bandwidth, low latency, and low power consumption. However, existing photonic computing schemes (such as photonic chips based on Mach-Zehnder interferometers) typically face the following problems:

[0004] 1. Low storage density: Due to the limitations of device size, a single chip can only integrate a few thousand neurons, which is difficult to support advanced generative models that require tens of billions of neurons. It usually still needs to rely on external electrical memory, and has not truly solved the memory wall problem.

[0005] 2. Limited modulation mechanism: Traditional thermo-optic or electro-optic modulators are large in size and consume a lot of power, making it difficult to achieve large-scale integration.

[0006] 3. Lack of nonlinearity: Photons themselves do not have nonlinear interactions, making it difficult to implement activation functions (such as ReLU) in neural networks using all-optical methods.

[0007] Phase change materials (such as GST) are well-established due to their widespread use in optical disc storage and phase change memory (PCM). They exhibit a significant difference in refractive index between crystalline and amorphous states and possess non-volatility (data is not lost after power failure). However, current technologies primarily utilize them for binary data storage, with few reports of their application as holographic optical elements in large-scale matrix computations.

[0008] Therefore, there is an urgent need for an in-memory computing architecture that can combine high-density optical storage with high-speed optical computing to achieve real-time inference of large models in a low-cost and low-power manner. Summary of the Invention

[0009] One objective of this invention is to propose an in-memory photonic computing system and method based on phase change metasurfaces and holographic light field modulation. The core concept is to use mature Blu-ray discs or phase change wafers as a "holographic weight library" to pre-"freeze" the weight matrix of the neural network in the physical medium and complete the calculation instantaneously using light diffraction, thereby completely eliminating the weight transfer overhead in the inference process.

[0010] According to an embodiment of the present invention, an in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation includes:

[0011] The light source module is configured to generate a highly coherent readout beam and to expand and collimate the beam to form a planar array of parallel light covering a specific computing region.

[0012] An input modulation module, located downstream of the light source module, is configured to receive a digital input vector and load the vector into the optical field distribution of the area array parallel light to generate a structured light field carrying input information.

[0013] The holographic storage module, located downstream of the optical path of the input modulation module, includes at least one holographic metasurface storage medium constructed based on phase change material. The medium is pre-etched with Fourier holograms or computational holograms corresponding to the weight matrix of the neural network. When the structured light field irradiates the surface of the medium, diffraction and interference occur, and matrix multiplication is completed during the propagation of the light path.

[0014] The photoelectric activation module, located downstream of the optical path of the holographic storage module, is configured to receive the diffracted light field after calculation, convert it into an electrical signal, and perform activation function calculation using the nonlinear current-voltage characteristics of the analog circuit.

[0015] The control and synchronization module is configured to coordinate the data loading timing of the input modulation module with the physical position or beam scanning position of the holographic storage module, thereby enabling multi-layer continuous inference of the neural network.

[0016] Optionally, the storage medium in the holographic storage module is a Blu-ray Disc structure or a wafer structure based on chalcogenide phase change alloy (GST); the medium utilizes the difference in complex refractive index of the phase change material in crystalline and amorphous states to modulate the amplitude or phase of the incident light; the hologram on the medium is generated using a computer holographic algorithm (CGH) and pre-solidified in the medium recording layer through laser direct writing or photolithography to form a static weight database.

[0017] Optionally, the system employs a parallel reading mechanism; the spot diameter of the parallel light generated by the light source module covers a complete weight matrix region or its sub-block region on the medium; when the holographic storage and computing module performs calculations, the medium remains relatively stationary or in a flowing motion state, and utilizes the parallel diffraction characteristics of light to complete the multiply-add (MACs) operations in the entire region at the instant the light flies, without the need for point-by-point scanning and reading.

[0018] Optionally, the control and synchronization module includes a photoelectric encoder or a position sensor; when the medium is a rotating optical disc structure, the encoder monitors the rotation angle of the optical disc in real time; when the optical disc rotates to the hologram region corresponding to a specific layer of the neural network, the control and synchronization module triggers the input modulation module to refresh the corresponding input vector, and synchronously triggers the photoelectric activation module to perform signal acquisition, thereby realizing the pipelined operation of inter-layer computation.

[0019] An in-memory photonic computing method based on phase transition metasurfaces and holographic light field modulation according to an embodiment of the present invention includes the following steps:

[0020] Preprocessing steps: The weight matrix of the neural network to be deployed is divided into blocks and holographically encoded to generate corresponding holographic data, and the holographic data is physically recorded onto a predetermined track or region of the phase change metasurface storage medium;

[0021] Input loading steps: Obtain the input vector of the current inference task, and use a spatial light modulator to map the vector into the spatial intensity distribution or phase distribution of the incident beam;

[0022] Light speed calculation steps: control the incident light beam to irradiate the corresponding weight region of the storage medium, and utilize the diffraction characteristics of the medium to realize optical convolution or matrix multiplication of the input vector and the weight matrix during light propagation;

[0023] Nonlinear activation step: The photodetector array receives the calculated light field, converts the light intensity signal into photocurrent, and applies a nonlinear transformation to the photocurrent through a diode or transistor circuit to obtain the activated output vector;

[0024] Interlayer iteration steps: The output vector is used as the input vector of the next layer network to drive the next level light source or modulator, and the storage medium is controlled to switch to the holographic region of the next layer weights until the inference of the entire network is completed.

[0025] Optionally, in the preprocessing step, the Gerchberg-Saxton algorithm or the point source holography algorithm is used to calculate the pure phase hologram or complex amplitude hologram of the weight matrix; and the calculated hologram is discretized into a binary or multi-valued bitmap, which is mapped to the crystallization degree distribution of the phase change material.

[0026] Optionally, in the interlayer iteration step, an acousto-optic deflector (AOD) or a microelectromechanical system (MEMS) galvanometer is used to rapidly change the incident angle or position of the readout beam, thereby achieving microsecond-level switching of different weight regions on a stationary storage medium wafer, replacing mechanical rotation seeking.

[0027] Optionally, for large language model (LLM) inference tasks, multiple storage media can be stacked or multiple regions can be spliced ​​to store ultra-large-scale parameters. During the inference process, wavelength division multiplexing (WDM) technology is used to read the weights of storage layers or different regions at different depths using readout light of different wavelengths, so as to realize parallel computing of multi-layer networks.

[0028] The beneficial effects of this invention are:

[0029] This invention achieves ultimate in-memory computing: it directly utilizes the storage medium (optical disc / wafer) itself as a computing element, eliminating the need to move weight data between memory and processor, fundamentally breaking the von Neumann bottleneck and achieving zero-power weight reading.

[0030] This invention boasts extremely high computing power density and energy efficiency: by utilizing the parallel diffraction characteristics of light, a single illumination can complete matrix multiplication with millions of parameters, and the calculation speed is only limited by the speed of light and the photoelectric conversion speed, with an energy efficiency far exceeding that of traditional GPUs.

[0031] This invention is low-cost and easily scalable: it utilizes the mature Blu-ray disc industry chain or phase-change storage technology, with extremely low media cost and huge capacity (25GB+ per disc), and can be easily expanded through stacking to support large model inference with hundreds of billions of parameters. Attached Figure Description

[0032] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0033] Figure 1 This is a diagram illustrating the overall architecture of a memory-integrated photonic computing system based on phase-change metasurfaces and holographic light field modulation proposed in this invention.

[0034] Figure 2 This is a schematic diagram of the parallel reading principle of the holographic storage and computing module proposed in this invention.

[0035] Figure 3 This is the streaming inference timing control diagram based on a rotating optical disc proposed in this invention;

[0036] Figure 4 This is a circuit diagram of the photoelectric nonlinear activation module proposed in this invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0038] refer to Figure 1-4 A memory-integrated photonic computing system based on phase transition metasurfaces and holographic light field modulation, comprising:

[0039] The light source module uses a 405nm wavelength blue laser diode, combined with a beam expander lens group, to generate a collimated parallel beam with a diameter of about 1cm, which serves as the readout carrier of the system.

[0040] The input modulation module uses a digital micromirror device (DMD) or a liquid crystal spatial light modulator (SLM) to receive input vector data (such as token embedding in large model inference) from the host computer. By controlling the micromirror to flip or the liquid crystal to deflect, the electrical signal is converted into the spatial intensity distribution of the light beam.

[0041] The core component of the holographic storage and computing module is a Blu-ray disc containing neural network weights. The disc is mounted on a precision electric spindle and rotates at a constant angular velocity. The recording layer of the disc is made of GST phase-change material, and the weight matrix is ​​pre-converted into holographic interference fringes using a holographic encoding algorithm, then solidified onto the disc tracks in a crystalline / amorphous phase-change manner. When a modulated structured beam of light illuminates the disc surface, the disc acts as a complex diffraction grating, performing wavefront transformation on the incident light. The far-field distribution of the diffracted light is the result of matrix multiplication.

[0042] The photoelectric activation module, located at the Fourier plane of the optical disc's reflective / transmittive optical path, consists of a high-speed CMOS image sensor or a photodiode array. It receives the diffracted light spot and converts the light intensity into a current signal. This current signal then passes through an analog circuit composed of diodes and resistors. Utilizing the unidirectional conductivity and nonlinear current-voltage characteristics of the diodes, the ReLU activation function is simulated to complete the nonlinear transformation.

[0043] The control and synchronization module includes a high-precision photoelectric encoder mounted on the spindle motor. The encoder provides real-time feedback on the optical disc's angular position. When the optical disc rotates to a preset sector (corresponding to a certain layer weight in the network), the controller sends a trigger signal to synchronously start the DMD to refresh input data and the CMOS to acquire output data, ensuring spatiotemporal alignment during the calculation process.

[0044] Example 1: To verify the feasibility of the present invention in practice, a prototype machine was built to run single-layer inference of the Qwen3-8B model (weight file is 16G).

[0045] First, weight preprocessing is performed: the Attention weight matrix of the first layer of the Qwen3 model is extracted, and its pure phase hologram is calculated using the Gerchberg-Saxton algorithm. The phase is then... Figure 2 Values ​​are converted to 0 / 1 bitmaps.

[0046] Then, the recording process is performed: the bitmap data described above is written to the inner track of a blank BD-RE disc using a Blu-ray burner.

[0047] Next, a demonstration of the reasoning process is performed: The system is powered on, and the optical disc is accelerated to 10,000 rpm. The host computer sends an input vector to the DMD. When the encoder detects that the optical disc has rotated to the inner recording area, it triggers the DMD to expose for 20 microseconds. The laser beam, modulated by the DMD, illuminates the optical disc, and the reflected light is focused onto the CMOS target surface by a lens. The light spot image acquired by the CMOS is digitized and compared with the theoretical calculation results. The cosine similarity is greater than 0.95, verifying the correctness of the light speed matrix multiplication.

[0048] Example 2: This example demonstrates how to use "streaming" to achieve continuous inference in a multi-layer network.

[0049] The 36 weights of the Qwen3 model are sequentially recorded on different radius tracks of an optical disc (or different sectors of the same track).

[0050] At the start of inference, the optical disc continues to rotate. In the first millisecond, the optical head is aligned with the first layer area, and the result of the first layer is calculated. In the second millisecond, the optical disc rotates to a certain angle, the optical head is aligned with the second layer area, and the DMD is refreshed to the output result of the first layer (as the input of the second layer), and the second layer is calculated.

[0051] This cycle repeats, and the entire inference process of the 36-layer network can be completed in one rotation of the optical disc (approximately 6 milliseconds), achieving extremely high inference throughput.

[0052] Example 3:

[0053] This embodiment demonstrates the software architecture and performance simulation results of the present invention.

[0054] A software stack called "Photon Inference Engine" was built, which includes two core modules: "Holographic Weight Compiler" and "Photon Runtime".

[0055] The holographic weight compiler module employs a block-based Gerchberg-Saxton algorithm to divide a large-scale weight matrix (e.g., 4096×4096) into multiple sub-blocks (e.g., 1024×1024), and calculates the holographic phase map for each sub-block. Its core algorithm logic is as follows:

[0056] def gerchberg_saxton_tiled(self, weight_matrix, iterations=20):

[0057] # Tiled Gerchberg-Saxton algorithm: Split the large weight matrix into tiles and calculate holograms separately

[0058] H, W = weight_matrix.shape

[0059] # Zero-padding to multiples of tile_size

[0060] pad_h = (self.tile_size - H % self.tile_size) % self.tile_size

[0061] pad_w = (self.tile_size - W % self.tile_size) % self.tile_size

[0062] padded_W = np.pad(weight_matrix, ((0, pad_h), (0, pad_w)), mode='constant')

[0063] h_tiles = padded_W.shape[0] / / self.tile_size

[0064] w_tiles = padded_W.shape[1] / / self.tile_size

[0065] hologram_atlas = np.zeros_like(padded_W, dtype=np.uint8)

[0066] for i in range(h_tiles):

[0067] for j in range(w_tiles):

[0068] # Extract sub-tile

[0069] tile = padded_W[i*self.tile_size:(i+1)*self.tile_size,

[0070] j*self.tile_size:(j+1)*self.tile_size]

[0071] # Calculate the hologram of this sub-block (GS core iteration)

[0072] phase = self._gs_core(tile, iterations)

[0073] bitmap = self._phase_to_bitmap(phase)

[0074] #Enter image set

[0075] hologram_atlas[i*self.tile_size:(i+1)*self.tile_size,

[0076] j*self.tile_size:(j+1)*self.tile_size] = bitmap

[0077] return hologram_atlas

[0078] The PhotonRuntime module encapsulates the low-level hardware control interface, providing standard Linear operator interfaces to upper-level deep learning frameworks (such as PyTorch). Its core matrix multiplication (matmul) control logic is shown below:

[0079] class PhotonRuntime:

[0080] def matmul(self, input_vector, disc_address, in_features, out_features):

[0081] #The photonic computation process of y = Wx

[0082] #1. Hardware Seek: Controlling the spindle motor to rotate to the optical disc sector with the corresponding weight.

[0083] self.motor.seek(disc_address)

[0084] #2. Input Modulation: Reshape the input vector into a two-dimensional matrix and drive the DMD display.

[0085] input_pattern = self._vector_to_pattern(input_vector)

[0086] self.dmd.display(input_pattern)

[0087] #3. Optical computation: physical process (light speed travel + diffraction interference), taking approximately 1 ns.

[0088] #Laser beam -> DMD reflection -> Optical disc diffraction -> Lens focusing

[0089] #4. Detection: Trigger a high-speed camera or photodiode array to collect the diffraction spot.

[0090] raw_image = self.camera.capture()

[0091] #5. Post-processing: Map the spot image back to the output vector

[0092] output_vector = self._image_to_vector(raw_image, out_features)

[0093] return output_vector

[0094] Through physical simulation comparison, when performing a 4096×4096 matrix multiplication, the traditional GPU is limited by the memory bandwidth and takes about 30ms; while the photonic computing architecture of this invention, with the use of a fully analog signal processing path (DMD modulation + light speed flight + direct photoelectric connection), has a theoretical latency of only 20 microseconds, achieving a physical speedup ratio of about 1500 times, and reducing energy consumption by three orders of magnitude.

[0095] Further system-level simulations (based on Qwen3-8B model parameters) show:

[0096] 1. Light-speed throughput: With a CD rotation speed of 10,000 RPM, one rotation takes only 6ms. Since all layer weights are distributed along the track, theoretically, a complete 36-layer forward inference can be completed every 6ms, with a throughput of 166 tokens / s, far exceeding human reading speed.

[0097] Example 4:

[0098] This embodiment demonstrates a hardware-software co-processing strategy for special operators (such as RoPE and Softmax) in the Transformer architecture.

[0099] Since photonic computing excels at linear matrix operations (O(N^2)) but not at complex logic control and nonlinear operations (O(N)), this invention adopts a "focus on the big picture and let go of the small details" strategy.

[0100] For Rotation Position Encoding (RoPE), this invention does not attempt to implement it through complex optical components in the optical path. Instead, it utilizes the host computer CPU to perform the rotation operation in the digital domain before the input vector is loaded into the DMD. Since the computational complexity of RoPE is only O(N), which is negligible compared to matrix multiplication O(N^2), it will not become a system bottleneck.

[0101] For Softmax operation, this invention utilizes the exponential voltage-current characteristic of diodes (I ∝ exp(V)) to construct an analog exponential circuit, and combines it with a capacitor integrator circuit to achieve normalization, thereby completing the probability distribution calculation in the analog electrical domain and avoiding frequent analog-to-digital conversion (ADC) overhead.

[0102] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A memory-integrated photonic computing system based on phase transition metasurfaces and holographic light field modulation, characterized in that, include: The light source module is configured to generate a highly coherent readout beam and to expand and collimate the beam to form a planar array of parallel light covering a specific computing region. An input modulation module, located downstream of the light source module, is configured to receive a digital input vector and load the vector into the optical field distribution of the area array parallel light to generate a structured light field carrying input information. The holographic storage module, located downstream of the optical path of the input modulation module, includes at least one holographic metasurface storage medium constructed based on phase change material. The medium is pre-etched with Fourier holograms or computational holograms corresponding to the weight matrix of the neural network. When the structured light field irradiates the surface of the medium, diffraction and interference occur, and matrix multiplication is completed during the propagation of the light path. The photoelectric activation module, located downstream of the optical path of the holographic storage module, is configured to receive the diffracted light field after calculation, convert it into an electrical signal, and perform activation function calculation using the nonlinear current-voltage characteristics of the analog circuit. The control and synchronization module is configured to coordinate the data loading timing of the input modulation module with the physical position or beam scanning position of the holographic storage module, thereby enabling multi-layer continuous inference of the neural network.

2. The in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation according to claim 1, characterized in that, The storage medium in the holographic storage module is a Blu-ray disc structure or a wafer structure based on a sulfide phase change alloy. The medium utilizes the difference in complex refractive index of the phase change material in crystalline and amorphous states to modulate the amplitude or phase of the incident light; The hologram on the medium is generated using a computer holographic algorithm and pre-solidified in the medium recording layer using laser direct writing or photolithography to form a static weight database.

3. The in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation according to claim 1, characterized in that, The system employs a parallel readout mechanism for area arrays. The spot diameter of the area array parallel light generated by the light source module covers a complete weight matrix region or a sub-block region on the medium. When the holographic storage and computing module performs calculations, the medium remains relatively stationary or in a state of fluid motion. Utilizing the parallel diffraction characteristics of light, it completes multiplication and addition operations across the entire region in the instant the light travels, eliminating the need for point-by-point scanning and reading.

4. The in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation according to claim 1, characterized in that, The control and synchronization module includes a photoelectric encoder or a position sensor: When the medium is a rotating optical disc structure, the encoder monitors the rotation angle of the optical disc in real time; When the optical disc rotates to the holographic region corresponding to a specific layer of the neural network, the control and synchronization module triggers the input modulation module to refresh the corresponding input vector and simultaneously triggers the photoelectric activation module to acquire signals, thereby realizing the pipelined operation of inter-layer computation.

5. A memory-integrated photonic computing method based on phase transition metasurfaces and holographic light field modulation, characterized in that, Includes the following steps: Preprocessing steps: The weight matrix of the neural network to be deployed is divided into blocks and holographically encoded to generate corresponding holographic data, and the holographic data is physically recorded onto a predetermined track or region of the phase change metasurface storage medium; Input loading steps: Obtain the input vector of the current inference task, and use a spatial light modulator to map the vector into the spatial intensity distribution or phase distribution of the incident beam; Light speed calculation steps: control the incident light beam to irradiate the corresponding weight region of the storage medium, and utilize the diffraction characteristics of the medium to realize optical convolution or matrix multiplication of the input vector and the weight matrix during light propagation; Nonlinear activation step: The photodetector array receives the calculated light field, converts the light intensity signal into photocurrent, and applies a nonlinear transformation to the photocurrent through a diode or transistor circuit to obtain the activated output vector; Interlayer iteration steps: The output vector is used as the input vector of the next layer network to drive the next level light source or modulator, and the storage medium is controlled to switch to the holographic region of the next layer weights until the inference of the entire network is completed.

6. The in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation according to claim 5, characterized in that, In the preprocessing step, the Gerchberg-Saxton algorithm or the point source holography algorithm is used to calculate the pure phase hologram or complex amplitude hologram of the weight matrix; The calculated hologram is discretized into a binary or multi-valued bitmap, which is then mapped to the crystallization degree distribution of the phase change material.

7. The in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation according to claim 5, characterized in that, In the interlayer iteration step, the incident angle or position of the readout beam is rapidly changed by using an acousto-optic deflector or a microelectromechanical system galvanometer, realizing microsecond-level switching of different weight regions on a stationary storage medium wafer, replacing mechanical rotation seeking.

8. The in-memory photonic computing system based on phase transition metasurfaces and holographic light field modulation according to claim 5, characterized in that, In the interlayer iteration step, the incident angle or position of the readout beam is rapidly changed by using an acousto-optic deflector or a microelectromechanical system galvanometer, realizing microsecond-level switching of different weight regions on a stationary storage medium wafer, replacing mechanical rotation seeking.

Images