An operator scheduling method and system based on a photoelectric throughput cost atlas

Abstract

The application discloses an operator scheduling method and system based on an optoelectronic throughput cost atlas. The method comprises the following steps: analyzing a target neural network model to obtain the types of operators in the model and the data dependency relationship between the operators; constructing an optoelectronic throughput cost atlas by taking each operator as a node and the data dependency relationship between the operators as a connecting edge; traversing the optoelectronic throughput cost atlas to generate a continuous optical domain calculation subgraph and a continuous electrical domain calculation subgraph, and using the two subgraphs to construct multiple groups of candidate scheduling schemes; and calculating the comprehensive execution cost of each group of candidate scheduling schemes based on the execution cost of each operator and the conversion cost of each optoelectronic conversion position, and selecting the candidate scheduling scheme with the lowest comprehensive execution cost as a target scheduling scheme. The application can solve the problem that the overall performance of the system is reduced due to the scheduling of a single operator in the existing optoelectronic hybrid computing scheme.

Description

Technical Field

[0001] This invention belongs to the field of optoelectronic hybrid neural network technology, and particularly relates to an operator scheduling method and system based on optoelectronic throughput cost graph. Background Technology

[0002] With the advancement of large model technology, models now incorporate an increasing number of operators such as matrix multiplication, vector-matrix multiplication, normalization, activation functions, and Softmax. To improve the computational speed of neural networks and reduce energy consumption, current technologies typically employ dedicated acceleration chips to perform model inference tasks.

[0003] In neural network acceleration chips, optoelectronic hybrid computing is a technical solution to improve the throughput of matrix operations. This solution utilizes the characteristics of parallel propagation of optical signals and wavelength division multiplexing to offload operators suitable for parallel processing, such as matrix multiplication, vector-matrix multiplication, or matrix multiplication-addition, to the photonic computing array for execution, while on-chip electrical domain digital logic units perform operations such as normalization, nonlinear activation, comparison, conditional control, and data storage.

[0004] Taking matrix multiplication as an example, optical domain computation can be expressed as: in, For the input tensor, This is the weight matrix. This is the result of matrix multiplication and addition.

[0005] In existing optoelectronic hybrid acceleration schemes, digital input data, after passing through a digital-to-analog converter or driving circuit, is loaded onto an electro-optic modulator, thereby encoding it into an analog optical signal. A weighting matrix is ​​mapped to a photonic computing array via a micro-ring resonator, a Mach-Zehnder interferometer, or other tunable optical computing units. After completing matrix operations, the photonic computing array outputs simulation results carried by optical power, intensity difference, phase, or other optical physical quantities. These simulation results are then converted into digital signals by a photodetector and an analog-to-digital converter, and further processed by an electronic domain digital logic unit.

[0006] A typical data processing path is as follows: digital input - DAC or driver circuit - electro-optic modulator - photonic computing array - photodetector - ADC - electrical domain digital logic unit.

[0007] Neural network models can typically be represented as computational graphs consisting of multiple operator nodes and connecting edges. Existing scheduling schemes usually determine whether an operator is executed by a photonic computing array or an electronic digital logic unit based on its type, computational complexity, or execution time. For example, matrix multiplication operators are scheduled to the optical domain, while operators such as Softmax, LayerNorm, and activation functions are scheduled to the electronic domain.

[0008] In the above scheduling methods, operators are usually evaluated in isolation, and the scheduling process only considers the execution speed of a single operator in the optical or electrical domain, which leads to a decrease in the actual throughput of optical domain computing and consequently a decrease in the overall system performance. Summary of the Invention

[0009] The purpose of this invention is to provide an operator scheduling method and system based on photoelectric throughput cost graph, which partially solves or alleviates at least one of the above-mentioned shortcomings in the prior art. It can avoid the problem of system performance degradation caused by independent planning and scheduling of each operator in the existing photoelectric hybrid computing scheduling scheme, and only considering the amount of computation or execution time, thereby improving the overall system performance to a certain extent.

[0010] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution: A first aspect of the present invention is to provide an operator scheduling method based on photoelectric throughput cost graphs, comprising the following steps: S100, parse the target neural network model and obtain the type of each operator in the target neural network model and the data dependencies between operators; S200: Using each operator as a node and the data dependencies between operators as connecting edges, a photoelectric throughput cost map is constructed; wherein each node is labeled with the execution domain label of the corresponding operator, and the execution domain label includes the optical domain and the electrical domain; each connecting edge is labeled with a corresponding weight, and the weight is used to characterize the photoelectric conversion cost between the two nodes of the connecting edge. S300, traverse the photoelectric throughput cost map, and perform optical domain operator merging with nodes marked as optical domains as starting points to obtain at least one continuous optical domain computation subgraph. Perform electrical domain operator merging with nodes marked as electrical domains as starting points to obtain at least one continuous electrical domain computation subgraph. S400, Repeat step S300 by changing different nodes as starting points to generate multiple continuous optical domain computation subgraphs and continuous electrical domain computation subgraphs to construct multiple sets of candidate scheduling schemes; Each set of candidate scheduling schemes includes at least one continuous optical domain computation subgraph and / or at least one continuous electrical domain computation subgraph and the photoelectric conversion position corresponding to its boundary; S500 calculates the comprehensive execution cost of each candidate scheduling scheme based on the execution cost of each operator and the conversion cost of each photoelectric conversion position, and selects the candidate scheduling scheme with the lowest comprehensive execution cost as the target scheduling scheme.

[0011] Furthermore, in step S200, the method for calculating the weight of each connecting edge includes: Determine whether the execution domain labels of the nodes at both ends of the connection edge are the same; if the execution domain labels are the same, the weight of the connection edge is the first preset value; if the execution domain labels are different, obtain the delay, energy consumption and bandwidth occupation generated by the photoelectric conversion of the two nodes according to the data transmission direction corresponding to the connection edge, and obtain the photoelectric conversion cost of the connection edge as the weight through weighted calculation. The photoelectric conversion cost is calculated using the following formula: Calculate; where, The photoelectric conversion cost between the i-th node and the j-th node at both ends of the connecting edge is... The delay caused by the photoelectric conversion between the i-th node and the j-th node; The energy consumption generated by photoelectric conversion between the i-th node and the j-th node; To perform data caching or on-chip bandwidth usage between the i-th node and the j-th node; , These are preset weighting coefficients for the delay and energy consumption generated by photoelectric conversion, respectively. This is a preset weighting coefficient for bandwidth usage.

[0012] Furthermore, the step of generating the continuous optical domain computational subgraph in step S300 includes: S301A, select an undivided node with the execution domain label as an optical domain node as the starting point of the current continuous optical domain computation subgraph; S302A: According to the topological order in the photoelectric throughput cost map, determine whether the adjacent nodes satisfy the preset optical domain operator merging constraint conditions. If they satisfy the conditions, merge the adjacent nodes into the current continuous optical domain calculation subgraph. If they do not satisfy the conditions, stop expanding the current continuous optical domain calculation subgraph. S303A sets the photoelectric conversion trigger position at the boundary of the current continuous optical domain computation subgraph; S304A: Select the next undivided node with the execution domain label as optical domain as the new starting point, and repeat steps S302A to S303A until all nodes with the execution domain label as optical domain are divided into the corresponding continuous optical domain computation subgraph.

[0013] Furthermore, the step of generating the continuous electric domain computational subgraph in step S300 includes: S301B, Select an undivided node with the domain label "electric domain" as the starting point of the current continuous point domain computation subgraph; S302B: According to the topological order in the photoelectric throughput cost graph, determine whether the adjacent nodes satisfy the preset electric domain operator merging constraint conditions. If they satisfy the conditions, merge the adjacent nodes into the current continuous electric domain calculation subgraph. If they do not satisfy the conditions, stop expanding the current continuous electric domain calculation subgraph. S303B sets the photoelectric conversion position at the boundary of the current continuous electric domain computation subgraph; S304B: Select the next undivided node with the execution domain label "electric domain" as the new starting point, and repeat steps S302B to S303B until all nodes with the execution domain label "electric domain" are divided into the corresponding continuous electric domain computation subgraph.

[0014] Furthermore, the optical domain operator merging constraint includes at least one of the following: The photonic computing array resources required to execute the current continuous optical domain computation subgraph are less than or equal to the currently available array capacity; During the execution of the current continuous optical domain computation subgraph, the dynamic range of the simulated optical signal is within the operating range allowed by the optoelectronic device; The cumulative error of the current continuous optical domain computation subgraph is less than or equal to a preset error threshold. The cumulative error includes the cumulative values ​​of nonlinear noise, phase error, and propagation loss. The optical interconnect resources required for the current continuous optical domain computation subgraph are less than or equal to the currently available routing resources; The operators corresponding to all nodes in the current continuous optical domain computation subgraph belong to the linear operator types supported by the photonic computing array.

[0015] Furthermore, the electric domain operator merging constraint includes at least one of the following: The computational resources required to execute the current continuous electrical domain computation subgraph are less than or equal to the upper limit of parallelism of the electrical domain digital logic unit; The size of the intermediate results temporarily stored during the current continuous electrical domain computation subgraph is less than or equal to the on-chip cache capacity; The bus bandwidth required to execute the current continuous electrical domain computation subgraph is less than or equal to the bandwidth limit of the on-chip network; The total execution latency of the current continuous electrical domain computation subgraph is less than or equal to a preset latency threshold; The hardware accelerators required for the current continuous electric domain computation subgraph match the currently available accelerator resources.

[0016] Further, in step S302A, at least one candidate continuous optical domain computation subgraph is generated for each starting point, and the benefit value of each candidate continuous optical domain computation subgraph is calculated. Then, candidate continuous optical domain computation subgraphs with benefit values ​​greater than or equal to preset target benefits are selected as the target continuous optical domain computation subgraph for that starting point. The reward value of the candidate continuous optical domain computation subgraph is calculated using the following formula: Calculate; where, For optical domain gain value, To reduce the photoelectric conversion cost of performing continuous optical domain computation based on the candidate continuous optical domain computation subgraph, The computational cost reduction for computing the candidate continuous optical domain subgraph when performed in the optical domain compared to when performed in the electrical domain. This refers to the cost incurred by consuming optical routing resources when performing continuous optical domain computation based on the candidate continuous optical domain computation subgraph. These are pre-set risk penalty items.

[0017] Further, in step S302B, at least one candidate continuous electric domain computation subgraph is generated for each starting point, and the benefit value of each candidate electric domain computation subgraph is calculated. Then, candidate continuous electric domain computation subgraphs with benefit values ​​greater than or equal to preset target benefits are selected as the target continuous electric domain computation subgraph for that starting point. The reward value of the candidate continuous optical domain computation subgraph is calculated using the following formula: calculate; in, For electric domain revenue value, To reduce the photoelectric conversion cost by performing continuous electric domain calculations based on the candidate continuous electric domain calculation subgraph; The benefit of performing continuous electric domain computation based on the candidate continuous electric domain computation subgraph is relative to the benefit of performing the computation on each node in the candidate continuous electric domain computation subgraph separately. The cost of potential competition for electric domain resources after merging electric domain operators; Risks in the pre-defined execution of electrical domain calculations.

[0018] Furthermore, the process of setting the photoelectric conversion position in step S303A is as follows: At the boundary of the continuous optical domain computation subgraph, determine whether the execution domain label of the adjacent node connected to the boundary node is an electrical domain; if the adjacent node is an electrical domain, insert a photoelectric conversion trigger instruction at the boundary to mark the photoelectric conversion position.

[0019] Further, the process of setting the photoelectric conversion position in step S303B is as follows: at the boundary of the continuous electrical domain calculation subgraph, it is determined whether the execution domain label of the adjacent node connected to the boundary node is an optical domain; if the adjacent node is an optical domain, a photoelectric conversion trigger instruction is inserted at the boundary to mark the photoelectric conversion trigger position, or / and, an on-chip cache write position is inserted.

[0020] A second aspect of the present invention is to provide an operator scheduling system based on photoelectric throughput cost graphs, comprising: The parsing module is used to parse the target neural network model and obtain the types of each operator in the target neural network model and the data dependencies between operators. The cost graph construction module is used to construct a photoelectric throughput cost graph with each operator as a node and the data dependencies between operators as connecting edges. Each node is labeled with the execution domain label of the corresponding operator, which includes the optical domain and the electrical domain. Each connecting edge is labeled with a corresponding weight, which is used to characterize the photoelectric conversion cost between the nodes at both ends of the connecting edge. The subgraph generation module is used to traverse the photoelectric throughput cost map, and perform optical domain operator merging with nodes labeled as optical domains as starting points to obtain at least one continuous optical domain computation subgraph, and perform electrical domain operator merging with nodes labeled as electrical domains as starting points to obtain at least one continuous electrical domain computation subgraph. The candidate scheduling scheme generation module is used to change different nodes as starting points to trigger the subgraph generation module to generate multiple continuous optical domain computation subgraphs and continuous electrical domain computation subgraphs to construct multiple sets of candidate scheduling schemes; wherein, each set of candidate scheduling schemes includes several continuous optical domain computation subgraphs and / or continuous electrical domain computation subgraphs and the photoelectric conversion positions corresponding to their boundaries; The target scheduling scheme selection module is used to calculate the comprehensive execution cost of each group of candidate scheduling schemes based on the execution cost of the corresponding operator of each node and the conversion cost of each photoelectric conversion position, and select the candidate scheduling scheme with the lowest comprehensive execution cost as the target scheduling scheme.

[0021] Beneficial Effects: Traditional scheduling schemes isolate operators and statically classify them based solely on operator type, scheduling operators based on computational cost or execution time. For example, linear operators are scheduled for optical computation, while nonlinear operators are scheduled for electrical computation. However, in actual operation, when adjacent operators execute in different computational domains, conversion operations such as ADC, DAC, electro-optic modulators, and photodetectors need to be inserted. Frequent photoelectric conversions during system operation increase processing latency and energy consumption, and reduce the actual throughput benefits of optical computation. Therefore, even if a single operator executes faster in the optical domain, the overall system performance may actually decrease. This application, however, transforms the topology of the entire neural network model into a graph and optimizes the graph based on the photoelectric conversion cost between operators, generating corresponding operator scheduling schemes based on the optimized graph. In other words, this application transforms the scheduling problem of different types of operators into a global cost optimization problem of a computational graph. Specifically, the delay, energy consumption, and bandwidth usage generated during the photoelectric conversion process, which are distributed throughout the entire neural network model, are quantified and mapped into a unified metric: photoelectric conversion cost. Based on this cost, a "cost-aware" graph is constructed, and image segmentation (such as optical domain operator merging and / or electrical domain operator merging) is performed for optimization. Finally, dynamic decisions are made based on the optimized image to obtain the corresponding scheduling scheme. Furthermore, due to the construction of this graph, different scheduling schemes can be provided for different computational performance requirements. For example, for scenarios with high overall system performance requirements, both optical domain operator merging and electrical domain operator merging optimizations are performed simultaneously; for scenarios with relatively low overall system performance requirements, only optical domain operator merging optimization or electrical domain operator merging optimization is performed.

[0022] In actual operation, various photoelectric conversions are involved: digital-to-analog conversion, analog-to-digital conversion, electro-optic modulation, and photodetection. These photoelectric conversion links are independent and distributed among different operators, and belong to different forms of photoelectric conversion, i.e., heterogeneous conversion: DAC / ADC belongs to the digital-analog mixed signal domain, E / O modulation belongs to the analog-photonic nonlinear domain, and O / E detection belongs to the photonic-analog weak signal domain. Therefore, in order to construct a graph with "cost awareness," this application quantifies the photoelectric conversion costs in different photoelectric conversion links, that is, provides a unified metric. Specifically, the delay, energy consumption, and bandwidth occupation generated by heterogeneous conversion links such as analog-to-digital conversion, digital-to-analog conversion, electro-optic modulation, and photodetection involved in the system's execution of computational tasks (i.e., operation) are quantified and mapped to the weights of connection edges. This transforms the originally dispersed and independent physical conversion overhead into a computable and comparable unified metric in the graph structure, thereby making it possible to construct a graph with "cost awareness," providing a quantitative decision-making basis for subsequent operator scheduling (or computational path planning).

[0023] Furthermore, this invention can simultaneously perform optical domain operator merging and electrical domain operator merging. For example, on the optical domain side, nodes corresponding to adjacent linear operators can be merged, allowing analog optical signals to propagate directly in cascaded form, eliminating unnecessary intermediate analog-to-digital and digital-to-analog conversion processes. On the electrical domain side, adjacent nonlinear, reduction, and control operators can be merged, allowing for simultaneous exploitation of the benefits of continuous execution across both domains. This further reduces the total number of global photoelectric conversions and the total overhead, fully releasing the performance advantages of both types of computing units and effectively improving the end-to-end inference throughput and energy efficiency of the photoelectric hybrid computing architecture. Moreover, intermediate results can be directly reused in the on-chip cache, avoiding repeated, circuitous data transmission between the photoelectric and electrical domains.

[0024] To address the different hardware characteristics and resource boundaries of the optical and electrical domains, appropriate merging constraint rules are established for each. For example, physical constraints such as photonic computing array capacity, analog signal dynamic range, cumulative error, and waveguide routing resources are set on the optical domain side to ensure that the continuous optical domain paths obtained based on continuous optical domain computing subgraphs do not exceed the operating limits of optical devices, thereby avoiding exceeding computational accuracy limits and insufficient hardware resources. Similarly, resource constraints such as computing unit parallelism, on-chip cache capacity, on-chip network bandwidth, and total execution latency are set on the electrical domain side to avoid problems such as computational resource contention, cache overflow, and communication congestion caused by excessive merging in the electrical domain. The dual-domain constraint system is independent yet collaborative, ensuring that all subgraph merging schemes are within a reasonable range of hardware capacity, maximizing the benefits of continuous execution while minimizing the risk of performance degradation. Preferably, the priority of optical domain operator merging is higher than that of electrical domain operator merging.

[0025] This invention constructs a unified cost measurement system based on a photoelectric throughput cost map. It incorporates optical domain execution cost, electrical domain execution cost, cross-domain photoelectric conversion cost, and dual-domain operational risk cost into a single comprehensive execution cost evaluation formula, and selects the optimal scheduling scheme based on this comprehensive execution cost. Specifically, by changing the starting point, multiple sets of differentiated candidate scheduling schemes are generated. Then, a global selection is performed with the goal of minimizing the comprehensive execution cost, ultimately yielding the globally optimal scheduling result. Compared to schemes that only perform local optimization for a single domain (such as the optical or electrical domain), this approach balances the gains from optical domain acceleration, electrical domain data reuse, and cross-domain conversion overhead from a holistic perspective of the map, maximizing end-to-end comprehensive performance. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0027] Figure 1 This is a flowchart of the present invention; Figure 2 This is a structural diagram of the present invention; Figure 3 This is the initial photoelectric throughput cost map constructed in the operator scheduling method based on the present invention; Figure 4 For the reaction Figure 3 A schematic diagram of a continuous optical domain computational subgraph and a continuous electric domain computational subgraph obtained after merging optical domain operators and electric domain operators from the initial photoelectric throughput cost map. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0029] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0030] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0031] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0032] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0033] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0034] In this article, "computational graph" refers to the core data structure and method used in AI systems (deep learning frameworks) to represent mathematical computation processes using a directed acyclic graph (DAG). This type of DAG is called a computational graph. It is used to describe the computational logic and states of neural network models during training and inference, decomposing complex model computations into a series of basic, interrelated operations. For example, representing a mathematical computation process as a graph structure composed of nodes and edges answers the question, "In what order do these computations occur, and what are the dependencies between variables?"

[0035] Example 1: like Figure 1 As shown, this embodiment provides an operator scheduling method based on photoelectric throughput cost graphs, the specific steps of which include: Step S100: Analysis of the target neural network model.

[0036] In some embodiments, parsing the target neural network model file to be executed involves traversing a computation graph (which is existing technology and will not be described in detail here) to extract the types of all operators in the model, the data dependencies between operators, and recording the input / output tensor size, data precision, and computational parameters of each operator. For models with branching structures, the dependencies and control logic between branches are preserved; for fusionable adjacent operators, such as convolutional layers and batch normalization layers, operator fusion operations can be performed before parsing to reduce the number of nodes and improve scheduling efficiency. After parsing, the original computation graph data containing all operators and data dependency edges is output.

[0037] Step S200: Construct a photoelectric throughput cost map.

[0038] In some embodiments, a photoelectric throughput cost map is constructed using each operator as a node and the data dependencies between operators as connecting edges. Each node is labeled with the execution domain of the corresponding operator, which includes the optical domain and the electrical domain. Each connecting edge is labeled with a corresponding weight, which is used to characterize the photoelectric conversion cost (such as the conversion cost from light to electricity or from electricity to light) between the nodes at both ends of the connecting edge, thereby obtaining a map characterizing the photoelectric throughput cost corresponding to the target neural network model. For example, when the execution domains of the nodes at both ends of the connecting edge are the same, the photoelectric conversion cost between them is a first preset threshold, such as 0. Alternatively, when the nodes at both ends of the connecting edge belong to different execution domains, the photoelectric conversion cost is calculated based on the delay, energy consumption, and bandwidth occupied by the photoelectric conversion between them.

[0039] Specifically, using each operator obtained from the parsing as a node and the data dependencies between operators as directed edges, an initial photoelectric throughput cost graph is constructed, denoted as a directed graph G=(V,E), where V is the set of all nodes corresponding to all operators, and E is the set of all edges. Each node is labeled with its corresponding execution domain label, denoted as D(v...). i ), where v i For the i-th node, D(v) i The value of ) can be either in the optical domain or the electric domain.

[0040] The initial labeling rule is to label linear operators supported by the photonic computing array, such as matrix multiplication, vector-matrix multiplication, and matrix multiplication-addition, as optical domain operators. The following operators are labeled as electrical domain operators, serving as digital hard constraint boundaries for the electrical domain: Softmax operators performing exponential operations, division, or global reduction; LayerNorm operators performing mean, variance calculation, and normalization; activation operators performing piecewise judgments or complex nonlinear functions; control logic operators performing conditional branching, comparison, sorting, or dynamic indexing; and other operators not currently supported by the photonic computing array.

[0041] By setting a digital hard constraint boundary in the electrical domain, photoelectric conversion positions can be more rationally placed during subsequent graph optimization, reducing the probability of unreasonable photoelectric conversion positions. For example, during optical domain merging, when the continuous optical domain computation subgraph expands to the predecessor node of the digital hard constraint boundary in the electrical domain (e.g., the node whose calculation result is the input of the operator of the digital hard constraint boundary in the electrical domain, i.e., the corresponding node of the upstream adjacent operator of the operator of the digital hard constraint boundary in the electrical domain), the expansion stops and a photoelectric conversion trigger command is inserted at that position. This allows for the placement of corresponding photoelectric conversion modules, such as ADCs or DACs, at that position during the subsequent design of the photoelectric hybrid computing architecture. Furthermore, during the actual computation, the corresponding photoelectric conversion trigger command is sent to the photoelectric conversion module to trigger it to perform the corresponding photoelectric conversion. Subsequent steps will be described in detail and will not be repeated here.

[0042] It is also worth noting that there are no hard constraint boundaries in the optical domain. The reason for this is that the core of the hard constraint boundary in the electrical domain is functional infeasibility. Operators such as Softmax, LayerNorm, nonlinear activation, and conditional control rely on global reduction, high-precision numerical comparison, and branch judgment logic. However, current optoelectronic hybrid computing architectures cannot efficiently and accurately implement these operations from a physical perspective. This is a hard limitation that the optical domain simply cannot fulfill, and therefore it is a forced termination point for the expansion of continuous optical domain computational subgraphs.

[0043] However, electrical domain digital logic is a general-purpose computing platform. All linear operators supported by the optical domain can be computed using an electrical domain digital multiply-accumulate array. There are no operators that cannot be executed in the electrical domain and can only run in the optical domain. The hardware capabilities of the optical and electrical domains are inherently unequal: the optical domain is a dedicated acceleration unit that can only cover linear operations; the electrical domain is a general-purpose computing unit that can cover all operator types. Therefore, there are no hard constraints on the functional boundaries of the optical domain.

[0044] Furthermore, a corresponding weight is calculated and labeled for each connection edge. The weight is used to characterize the photoelectric conversion cost between the two nodes of the connection edge. The calculation method for the weight of each connection edge includes: Determine if the execution domain labels of the nodes at both ends of the connection edge are the same; if the execution domain labels are the same, the weight of the connection edge is reset to a first preset value (e.g., 0); if the execution domain labels are different, obtain the delay, energy consumption, and bandwidth occupancy generated by photoelectric conversion according to the data transmission direction corresponding to the connection edge, and calculate the photoelectric conversion cost of the connection edge as the weight through weighted calculation. Specifically, the photoelectric conversion cost uses the formula: calculate; in, The photoelectric conversion cost is the cost of performing photoelectric conversion between the i-th node and the j-th node at both ends of the connecting edge. The delay caused by the photoelectric conversion between the i-th node and the j-th node; The energy consumption generated by photoelectric conversion between the i-th node and the j-th node; This refers to the data buffering or on-chip bandwidth usage generated during photoelectric conversion between the i-th node and the j-th node. , These are preset weighting coefficients for the delay and energy consumption generated by photoelectric conversion, respectively. This is a preset weighting coefficient for bandwidth usage. The coefficient can be adjusted according to scheduling objectives; for example, it can be increased when prioritizing inference latency optimization. The value of can be increased when prioritizing energy efficiency optimization. The values ​​of are preferably obtained through simulation, and can also be obtained based on historical data statistical analysis.

[0045] In some embodiments, the composition of photoelectric conversion cost varies depending on the direction of data transmission. When data flows from the electrical domain to the optical domain, the photoelectric conversion cost includes the delay, power consumption, and bandwidth occupancy of the digital-to-analog converter, driving circuit, and electro-optic modulator; when data flows from the optical domain to the electrical domain, the photoelectric conversion cost includes the delay, power consumption, and bandwidth occupancy of the photodetector and analog-to-digital converter.

[0046] Of course, in other embodiments, the calculation may also need to take into account the tensor size corresponding to the connecting edge. That is, the larger the data volume, the higher the time and energy required for the conversion, and the higher the corresponding conversion cost.

[0047] Step S300: Generate a continuous optical domain computation subgraph.

[0048] In some embodiments, the photoelectric throughput cost map is traversed, and optical domain operator merging and electrical domain operator merging are performed respectively (i.e., the image labeled with the execution domain label and weight is optimized).

[0049] Specifically, the optical domain operators are merged starting from the operator nodes labeled as optical domain nodes, and within the preset optical domain operator merging constraints and electrical domain digital hard constraint boundaries, their downstream adjacent nodes are merged according to the topological order in the photoelectric throughput cost graph to generate at least one continuous optical domain computation subgraph. Specifically, step S300 includes the following steps: Step S301A: Select the starting point of the continuous optical domain computation subgraph. Traverse all nodes in the photoelectric throughput cost graph, randomly select or specify a node that has not yet been assigned to any continuous optical domain computation subgraph and whose execution domain label is optical domain, and use it as the starting point of the current continuous optical domain computation subgraph. Initialize the current continuous optical domain computation subgraph, which only includes this starting point.

[0050] Step S302A: Perform node merging and constraint verification for nodes labeled as optical domain. Following the topological order of the photoelectric throughput cost graph, sequentially select the adjacent nodes of the boundary nodes of the current continuous optical domain computation subgraph, and determine whether the adjacent node satisfies the preset optical domain operator merging constraint conditions. If all constraint conditions are satisfied, merge the adjacent node into the current continuous optical domain computation subgraph and continue attempting to merge the next adjacent node. If any constraint condition is not satisfied, stop expanding the current continuous optical domain computation subgraph.

[0051] Preferably, the downstream adjacent nodes of the boundary nodes of the current continuous optical domain computation subgraph are selected for merging. One of the inputs of the downstream adjacent nodes is the computation result of the boundary node, that is, merging is carried out sequentially in the downstream direction of data transmission. Of course, the predecessor node (whose output is one of the inputs of the boundary node) can also be selected for merging, that is, merging is carried out sequentially in the upstream direction of data transmission; or merging can be carried out sequentially in both upstream and downstream directions at the same time. The specific merging direction can be adaptively selected according to the position of the starting point. For example, if the starting point has no predecessor node, merging can be carried out sequentially in the downstream direction; if the starting point has no downstream adjacent nodes, merging can be carried out sequentially in the upstream direction; if the starting point has both predecessor and downstream adjacent nodes, merging can be carried out sequentially in any direction, or merging can be carried out in both directions at the same time.

[0052] Furthermore, the graph of this application can be applied to the operator path planning stage as well as the specific computation task execution stage. Preferably, in the operator path planning stage, the expansion direction of the subgraph can be selected in any one direction or in both directions simultaneously, depending on actual needs; while in the computation task execution stage, the expansion is carried out by merging directions sequentially downstream.

[0053] For example, see Figure 3 When taking linear operator A, which has not been assigned to any graph, as the starting point (at this time, linear operator A itself is the boundary node of the current subgraph), since linear operator B, which is the downstream adjacent node of linear operator A (that is, the downstream node that has a dependency relationship with the linear operator), has not been assigned to any subgraph and meets the merging constraint, it is merged with linear operator A.

[0054] In some embodiments, the optical domain operator merging constraints include the following five items, one or more of which can be selected and used in combination depending on the hardware architecture: The first item includes a photonic computing array capacity constraint: the photonic computing array resources required to execute the current continuous optical domain computing subgraph are less than or equal to the currently available array capacity. The array resources required for the current continuous optical domain computing subgraph can be obtained by summing the number of optical computing units required for all linear operators within the current continuous subgraph. The available array capacity is the total number of idle optical computing units in the current chip.

[0055] The second item includes the dynamic range constraint of the analog signal: during the execution of the current continuous optical domain computation subgraph, the dynamic range of the analog optical signal always remains within the allowable operating range of the optoelectronic device. The dynamic range can be derived step by step based on the numerical range of the input tensor, the weight range of each layer of operators, and the operational gain, ensuring that the power of the optical signal does not exceed the linear operating range of the electro-optic modulator and photodetector, thus avoiding signal distortion.

[0056] The third item includes a cumulative error constraint: the cumulative error of the current continuous optical domain computation subgraph is less than or equal to a preset error threshold. The cumulative error includes nonlinear noise and phase error introduced by each optical computation unit within the current continuous optical domain subgraph, as well as propagation loss caused by the optical signal propagating in the waveguide. These errors accumulate progressively with the increase in the number of nodes within the current continuous subgraph. This error threshold is preset according to the accuracy requirements of the neural network to ensure that the accuracy of the results after continuous optical domain computation meets the inference requirements.

[0057] The fourth constraint is waveguide routing resource constraints: the optical interconnect resources required for the current continuous optical domain computation subgraph are less than or equal to the currently available routing resources. These optical interconnect resources include the silicon-based waveguide length, the number of optical couplers, and the number of wavelength division multiplexing channels required to connect each optical computation node. It must be ensured that all nodes within the current continuous optical domain computation subgraph can be physically connected via on-chip optical waveguides.

[0058] The fifth constraint is operator compatibility: all operators corresponding to nodes in the current continuous optical domain computation subgraph must be linear operators supported by the photonic computing array. This constraint ensures that operators corresponding to nodes included in the continuous optical domain computation subgraph can be correctly executed in the optical domain, preventing operators that cannot be implemented in the optical domain from being included in the continuous optical domain path.

[0059] For example, see Figure 3 In the initial spectrum, the Softmax operator and LayerNorm operator of the four operators are scheduled to the electrical domain, while the other two operators, linear operator A and linear operator B, are scheduled to the optical domain. Other operators, such as on-chip buffer and output buffer, are also scheduled to the electrical domain.

[0060] In this graph, the output of linear operator A is input into linear operator B for computation. Therefore, the two linear operators are dependent (i.e., linear operators A and B are adjacent and scheduled to OPU1 and OPU2, respectively, to perform linear computations in the optical domain). The weight w between these two operator nodes is marked as 0 on the edge connecting them (since both linear operators A and B are scheduled to the optical domain, optical domain merging can be performed, resulting in...). Figure 4 The continuous optical domain computation subgraph shown in the upper wireframe). The weights between two operators with dependencies (i.e., adjacent operators): the Softmax operator and the linear operator A, are both weighted at w=0.4; the weight between the LayerNorm operator and the on-chip cache is also w=0.4. (Since both the LayerNorm operator and the on-chip cache are scheduled to the electrical domain and meet the merging conditions, electrical domain merging can be performed, resulting in...) Figure 4The continuous electrical domain computation subgraph shown in the wireframe below has a weight w=0.7 between linear operator A and the output buffer, a weight w=2.6 between linear operator A and the LayerNorm operator, and a weight w=2.1 between the on-chip buffer and linear operator B.

[0061] See Figure 3 As shown, since linear operator A is scheduled to perform computation in the optical domain, while the LayerNorm operator is scheduled to perform computation in the electrical domain, and there is a dependency between the two, an ADC needs to be set between them to convert the analog optical signal obtained by OPU1 executing linear operator A into an analog electrical signal as one of the inputs of the LayerNorm operator. Therefore, this position 1 needs to insert a photoelectric conversion trigger instruction. Accordingly, based on this diagram, a corresponding ADC will be inserted in the photoelectric hybrid computing structure to convert the analog optical signal into an analog electrical signal, such as... Figure 3 As shown, an ADC is inserted at position 1 (of course, if there is already an ADC at this position, you only need to send the corresponding photoelectric conversion trigger command to the ADC to control it to perform the corresponding photoelectric conversion).

[0062] Similarly, such as Figure 3 and Figure 4 As shown, position 2 between the on-chip cache TMEM (Tensor Memory) and the linear operator B requires the insertion of a photoelectric conversion trigger instruction. Correspondingly, a corresponding DAC will be inserted at position 2 in the photoelectric hybrid computing structure to convert the analog electrical signal into an analog optical signal. Similarly... Figure 4 The corresponding photoelectric conversion modules need to be inserted in positions 3 and 4.

[0063] In some embodiments, since various resources and states change rapidly during system operation, for each continuous optical domain computation subgraph, before merging adjacent nodes with the execution domain label "optical domain," it is determined whether the current continuous optical domain computation subgraph meets one or more of the first to fourth items mentioned above. If any item is not met, the expansion stops, meaning the adjacent node is not included in the current continuous optical domain computation subgraph. Furthermore, after merging, it is also possible to again determine whether the merged current continuous optical domain computation subgraph meets one or more of the first to fourth items mentioned above. If any item is not met, the expansion stops, meaning the adjacent node is removed from the current continuous optical domain computation subgraph.

[0064] Accordingly, after stopping expansion, the adjacent node can be used as the starting point of the next continuous optical domain computation subgraph, or as a candidate node to be merged into other continuous optical domain computation subgraphs, depending on the actual situation. Of course, in some other embodiments, after stopping expansion, the adjacent node can also be scheduled from the optical domain to the electrical domain. More preferably, it can be determined whether the benefit of performing the adjacent node in the optical domain is higher than the photoelectric conversion cost of performing one photoelectric conversion. If so, the adjacent node is used as the starting point of the next continuous optical domain computation subgraph, or as a candidate node to be merged into other continuous optical domain computation subgraphs, depending on the actual situation; otherwise, the adjacent node is scheduled from the optical domain to the electrical domain.

[0065] In other embodiments, during the optical domain operator merging process, at least one candidate continuous optical domain computation subgraph can be generated for each starting point by setting different extension stopping constraints, and the profit value of each candidate continuous optical domain computation subgraph can be calculated. Then, candidate continuous optical domain computation subgraphs with profit values ​​greater than or equal to preset target profit are selected as the target continuous optical domain computation subgraph corresponding to that starting point.

[0066] Preferably, the expansion stopping constraint can be at least one of the optical domain operator merging constraint conditions. If there are two or more, the fifth condition is the basic necessary condition, and the remaining conditions can be selected according to actual needs. For example, the expansion stopping constraint condition set at the first type of starting point in the spectrum only includes the fifth condition, the expansion stopping constraint condition for the second type of starting point device includes the fifth condition as well as any one of the remaining conditions in the optical domain operator merging constraint condition, and the expansion stopping constraint condition set at the third type of starting point includes the fifth condition as well as any two, three, or four of the remaining conditions.

[0067] Specifically, the reward value of the candidate continuous optical domain computation subgraph is calculated using the formula: calculate; in, The higher the profit value, the higher the overall profit of merging the nodes in this segment into a continuous optical domain computational subgraph. Compared to the photoelectric conversion cost calculated separately for each node corresponding to the operator in this candidate continuous optical domain computation subgraph before optical domain operator merging, this represents the reduction in photoelectric conversion cost, including ADC conversion cost and DAC conversion cost, achieved by performing continuous optical domain computation based on this candidate continuous optical domain computation subgraph (i.e., all operators corresponding to all nodes in this subgraph are executed within the optical domain without intermediate photoelectric conversion). The computational cost reduction for performing the computation of the candidate continuous optical domain subgraph in the optical domain compared to performing the corresponding operators of each node in the subgraph in the electrical domain. This refers to the cost of optical routing resources used for continuous optical domain computation based on this candidate continuous optical domain computation subgraph. This is a preset risk penalty term used to characterize the potential risks arising from the error of the candidate continuous optical domain computation subgraph approaching a preset threshold and the resource consumption approaching a preset upper limit.

[0068] Step S303A: Set the photoelectric conversion position. The photoelectric conversion position is set at the boundary of the current continuous optical domain computation subgraph. Specifically, at the boundary of the continuous optical domain computation subgraph, it is determined whether the execution domain label of the subsequent node connected to the boundary node (i.e., the node connected to the boundary node along the merging direction, such as the downstream adjacent node or predecessor node of the boundary node) is in the electrical domain. If the subsequent node is in the electrical domain, a photoelectric conversion trigger instruction is inserted at the boundary to mark the photoelectric conversion position. Specifically, this includes an analog-to-digital conversion trigger instruction, used to control the hardware to convert the optical domain analog output into a digital electrical signal and send it to the downstream adjacent node in the electrical domain for execution. Of course, in the subsequent photoelectric computation structure design, the corresponding ADC or DAC can be set at the appropriate position according to the photoelectric conversion trigger instruction in the graph, thereby triggering the ADC or DAC to perform photoelectric conversion under the triggering action of the photoelectric conversion trigger instruction.

[0069] It is worth noting that the "photoelectric conversion position" and "photoelectric conversion trigger command" mentioned in this invention can actually be either analog-to-digital conversion or digital-to-analog conversion, depending on the direction of signal flow. For example, a signal flowing from the optical domain to the electrical domain is an analog-to-digital conversion, while a signal flowing from the electrical domain to the optical domain is a digital-to-analog conversion.

[0070] Step S304A: Traverse all nodes whose execution domain label is optical domain, and repeat steps S302A to S303A to generate a new continuous optical domain computation subgraph.

[0071] In some embodiments, the next undivided node whose execution domain is optical domain is selected as the starting point of a new continuous optical domain computation subgraph along the topological order, and steps S302A to S303A are repeated to generate a new continuous optical domain computation subgraph until all nodes in the photoelectric throughput cost map whose execution domain label is optical domain are divided into the corresponding continuous optical domain computation subgraph.

[0072] Furthermore, in the process of merging optical domain operators, in addition to considering the aforementioned merging constraints, it is also necessary to consider the electrical domain digital hard constraint boundary. Specifically, for operators pre-marked as electrical domain digital hard constraint boundaries, a forced termination rule is executed during the expansion of the continuous optical domain computation subgraph. That is, when the continuous optical domain computation subgraph expands to the predecessor node of the electrical domain digital hard constraint boundary (this predecessor node serves as the boundary node of the current continuous optical domain computation subgraph, is connected as the corresponding node of the electrical domain digital hard constraint boundary, and its output will be used as one of the inputs of the corresponding node of the electrical domain digital hard constraint boundary), the expansion of the continuous optical domain computation subgraph stops regardless of whether the constraint conditions are met, and a photoelectric conversion trigger command is inserted at this position to ensure that the operator serving as the electrical domain digital hard constraint boundary is executed in the electrical domain. Specifically, the type of electrical domain digital hard constraint boundary operator is described in step S200, and will not be repeated here.

[0073] In some embodiments, during actual operation, after the operator serving as the digital hard constraint boundary of the electrical domain completes the electrical domain calculation, or during the subgraph expansion process, after inserting a photoelectric conversion trigger instruction at the corresponding position, it is further determined whether the subsequent nodes (such as downstream adjacent nodes) of the node corresponding to the operator serving as the digital hard constraint boundary of the electrical domain meet the optical domain execution conditions. If the operator corresponding to the subsequent node is a linear operator executable in the optical domain, and the benefit of subsequent continuous optical domain execution is higher than the photoelectric conversion cost of one electro-to-optical conversion (if the computational cost required for the operator corresponding to the downstream adjacent node to execute in the optical domain is less than or equal to the photoelectric conversion cost of one electro-to-optical conversion), then a photoelectric conversion trigger instruction and a modulation instruction are inserted at the corresponding position to convert the digital result back into an optical signal for execution in the optical domain. Of course, if the operator corresponding to the subsequent node, such as the downstream adjacent node, is a linear operator executable in the optical domain, but the benefit of subsequent continuous optical domain execution is less than or equal to the photoelectric conversion cost of one electro-to-optical conversion, then a photoelectric conversion trigger instruction is not inserted at that position, but the subsequent operator is scheduled to the electrical domain.

[0074] In some embodiments, the electric domain operator merging is performed by starting with nodes labeled as electric domains and merging subsequent nodes that have not been partitioned and whose execution domain label is electric domain, according to the topological order in the photoelectric throughput cost graph, such as downstream adjacent nodes and / or upstream predecessor nodes. Under preset electric domain operator merging constraints, at least one continuous electric domain computation subgraph is generated. The steps and principles for generating the continuous electric domain computation subgraph are largely the same as those for the optical domain computation subgraph, only the constraints are different, specifically including: S301B: Select an undivided node with the domain label "electric domain" as the starting point of the current continuous electric domain computation subgraph.

[0075] S302B: According to the topological order in the photoelectric throughput cost graph, determine whether the adjacent nodes satisfy the preset electric domain operator merging constraint conditions. If they satisfy the conditions, merge the adjacent nodes into the current continuous electric domain calculation subgraph. If they do not satisfy the conditions, stop expanding the current continuous electric domain calculation subgraph.

[0076] Following the previous example, see [link / reference]. Figure 4 When the LayerNorm operator is used as the starting point, and its downstream neighboring node has not yet been assigned to any continuous electrical domain computation subgraph and its execution domain is the electrical domain, and it meets the electrical domain operator merging constraint, then it is merged with the starting point using the electrical domain operator to obtain a continuous electrical domain computation subgraph. Accordingly, photoelectric conversion trigger instructions need to be inserted between the boundary nodes of adjacent continuous electrical domain computation subgraphs and continuous optical domain computation subgraphs.

[0077] In this embodiment, the electrical domain operator merging constraint is a mandatory verification rule during the expansion process of the continuous electrical domain computation subgraph. The optical domain operator merging constraint ensures that continuous execution in the optical domain does not break the boundary between optical physics and photonic hardware, while the electrical domain operator merging constraint ensures that after continuous merging of electrical domain operators, the overall performance will not be degraded due to problems such as resource overload, storage overflow, and bandwidth congestion, thus ensuring that the merging benefits of the continuous electrical domain computation subgraph always exceed the merging costs.

[0078] In some embodiments, the electric domain operator merging constraint includes at least one of the following: The first condition is that the computational resources required to execute the current continuous electrical domain computation subgraph are less than or equal to the upper limit of the parallelism of the electrical domain digital logic units. The second condition is that the size of the intermediate results temporarily stored during the current continuous electrical domain calculation subgraph is less than or equal to the on-chip cache capacity. The third item: The bus bandwidth required to execute the current continuous electrical domain calculation subgraph is less than or equal to the bandwidth limit of the on-chip network; Fourth item: The total execution delay of the current continuous electrical domain computation subgraph is less than or equal to a preset delay threshold; Fifth item: The hardware accelerators required for the current continuous electric domain computation subgraph match the currently available accelerator resources.

[0079] In other embodiments, during the merging of electric domain operators, for each starting point, at least one candidate continuous electric domain computation subgraph can be generated by setting different extended stopping constraints (such as at least one of the aforementioned electric domain operator merging constraints; for example, the extended stopping constraints set for some starting points are any one of the electric domain operator merging constraints, and the extended stopping constraints set for other starting points are any two, three, or five of the electric domain operator merging constraints, which are adjusted according to the actual working conditions). Candidate continuous electric domain computation subgraphs with a benefit value greater than or equal to a preset target benefit are selected from these and used as the target continuous electric domain computation subgraph corresponding to that starting point.

[0080] Specifically, the reward value of the candidate continuous electric domain computation subgraph is calculated using the formula: calculate; in, This represents the electric domain benefit value. A higher benefit value indicates a higher overall benefit from merging this segment of operators into a continuous electric domain computational subgraph. To reduce the cost of ADC and DAC conversion by performing continuous electrical domain operations; Compared to executing these electrical domain operators separately, executing them sequentially may reduce context switching overhead or provide benefits from improved data locality. The costs associated with potential competition for electricity resources following the merger; Risks associated with execution in the electrical domain.

[0081] S303B sets the photoelectric conversion position at the boundary of the current continuous electric domain computation subgraph.

[0082] In some embodiments, the process of setting the photoelectric conversion position specifically includes: at the boundary of the continuous electrical domain computation subgraph, determining whether the execution domain label of the downstream adjacent node connected to the boundary node is an optical domain; if the downstream node is an optical domain, then inserting a photoelectric conversion trigger instruction at the boundary as the photoelectric conversion trigger position.

[0083] S304B: Select the next undivided node with the execution domain label "electric domain" as the new starting point, and repeat steps S302B to S303B until all nodes with the execution domain label "electric domain" are divided into the corresponding continuous electric domain computation subgraph.

[0084] Understandably, based on the same principle, after completing the electrical domain merging, the subsequent nodes of its boundary nodes are further judged, such as whether the downstream adjacent nodes meet the optical domain execution conditions. If the operator corresponding to the subsequent node is a linear operator executable in the optical domain, and the benefit of subsequent continuous optical domain execution is higher than the photoelectric conversion cost of one electro-optical conversion (e.g., the execution cost of the operator corresponding to the subsequent node in the optical domain is less than the photoelectric conversion cost of electro-optical conversion), then photoelectric conversion trigger instructions and modulation instructions are inserted at the corresponding positions to convert the digital results back into optical signals for execution in the optical domain. Alternatively, on-chip buffer write positions can be inserted to temporarily store the intermediate results calculated in the electrical domain.

[0085] In this embodiment, after optimizing the photoelectric throughput cost map by combining the above-mentioned electric domain operators and optical domain operators, a corresponding scheduling scheme can be generated based on the photoelectric throughput cost map.

[0086] Example 2: This invention also provides another operator scheduling method based on photoelectric throughput cost graphs, which includes steps S100-S300 of the above embodiments. The difference is that the operator scheduling method based on photoelectric throughput cost graphs in this embodiment can also change the starting point of at least one continuous optical domain computation subgraph and / or at least one continuous electrical domain computation subgraph, thereby generating different candidate scheduling schemes. Of course, these subgraphs (such as continuous optical domain computation subgraphs and continuous electrical domain computation subgraphs) are not isolated. Changing the starting point of one subgraph may cause the starting points of other dependent subgraphs to also adaptively change.

[0087] Specifically, the operator scheduling method further includes: step S400, generating multiple sets of candidate scheduling schemes.

[0088] In some embodiments, for the optical domain and the electrical domain, different nodes can be used as starting points to repeat step S300, generating multiple continuous optical domain computation subgraphs and multiple continuous electrical domain computation subgraphs to construct multiple sets of candidate scheduling schemes; each set of candidate scheduling schemes includes at least one continuous optical domain computation subgraph and / or at least one continuous electrical domain computation subgraph and the photoelectric conversion position corresponding to its boundary.

[0089] Preferably, both the continuous optical domain computation subgraph and the continuous electrical domain computation subgraph adopt a greedy expansion mechanism under topological order, that is, starting from their respective starting points, subsequent nodes are merged sequentially along the data flow direction, such as downstream adjacent nodes or predecessor nodes, until the corresponding constraint condition is triggered and the expansion stops. Preferably, the priority of merging optical domain operators is higher than that of merging electrical domain operators, that is, optical domain operator merging is completed first, and then electrical domain operator merging is performed.

[0090] In both optical and electrical operator merging processes, due to limited global resources, subgraphs that initiate expansion first will occupy available resources and obtain more lenient expansion boundaries. Subgraphs initiated later can only complete merging within the remaining resource range, affecting their expansion scale and boundaries. Therefore, choosing different nodes as the starting points for subgraph expansion directly changes the merging priority and global resource allocation order, ultimately resulting in different full graph partitioning results and corresponding scheduling strategies. For example, in the optical domain, photonic computing array capacity, waveguide routing resources, and error budget are all globally finite resources; different starting points will lead to differences in the number, size, and boundary positions of consecutive optical domain computing subgraphs. Similarly, in the electrical domain, computational parallelism, on-chip cache capacity, and bus bandwidth are all globally finite resources; different starting points will lead to differences in the degree of merging, cache reuse rate, and resource congestion of consecutive electrical domain computing subgraphs.

[0091] Since the boundaries of the subgraph directly correspond to the insertion positions of photoelectric conversions, changes in the boundaries will directly change the total number of global photoelectric conversions and the total cost. Therefore, different combinations of starting points will eventually form scheduling schemes with different performance trade-offs.

[0092] In this embodiment, multiple continuous optical domain computation subgraphs and / or multiple continuous electrical domain computation subgraphs are generated by iterating through and changing the starting point, and completing the full graph partitioning round by round. First, all nodes marked as optical domains in the initial photoelectric throughput cost map obtained from the computation graph are traversed to form a candidate set of starting points. Each node can serve as the starting point for the first round of subgraph expansion. Then, a starting point is selected in sequence, and the entire process of step S300 is executed once, until all nodes marked as optical domains are partitioned into the corresponding continuous optical domain computation subgraphs, resulting in a subgraph set covering all nodes marked as optical domains; and until all nodes marked as electrical domains are partitioned into the corresponding continuous electrical domain computation subgraphs, resulting in a subgraph set covering all nodes marked as electrical domains. That is, each time the full graph partitioning is completed, a complete set of candidate scheduling schemes is generated. Then, the above process is repeated by changing the execution domain label to different nodes marked as optical domains as the starting point, and finally, multiple sets of independent candidate scheduling schemes with different partitioning strategies are obtained.

[0093] Each candidate scheduling scheme is a complete scheduling strategy covering all nodes in the entire graph. It includes several non-overlapping continuous optical domain computation subgraphs that collectively cover all optical domain nodes, and several non-overlapping continuous electrical domain computation subgraphs that collectively cover all electrical domain nodes. Each subgraph explicitly records its included operator node range, required hardware resources, cumulative error / congestion level, and expected execution cost, serving as the basis for subsequent calculations of operator execution costs and verification of constraint compliance. It also includes the set of photoelectric conversion locations.

[0094] For each boundary between the continuous optical domain computational subgraph and the corresponding continuous electrical domain computational subgraph, the node interface where a photoelectric conversion operation needs to be inserted is clearly marked. The locations where the output of the continuous optical domain computational subgraph connects to nodes marked in the electrical domain are correspondingly marked with photoelectric conversion locations used for analog-to-digital conversion; similarly, the locations where the output of the continuous electrical domain computational subgraph connects to nodes marked in the optical domain are correspondingly marked with photoelectric conversion locations used for digital-to-analog conversion. All photoelectric conversion locations collectively constitute the basis for calculating the photoelectric conversion overhead of this scheme.

[0095] In some embodiments, different candidate scheduling schemes differ in dimensions such as the number of subgraphs, average subgraph length (e.g., the number of nodes included), total number of photoelectric conversions, resource occupancy distribution, and degree of error accumulation: some schemes have a high degree of subgraph merging (e.g., fewer subgraphs included) and fewer photoelectric conversions, but the resource occupancy and error risk of a single subgraph are higher; some schemes have finer subgraph splitting and lower single-graph operation risk, but more photoelectric conversions and higher total conversion overhead.

[0096] Step S500: Select the optimal target scheduling scheme.

[0097] In some embodiments, by generating multiple sets of candidate scheduling schemes, a horizontal comparison can be made using a unified comprehensive execution cost index to select the globally optimal (e.g., the lowest comprehensive execution cost) scheduling result, thereby significantly improving the overall performance of the final scheduling scheme without increasing the complexity of the core algorithm.

[0098] Specifically, based on the execution cost of the corresponding operator of each node and the photoelectric conversion cost of each photoelectric conversion position, the comprehensive execution cost of each group of candidate scheduling schemes is calculated, and the candidate scheduling scheme with the lowest comprehensive execution cost is selected as the target scheduling scheme.

[0099] For example, the comprehensive execution cost of candidate scheduling schemes is calculated using the following formula: calculate; in, The overall execution cost of candidate scheduling schemes, For the i-th node The execution cost in the corresponding execution domain specifically includes: the weighted sum of execution latency and energy consumption; For the i-th node The execution domain label; V is the set of all nodes; For connecting edges The photoelectric conversion cost between the i-th node and the j-th node at both ends is the weight of the connecting edge calculated in step S200; E is the set of all connecting edges. The risk cost of error accumulation and resources approaching the limit is generated by factors such as error accumulation approaching the threshold and hardware resources approaching the limit. The preset risk penalty weight is used to control the proportion of risk factors in the total cost.

[0100] In some embodiments, after calculating the comprehensive execution cost of all candidate scheduling schemes, the scheme with the minimum comprehensive execution cost is selected as the final target scheduling scheme, and a complete hardware control instruction sequence is generated based on the target scheduling scheme, including execution domain allocation instructions, photoelectric conversion trigger instructions, data routing instructions and resource configuration instructions for each operator.

[0101] To further illustrate the execution process of this method, we will take a self-attention calculation segment in the Transformer model as an example for detailed explanation. The operators in this calculation segment are as follows: first linear projection layer, second linear projection layer, Softmax layer, LayerNorm layer, third linear projection layer, and fourth linear projection layer, and the operators are connected in sequence.

[0102] During the model parsing phase, 6 operator nodes and 5 connection edges were obtained. The input and output dimensions of each linear projection layer were recorded as 1024, and the data precision was FP16. Softmax and LayerNorm are global statistical operators.

[0103] During the cost graph construction phase, the execution domain labels of the four linear projection layers are initially marked as optical domains, and the Softmax and LayerNorm layers are marked as electrical domains, which are then determined as the digital hard constraint boundaries of the electrical domains. The weights of each connection edge are calculated. The connection edge from the second linear projection layer to the Softmax layer is in the optical-to-electrical conversion direction. Based on the tensor dimension, the conversion delay is calculated to be 8 clock cycles, the conversion energy consumption is 12 nanojoules, and the bandwidth occupancy is 1024×16 bits. These values ​​are then substituted into the edge weight formula to calculate the edge weight. The connection edge from the LayerNorm layer to the third linear projection layer is in the electrical-to-optical conversion direction, and the corresponding edge weight is calculated similarly. The connection edges between other adjacent linear layers are initially all within the optical domain, with a weight of 0.

[0104] During the subgraph generation phase, starting from the first linear projection layer, an attempt is made to merge the second linear projection layer. The merging constraints are verified: the total number of optical computing units required by the two linear layers is less than the array capacity, the signal dynamic range is within the device's operating range, the cumulative error is below the error threshold, waveguide routing resources are sufficient, and all operators are linearly compatible. All constraints are satisfied, therefore, the first continuous optical domain computational subgraph is formed. The attempt to merge the downstream Softmax layer continues. Since Softmax is a hard digital constraint boundary in the electrical domain, the expansion of the first optical domain subgraph is stopped, and an analog-to-digital conversion trigger instruction is inserted at the output of the second linear projection layer.

[0105] Then, starting from the third linear projection layer, the fourth linear projection layer is merged, and all constraints are verified to be satisfied, forming the second continuous optical domain computation subgraph. Since there are no more operators downstream of the subgraph, corresponding transformation instructions are set at the subgraph boundary according to the subsequent node type.

[0106] During the candidate scheme generation and screening stage, multiple candidate schemes are generated. For example, Scheme 1 is the division result of the above two continuous optical domain subgraphs; Scheme 2 is to classify the second linear projection layer into the electric domain and retain only the first linear projection layer as an independent optical domain subgraph.

[0107] The overall execution cost of the two schemes was calculated separately. Scheme 1 involves 2 photoelectric conversions and 4 linear layers in the optical domain; Scheme 2 involves 4 photoelectric conversions, 2 linear layers in the optical domain, and 2 linear layers in the electrical domain. Calculations showed that Scheme 1 has a lower conversion cost, higher optical domain computational benefit, and a lower overall execution cost. Therefore, Scheme 1 was selected as the target scheduling scheme.

[0108] During the instruction generation phase, control instructions are generated based on the target scheduling scheme. The first and second linear projection layers are scheduled to be executed by the photonic computing array, with intermediate data propagated directly in cascaded optical signals. After the output of the second linear projection layer, analog-to-digital conversion is triggered, and the data is sent to the electrical domain digital logic unit to execute Softma and LayerNorm sequentially. After the output of LayerNorm, digital-to-analog conversion and electro-optic modulation are triggered, and the data re-enters the optical domain for continuous execution by the third and fourth linear projection layers. Finally, the calculation result is output.

[0109] Example 3: Figure 2 As shown, based on the operator scheduling method based on the photoelectric throughput cost graph in Embodiment 1 above, this embodiment also provides an operator scheduling system based on the photoelectric throughput cost graph, which includes: The parsing module is used to parse the target neural network model and obtain the types of each operator in the target neural network model and the data dependencies between operators. The cost graph construction module is used to construct a photoelectric throughput cost graph with each operator as a node and the data dependencies between operators as connecting edges. Each node is labeled with the execution domain label of the corresponding operator, and the execution domain label includes the optical domain and the electrical domain. Each connecting edge is labeled with a corresponding weight, and the weight is used to characterize the photoelectric conversion cost between the nodes at both ends of the connecting edge. Specifically, the calculation method of the photoelectric conversion cost is described in Embodiment 1 above, and will not be repeated here. The subgraph generation module is used to traverse the photoelectric throughput cost map and perform optical domain operator merging and electrical domain operator merging respectively. Specifically, optical domain merging starts from nodes labeled as optical domains and merges adjacent nodes according to the topological order in the photoelectric throughput cost map, generating at least one continuous optical domain computation subgraph within preset optical domain operator merging constraints and electrical domain digital hard constraint boundaries. Electrical domain merging starts from nodes labeled as electrical domains and merges adjacent nodes according to the topological order in the photoelectric throughput cost map, generating at least one continuous electrical domain computation subgraph under preset electrical domain operator merging constraints. Specifically, the process of performing optical domain operator merging and electrical domain operator merging is described in Embodiment 1 above, and will not be repeated here. The candidate scheduling scheme generation module is used to change different nodes as starting points to trigger the subgraph generation module to generate multiple continuous optical domain computation subgraphs and continuous electrical domain computation subgraphs, thereby constructing multiple sets of candidate scheduling schemes; each set of candidate scheduling schemes includes several continuous optical domain computation subgraphs and / or continuous electrical domain computation subgraphs and the photoelectric conversion positions corresponding to their boundaries; The target scheduling scheme selection module is used to calculate the comprehensive execution cost of each group of candidate scheduling schemes based on the execution cost of the corresponding operator of each node and the conversion cost of each photoelectric conversion position, and select the candidate scheduling scheme with the lowest comprehensive execution cost as the target scheduling scheme.

[0110] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0112] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An operator scheduling method based on photoelectric throughput cost graph, characterized in that, Includes the following steps: S100, parse the target neural network model and obtain the type of each operator in the target neural network model and the data dependencies between operators; S200: Using each operator as a node and the data dependencies between operators as connecting edges, a photoelectric throughput cost map is constructed; wherein each node is labeled with the execution domain label of the corresponding operator, and the execution domain label includes the optical domain and the electrical domain; each connecting edge is labeled with a corresponding weight, and the weight is used to characterize the photoelectric conversion cost between the two nodes of the connecting edge. S300, traverse the photoelectric throughput cost map, and perform optical domain operator merging with nodes marked as optical domains as starting points to obtain at least one continuous optical domain computation subgraph. Perform electrical domain operator merging with nodes marked as electrical domains as starting points to obtain at least one continuous electrical domain computation subgraph. S400, Repeat step S300 by changing different nodes as starting points to generate multiple continuous optical domain computation subgraphs and continuous electrical domain computation subgraphs to construct multiple sets of candidate scheduling schemes; Each candidate scheduling scheme includes at least one continuous optical domain computation subgraph and at least one continuous electric domain computation subgraph and the photoelectric conversion positions corresponding to their boundaries; S500 calculates the comprehensive execution cost of each candidate scheduling scheme based on the execution cost of the corresponding operator of each node and the photoelectric conversion cost of each photoelectric conversion position, and selects the candidate scheduling scheme with the lowest comprehensive execution cost as the target scheduling scheme.

2. The operator scheduling method based on photoelectric throughput cost graph according to claim 1, characterized in that, Step S200, specifically the calculation of the weight of each of the connecting edges, includes: Determine whether the execution domain labels of the nodes at both ends of the connection edge are the same; if the execution domain labels are the same, reset the weight of the connection edge to the first preset value; if the execution domain labels are different, obtain the delay, energy consumption and bandwidth occupation generated by photoelectric conversion between the two nodes according to the data transmission direction corresponding to the connection edge, and obtain the photoelectric conversion cost of the connection edge as the weight through weighted calculation. The photoelectric conversion cost is calculated using the following formula: calculate; in, The photoelectric conversion cost between the i-th node and the j-th node at both ends of the connection edge. The delay caused by the photoelectric conversion between the i-th node and the j-th node; The energy consumption generated by photoelectric conversion between the i-th node and the j-th node; This refers to the data buffering or on-chip bandwidth usage generated during photoelectric conversion between the i-th node and the j-th node. , These are preset weighting coefficients for the delay and energy consumption generated by photoelectric conversion, respectively. This is a preset weighting coefficient for the on-chip bandwidth usage.

3. The operator scheduling method based on photoelectric throughput cost graph according to claim 1, characterized in that, The step of generating the continuous optical domain computational subgraph in step S300 includes: S301A: Select an undivided node with the execution domain label as optical domain as the starting point of the current continuous optical domain computation subgraph; S302A: According to the topological order in the photoelectric throughput cost map, determine whether the adjacent nodes of the starting point satisfy the preset optical domain operator merging constraint conditions. If they satisfy the conditions, merge the adjacent nodes into the current continuous optical domain calculation subgraph. If they do not satisfy the conditions, stop expanding the current continuous optical domain calculation subgraph. S303A, set the photoelectric conversion position at the boundary of the current continuous optical domain computation subgraph; S304A: Select the next undivided node with the execution domain label of optical domain as the new starting point, and repeat steps S302A to S303A until all optical domain operator nodes are divided into the corresponding continuous optical domain computation subgraph.

4. The operator scheduling method based on photoelectric throughput cost graph according to claim 1, characterized in that, The step of generating the continuous electric domain computation subgraph in step S300 includes: S301B, Select an undivided node with the execution domain label as the electric domain as the starting point of the current continuous electric domain computation subgraph; S302B: According to the topological order in the photoelectric throughput cost graph, determine whether the adjacent nodes of the starting point satisfy the preset electric domain operator merging constraint conditions. If they satisfy the conditions, merge the adjacent nodes into the current continuous electric domain calculation subgraph. If they do not satisfy the conditions, stop expanding the current continuous electric domain calculation subgraph. S303B, Set the photoelectric conversion position at the boundary of the current continuous electric domain calculation subgraph; S304B: Select the next unpartitioned node with the domain label "electric domain" as the new starting point, and repeat steps S302B to S303B until all electric domain operator nodes are partitioned into the corresponding continuous electric domain computation subgraph.

5. The operator scheduling method based on photoelectric throughput cost graph according to claim 3, characterized in that, The optical domain operator merging constraint includes at least one of the following: The photonic computing array resources required to execute the current continuous optical domain computation subgraph are less than or equal to the currently available array capacity; During the execution of the current continuous optical domain computation subgraph, the dynamic range of the simulated optical signal is within the operating range allowed by the optoelectronic device; The cumulative error of the current continuous optical domain computation subgraph is less than or equal to a preset error threshold, and the cumulative error includes the cumulative values ​​of nonlinear noise, phase error and propagation loss. The optical interconnect resources required for the current continuous optical domain computation subgraph are less than or equal to the currently available routing resources; The operators corresponding to all nodes in the current continuous optical domain computation subgraph belong to the linear operator types supported by the photonic computing array.

6. The operator scheduling method based on photoelectric throughput cost graph according to claim 4, characterized in that, The electric domain operator merging constraint includes at least one of the following: The computational resources required to execute the current continuous electrical domain computation subgraph are less than or equal to the upper limit of the parallelism of the electrical domain digital logic unit; The size of the intermediate results temporarily stored during the execution of the current continuous electrical domain computation subgraph is less than or equal to the on-chip cache capacity; The bus bandwidth required to execute the current continuous electrical domain computation subgraph is less than or equal to the bandwidth limit of the on-chip network; The total execution delay of the current continuous electrical domain computation subgraph is less than or equal to a preset delay threshold. The hardware accelerators required for the current continuous electric domain computation subgraph are matched with currently available accelerator resources.

7. The operator scheduling method based on photoelectric throughput cost graph according to claim 3, characterized in that, In step S302A, at least one candidate continuous optical domain computation subgraph is generated for each starting point, and the benefit value of each candidate continuous optical domain computation subgraph is calculated. Then, candidate continuous optical domain computation subgraphs with benefit values ​​greater than or equal to preset target benefits are selected as the target continuous optical domain computation subgraph for that starting point. The reward value of the candidate continuous optical domain computation subgraph is calculated using the following formula: calculate; in, For optical domain gain value, To reduce the photoelectric conversion cost of performing continuous optical domain computation based on the candidate continuous optical domain computation subgraph, The computational cost reduction for computing the candidate continuous optical domain subgraph when performed in the optical domain compared to when performed in the electrical domain. The cost of using optical routing resources to perform continuous optical domain computation based on the candidate continuous optical domain computation subgraph. These are pre-set risk penalty items.

8. The operator scheduling method based on photoelectric throughput cost graph according to claim 4, characterized in that, In step S302B, at least one candidate continuous electric domain computation subgraph is generated for each starting point, and the reward value of each candidate continuous electric domain computation subgraph is calculated. Then, candidate continuous electric domain computation subgraphs with reward values ​​greater than or equal to preset target rewards are selected as target continuous electric domain computation subgraphs for that starting point. The reward value of the candidate continuous electric domain calculation subgraph is calculated using the following formula: calculate; in, For the electric domain revenue value, The reduction in photoelectric conversion cost for continuous electric domain computation based on the candidate continuous electric domain computation subgraph; The benefit of performing continuous electric domain computation based on the candidate continuous electric domain computation subgraph is relative to the benefit of performing the computation on each node in the candidate continuous electric domain computation subgraph separately. The cost of competing for electrical domain resources after merging electrical domain operators; This is a pre-defined risk penalty for performing electrical domain calculations.

9. An operator scheduling method based on photoelectric throughput cost graph according to claim 3 or 4, characterized in that, The steps in step S303A for setting the photoelectric conversion position specifically include: At the boundary of the continuous optical domain computation subgraph, determine whether the execution domain label of the adjacent node connected to the boundary node is in the electrical domain; if the adjacent node is in the electrical domain, insert a photoelectric conversion trigger command at the boundary to indicate the photoelectric conversion position; or, The step of setting the photoelectric conversion position in step S303B specifically includes: At the boundary of the continuous electrical domain computation subgraph, determine whether the execution domain label of the adjacent node connected to the boundary node is an optical domain; if the adjacent node is an optical domain, insert a photoelectric conversion trigger instruction at the boundary to indicate the photoelectric conversion position, or / and insert an on-chip cache write position.

10. An operator scheduling system based on photoelectric throughput cost graph, characterized in that, include: The parsing module is used to parse the target neural network model and obtain the types of each operator in the target neural network model and the data dependencies between operators. The cost graph construction module is used to construct a photoelectric throughput cost graph with each operator as a node and the data dependencies between operators as connecting edges. Each node is labeled with the execution domain label of the corresponding operator, and the execution domain label includes the optical domain and the electrical domain. Each connecting edge is labeled with a corresponding weight, and the weight is used to characterize the photoelectric conversion cost between the two nodes of the connecting edge. The subgraph generation module is used to traverse the photoelectric throughput cost map, and perform optical domain operator merging with nodes labeled as optical domains as starting points to obtain at least one continuous optical domain computation subgraph, and perform electrical domain operator merging with nodes labeled as electrical domains as starting points to obtain at least one continuous electrical domain computation subgraph. The candidate scheduling scheme generation module is used to replace different nodes as starting points and trigger the subgraph generation module to generate multiple continuous optical domain computation subgraphs and multiple continuous electrical domain computation subgraphs based on the replaced starting points. This is used to construct multiple sets of candidate scheduling schemes based on the continuous optical domain computation subgraphs and their corresponding continuous electrical domain computation subgraphs. Each set of candidate scheduling schemes includes at least one continuous optical domain computation subgraph and at least one corresponding continuous electrical domain computation subgraph and the photoelectric conversion position corresponding to its boundary. The target scheduling scheme selection module is used to calculate the comprehensive execution cost of each group of candidate scheduling schemes based on the execution cost of the corresponding operator of each node and the conversion cost of each photoelectric conversion position, and select the candidate scheduling scheme with the lowest comprehensive execution cost as the target scheduling scheme.

Images