Task scheduling method and device, computer device, computer readable storage medium and computer program product

Abstract

The application provides a task scheduling method and device, computer equipment, computer readable storage medium and computer program product. The method comprises the following steps: monitoring and acquiring first state information of a plurality of optical computing units and second state information of a plurality of computing nodes in each optical computing unit; acquiring one or more data processing tasks, allocating one or more first optical computing units for each data processing task according to the first state information, and allocating one or more first optical computing nodes in each first optical computing unit for each data processing task according to the second state information, so as to execute the corresponding data processing task through the one or more first optical computing nodes in the one or more first optical computing units, thereby improving the data processing efficiency and reducing the system energy consumption.

Description

Technical Field

[0001] This application relates to optical computing technology, and more particularly to a task scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Object storage systems, as a cloud storage architecture suitable for massive amounts of unstructured data, address the scalability bottlenecks of traditional file systems through a flat structure and metadata management. However, in large-scale computing scenarios, as data processing tasks increase and their complexity grows, the number of computing devices involved in these tasks also increases. These electronic computing devices generate significant heat under high loads, requiring more energy for cooling, thus significantly increasing the system's energy consumption. Furthermore, the limited parallel processing capabilities of traditional task processing methods currently used in object storage systems make it difficult to meet the demands of large-scale data processing. Summary of the Invention

[0003] This application provides a task scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which improves data processing efficiency while reducing system energy consumption.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a task scheduling method, the method including: monitoring and acquiring first state information of multiple optical computing units and second state information of multiple computing nodes in each optical computing unit; One or more data processing tasks are acquired, one or more first optical computing units are allocated to each data processing task according to the first status information, and one or more first optical computing nodes in each first optical computing unit are allocated to each data processing task according to the second status information, so as to execute the corresponding data processing tasks through one or more first optical computing nodes in the one or more first optical computing units.

[0005] In the above scheme, the data processing task includes task-related information, which includes one or more of the following: task type, resource requirements, and task priority; the step of allocating one or more first optical computing units to each data processing task according to the first status information, and allocating one or more first optical computing nodes in each first optical computing unit to each data processing task according to the second status information, includes: allocating one or more first optical computing units to each data processing task according to the task-related information and the first status information, and allocating one or more first optical computing nodes in each first optical computing unit to each data processing task according to the task-related information and the second status information.

[0006] In the above scheme, the step of allocating one or more first optical computing units to each data processing task based on the task-related information and the first state information includes: processing the task-related information and the first state information through a first photonic neural network to obtain a first allocation strategy, and determining one or more first optical computing units based on the first allocation strategy, wherein the first allocation strategy includes the correspondence between each data processing task and the first optical computing unit.

[0007] In the above scheme, the step of allocating one or more first optical computing nodes in each first optical computing unit to each data processing task based on the task-related information and the second state information includes: processing the task-related information and the second state information corresponding to the first optical computing unit through a second photonic neural network to obtain a second allocation strategy, and determining one or more first optical computing nodes based on the second allocation strategy, wherein the second allocation strategy includes the correspondence between each data processing task and the first optical computing node.

[0008] In the above scheme, the first photonic neural network uses a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

[0009] In the above scheme, the second photonic neural network uses a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

[0010] This application provides a task scheduling device, the device comprising: a monitoring unit and a scheduling unit, wherein: The monitoring unit is used to monitor and acquire the first status information of multiple optical computing units and the second status information of multiple computing nodes in each optical computing unit. The scheduling unit is configured to acquire one or more data processing tasks; allocate one or more first optical computing units to each data processing task according to the first status information; and allocate one or more first optical computing nodes in each first optical computing unit to each data processing task according to the second status information, so as to execute each data processing task through one or more first optical computing nodes in the one or more first optical computing units.

[0011] This application provides a computer device, the computer device comprising: Memory is used to store executable instructions or computer programs. When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the task scheduling method provided in the embodiments of this application.

[0012] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the task scheduling method provided in this application.

[0013] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the task scheduling method provided in this application.

[0014] This application provides a task scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product. By monitoring optical computing units and optical computing nodes deployed in a monitoring system, one or more first optical computing units are allocated to each data processing task based on the first state information of the acquired optical computing units. Based on the second state information of the acquired optical computing nodes, one or more first optical computing nodes are allocated to each data processing task within each first optical computing unit. This schedules each data processing task to one or more first optical computing nodes within the corresponding one or more first optical computing units, allowing the optical computing nodes to execute the corresponding task processing. By integrating optical computing into the object storage system, data processing tasks can be dynamically scheduled based on the monitored global state (e.g., the state of optical computing units) and / or local state (e.g., the state of optical computing nodes). Furthermore, utilizing optical computing nodes to process complex tasks in the object storage system significantly improves data processing speed, reduces system energy consumption, and enhances the overall system performance. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the task scheduling method according to an embodiment of this application; Figure 2 This is a schematic diagram of the Mach-Zehnder interferometer in the embodiments of this application; Figure 3 This is a flowchart illustrating an example of how this application is used in a task scheduling method. Figure 4 This is a schematic diagram of the composition structure of the task scheduling device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] Before elaborating on the technical solution of this application, a brief description of the relevant technologies will be given first.

[0018] Object storage is a distributed storage architecture designed for massive amounts of unstructured data. It manages data as independent objects, with each object including the data itself, metadata, and a globally unique identifier. This architecture is suitable for storing massive amounts of unstructured data such as images, videos, and documents. It can be accessed through an Application Programming Interface (API) and has extremely high scalability and reliability.

[0019] Currently, several solutions have been proposed to improve the task processing efficiency of object storage systems in cloud storage and big data scenarios. The first is an object storage upload and download acceleration method. This method leverages geographically dispersed object storage data centers to enable remote users to transmit data to object storage from the nearest location, thus accelerating uploads. Data written to a temporary bucket from the nearest location acts as an edge node of the Content Delivery Network (CDN) during its validity period, accelerating downloads for nearby users. A central node efficiently manages the mapping relationship between users and their destination buckets, and the intranet between clusters accelerates data reading, improving the success rate of user data transmission and ensuring the stability of cloud-based user services.

[0020] The second method is to accelerate data traversal in object storage. In addition to storing object metadata, it also independently stores and persists the corresponding directory information. Object information is associated with the corresponding parent directory using a file system identifier (FsId, File systemIDentifier). Directory information is managed in memory in a flat manner. Directory information includes FsId, directory count, and a list of subdirectories (which do not need to be persisted). When adding an object, the corresponding directory information is generated from top to bottom by parsing the object name, and the corresponding directory count is incremented. When deleting an object, the directory is traversed from bottom to top by parsing the object name, and the count value of the corresponding directory is decremented. The directory information of the directory with a count of 0 needs to be removed. Finally, the changed results are persisted to improve query speed and increase the efficiency of database operations.

[0021] However, with the rapid growth of data volume, the complexity of tasks such as data compression, encryption, and analysis in object storage systems is constantly increasing. The parallel processing capabilities of the aforementioned task processing methods are limited and cannot meet the needs of large-scale data processing. For example, although multi-core processors in current systems can process multiple tasks in parallel, their scalability is limited by the physical constraints of the chip, regardless of the method used to improve task processing efficiency. When performing complex tasks such as data compression, encryption, and analysis, traditional processing methods still require multiple steps, and each step introduces additional latency and computational overhead, resulting in a significant increase in system energy consumption. How to improve data processing efficiency while reducing system energy consumption has become an urgent problem to be solved.

[0022] Based on this, this application proposes a task scheduling method that can be applied to an object storage system (hereinafter referred to as the system). Figure 1 This is a flowchart illustrating the task scheduling method according to an embodiment of this application; as shown Figure 1 As shown, the method includes: Step 101: Monitor and acquire the first state information of multiple optical computing units and the second state information of multiple computing nodes in each optical computing unit; Step 102: Obtain one or more data processing tasks, allocate one or more first optical computing units to each data processing task according to the first status information, and allocate one or more first optical computing nodes in each first optical computing unit to each data processing task according to the second status information, so as to execute the corresponding data processing tasks through the one or more first optical computing nodes in the one or more first optical computing units.

[0023] In this embodiment, by monitoring the status of multiple optical computing units deployed in the system and the status of multiple computing nodes deployed in each optical computing unit, the first status information corresponding to each optical computing unit and the second status information corresponding to each computing node are obtained.

[0024] In this embodiment, the first state information is related to each optical computing unit and is used to indicate the current state of the optical computing unit. Specifically, it may include one or more of the following: the first resource consumption information (or energy consumption level data) and the first load information of each optical computing unit.

[0025] The first resource consumption information is used to indicate the physical resources (such as memory, processor) and / or network resources consumed by the optical computing unit in processing the task; the first load information is used to indicate the workload or busy level that the optical computing unit is currently processing.

[0026] An optical computing unit is an integrated task processing unit in the system, consisting of one or more optical computing nodes. It is the core hardware unit that uses photons instead of electrons for task processing. Each optical computing unit serves as the top-level architecture, integrating one or more optical computing nodes to complete the entire processing of one or more tasks from input to output. In the embodiments of this application, the first state information can also be called global state information. That is, by obtaining the first state information of each optical computing unit, the current global state information of the system can be obtained.

[0027] The second state information is related to each computing node and is used to indicate the current state of the optical computing node. Specifically, it may include one or more of the following: the second resource consumption information (or energy consumption level data) of each optical computing node, the second load information, processing capacity, transmission rate, etc.

[0028] The second resource consumption information is used to represent the physical and / or network resources consumed by the current optical computing node in processing tasks; the second load information is used to represent the workload or busy level that the current optical computing node is currently processing; the processing capacity is used to represent the computing power and / or energy efficiency level that the current optical computing node has in performing parallel operations using photons per unit time. In essence, it is a comprehensive reflection of the efficiency and scale of the optical functional modules performing mathematical operations (such as matrix multiplication and convolution); the transmission rate is used to represent the data carrying capacity of optical signals when they are transmitted at high speed within or between nodes. Similar to the transmission rate of electrical signals, the transmission rate of optical signals is used to measure the "bandwidth" and "speed" of optical signal processing information.

[0029] Optical computing nodes are nodes in the system that specifically execute complete or partial tasks. Multiple optical computing nodes can achieve parallel processing of multiple tasks. They perform logical operations (such as AND, OR, NOT), mathematical operations (such as addition, multiplication), or data processing tasks (such as convolution, matrix operations) by manipulating optical signals. Specifically, they build computing circuits on chips by integrating optical components, thereby realizing the processing of complete or partial tasks. In the embodiments of this application, the second state information can also be called local state information. That is, by obtaining the second state information of each optical computing node, the current local state information of the system can be obtained.

[0030] It should be noted that this application does not impose specific limitations on the optical components integrated in the optical computing node, and they can be deployed according to different computing needs. For example, an optical computing node can integrate multiple interferometers, forming a grid-like layout through parallel connections between them, used to construct basic optical logic gates (such as AND gates, OR gates, NOT gates, etc.) to perform mathematical operations. Taking the integration of multiple Mach-Zehnder interferometers (MZIs) in an optical computing node as an example... Figure 2 This is a schematic diagram of the Mach-Zehnder interferometer in the embodiments of this application; as shown Figure 2As shown, the MZI includes two beam splitters (beam splitter 21 and beam splitter 22) and two adjustable phase shifters (adjustable phase shifter 23 and adjustable phase shifter 24).

[0031] In this design, beam splitters 21 and 22 are located at the input and output ends of the MZI, respectively. Beam splitter 21 is responsible for "splitting the light," that is, splitting the input light into two coherent beams with different propagation directions, guiding them into the two independent arms of the interferometer. Beam splitter 21 can be a 50:50 beam splitter to split the input light into two coherent beams of equal intensity. Beam splitter 22 is responsible for "combining the light," that is, recombining the light returning from the two arms. Beam splitter 22 can also be a 50:50 beam splitter, meaning that beam splitters 21 and 22 are symmetrical 50:50 beam splitters. The beam splitters can be designed using directional couplers. Through the coordinated work of beam splitters 21 and 22, the light waves interfere, thereby extracting phase information.

[0032] Adjustable phase shifters 23 and 24 are both located on the upper arm of the MZI and are used to configure phase shifts. By continuously changing the phase of one of the beams, the phase difference between the two beams at the beam splitter 22 can be precisely controlled to modulate the intensity of the output light.

[0033] The MZI includes two input ports (port a and port b) and two output ports (port c and port d). A portion of the optical signal at the input ports is based on the definition of transmittance (the transmittance coefficient can be set to...). The data is transmitted to the output port, and the remainder is based on the definition of reflectivity (the reflectivity can be set to...). The input ports of different MZIs are coupled to another output port and cascaded in parallel to construct a more complex optical path system for simulating matrix operations, enabling complex task processing by the optical computing node. An optical computing node composed of multiple interferometers can perform calculations within the speed of light range while consuming ultra-low energy. Therefore, task processing based on optical computing nodes can achieve better energy efficiency and lower latency than electronic processors, thus significantly improving task processing efficiency.

[0034] In addition, MZI-based optical storage units can be deployed in each optical computing node to store intermediate computing results and related data during task execution.

[0035] It should be noted that the embodiments of this application do not impose specific restrictions on the method of monitoring optical computing units and optical computing nodes. For example, the monitoring module can be deployed on each optical computing unit and each optical computing node to collect first status information and / or second status information in real time or periodically. The monitoring module can use tools or components such as Prometheus for indicator collection, Grafana for visual monitoring, and Alertmanager with alarm function to build a monitoring system. By collecting first status information and second status information, the monitoring of optical computing units and optical computing nodes can be realized.

[0036] In this embodiment, after acquiring one or more data processing tasks, one or more first optical computing units are allocated to each data processing task according to the global status information (i.e., the first status information) collected by the monitoring module, and one or more first optical computing nodes are selected in each first optical computing unit according to the local status information (i.e., the second status information) collected by the monitoring module, so as to execute the corresponding data processing tasks through one or more optical computing nodes in one or more first optical computing units.

[0037] In this embodiment, the data processing task can be a data processing task from the client or a data processing task from within the system, such as highly complex tasks like data compression, data encryption, and data analysis for large-scale data.

[0038] The first optical computing unit refers to the optical computing unit in the system that is selected or assigned to process the current data processing task, i.e., the optimal processing unit corresponding to the current data processing task; the first optical computing node refers to the optical computing node in the first optical computing unit that is selected or assigned to process the current data processing task, i.e., the optimal processing node corresponding to the current data processing task.

[0039] It should be noted that the allocation process of one or more data processing tasks in this application embodiment can be implemented by a scheduling module deployed in the system. After receiving one or more data processing tasks, the scheduling module performs load balancing on the optical computing units and optical computing nodes based on the first and second status information provided by the monitoring module, and allocates each data processing task to one or more optical computing nodes in one or more optical computing units for processing, so as to avoid overloading of some optical computing units and / or optical computing nodes, and ensure the efficient and reliable operation of the system through dynamic strategies. This application embodiment does not impose specific limitations on the scheduling module. For example, a scheduling model based on input / output operations per second (IOPS) can be used, or an input / output (I / O) scheduler can be used.

[0040] For example, when a scheduling module is used to allocate optical computing units and optical computing nodes in optical computing units to each data processing task, the scheduling module may include a global scheduler and a local scheduler. The global scheduler is used to allocate one or more corresponding first optical computing units to each data processing task according to the first state information of the optical computing units, so as to realize global scheduling of all data processing tasks to be executed; the local scheduler is used to allocate one or more first optical computing nodes in each corresponding first optical computing unit to each data processing task according to the second turntable information of the optical computing nodes, so as to realize local scheduling of each data processing task to be executed.

[0041] In this embodiment, by monitoring the optical computing units and optical computing nodes deployed in the system, one or more first optical computing units are allocated to each data processing task based on the first state information of the acquired optical computing units. Based on the second state information of the acquired optical computing nodes, one or more first optical computing nodes are allocated to each data processing task within each first optical computing unit. This schedules each data processing task to one or more first optical computing nodes within the corresponding one or more first optical computing units, allowing the optical computing nodes to execute the corresponding task processing. By integrating optical computing into the object storage system, data processing tasks can be dynamically scheduled based on the global state (e.g., the state of optical computing units) and / or local state (e.g., the state of optical computing nodes) of the monitored system. Furthermore, utilizing optical computing nodes to process complex tasks in the object storage system significantly improves data processing speed, reduces system energy consumption, and enhances the overall performance of the system.

[0042] In some optional implementations, the data processing task includes task-related information, which includes one or more of the following: task type, resource requirements, and task priority; the step of allocating one or more first optical computing units to each data processing task according to the first status information, and allocating one or more first optical computing nodes in each first optical computing unit to each data processing task according to the second status information, includes: allocating one or more first optical computing units to each data processing task according to the task-related information and the first status information, and allocating one or more first optical computing nodes in each first optical computing unit to each data processing task according to the task-related information and the second status information.

[0043] In this embodiment, the data processing task includes task-related information. The task-related information is used by the scheduling module in the system to allocate one or more first optical computing units to each data processing task in combination with the first state information, and to allocate one or more first optical computing nodes in each first optical computing unit to each data processing task in combination with the second state information.

[0044] Specifically, the scheduling module allocates one or more first optical computing units based on the task-related information and first status information of each data processing task, and allocates one or more first optical computing nodes in the corresponding first optical computing units based on the task-related information and second status information of each data processing task.

[0045] In this embodiment, task-related information includes one or more of the following: task type, resource requirements, and task priority. Task type refers to the type of each data processing task. This embodiment does not impose specific restrictions on the classification method of data processing tasks; for example, they can be classified according to processing task, processing object, processing method, processing stage, etc. Resource requirements refer to the computing resources required by each data processing task. This embodiment does not impose specific restrictions on the method for determining the resource requirements of each data processing task; for example, they can be determined based on the data volume and computational complexity of the task. Task priority refers to the processing priority of each data processing task, used to determine the execution order of tasks. This embodiment does not impose specific restrictions on the method for determining the task priority of each data processing task; for example, priorities can be assigned based on the importance, urgency, or resource consumption of the task.

[0046] In some alternative implementations, before acquiring one or more data processing tasks, the method further includes: receiving one or more task requests, and obtaining a task queue based on the one or more task requests, the task queue including the one or more data processing tasks.

[0047] In this embodiment, the received task request is parsed to obtain a task queue. The task queue includes the corresponding data processing task and the task-related information of the data processing task. The task queue is passed to the scheduling module so that the scheduling module can obtain one or more data processing modules and the task-related information of each data processing task.

[0048] In this embodiment, the task queue is a task storage structure used to store all data processing tasks to be executed. This application embodiment does not impose specific limitations on the data format of the task queue. For example, the task queue can be represented by a vector, mapping each data processing task and its corresponding task-related information to a vector space and providing it to the scheduling module. The head of the queue is located at the first position of the vector, and the tail is located at the last position of the vector.

[0049] In some optional implementations, the step of allocating one or more first optical computing units to each data processing task based on the task-related information and the first state information includes: processing the task-related information and the first state information through a first photonic neural network to obtain a first allocation strategy, and determining one or more first optical computing units based on the first allocation strategy, wherein the first allocation strategy includes the correspondence between each data processing task and the first optical computing unit.

[0050] In this embodiment, the specific process by which the scheduling module allocates one or more first optical computing units to each data processing task based on task-related information and first state information is as follows: the task-related information of each data processing task and the first state information of each optical computing unit are input into the first photonic neural network, the first photonic neural network is used for reasoning, a first allocation strategy is output, and one or more first optical computing units for executing each data processing task are determined according to the first allocation strategy.

[0051] For example, when a global scheduler is used to allocate optical computing units to each data processing task, its optical part is implemented by a first photonic neural network. The first photonic neural network is a photonic neural network deployed in the global scheduler. The input of the network is the task-related information of each data processing task (such as task type, resource requirements, task priority) and the first state information of each optical computing unit. The output is a first allocation strategy to ensure that each data processing task is allocated to the most suitable first optical computing unit for execution.

[0052] In this embodiment, the photonic neural network is an intelligent computing system that uses photons instead of electrons for information processing and computation. It borrows the structure of traditional electronic neural networks, replacing electrons with photons as the signal carrier. It simulates the connection and information transmission methods of neurons in the human brain through the propagation, interference, and diffraction characteristics of light signals. Compared to traditional electronic neural networks, photonic neural networks can achieve faster and lower energy consumption computation. In this embodiment, the first photonic neural network deployed in the global scheduler may specifically include an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a flattening layer, a fully connected layer, and an output layer. The training process of the first photonic neural network can be implemented on a field-programmable gate array (FPGA).

[0053] The first allocation strategy includes the correspondence between each data processing task and the first optical computing unit. That is, the first allocation strategy is used to indicate one or more first optical computing units that can execute each data processing task. The global scheduler can schedule each data processing task to the corresponding first optical computing unit for processing according to the correspondence between each data processing task and the first optical computing unit in the first allocation strategy.

[0054] In some alternative implementations, the first photonic neural network utilizes a linear multiplier to perform linear computation, the linear multiplier comprising multiple interferometers connected in parallel.

[0055] In this embodiment, the first photonic neural network deployed in the global scheduler uses a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

[0056] A linear multiplier can be represented using two unitary multipliers (i.e., unitary matrices) and a diagonal matrix. The diagonal matrix is ​​obtained through singular value decomposition (SVD). The multipliers and diagonal matrix are constructed using multiple interferometers connected in parallel, thus forming a linear multiplier. This application does not impose specific limitations on the interferometers used to construct the multipliers and diagonal matrix; for example, they can be... Figure 2 The Mach-Zehnder interferometer shown.

[0057] In some optional implementations, the step of allocating one or more first optical computing nodes in each first optical computing unit to each data processing task based on the task-related information and the second state information includes: processing the task-related information and the second state information corresponding to the first optical computing unit through a second photonic neural network to obtain a second allocation strategy, and determining one or more first optical computing nodes based on the second allocation strategy, wherein the second allocation strategy includes the correspondence between each data processing task and the first optical computing node.

[0058] In this embodiment, the specific process by which the scheduling module allocates one or more first optical computing nodes from each first optical computing unit to each data processing task based on task-related information and second state information is as follows: the first allocation strategy, task-related information of each data processing task, and second state information of each optical computing node are input into the second photonic neural network; the second photonic neural network is used for reasoning to output the second allocation strategy; and one or more first optical computing nodes from each first optical computing unit used to execute each data processing task are determined according to the second allocation strategy.

[0059] For example, when a local scheduler is used to allocate optical computing nodes to each data processing task, its optical part is implemented by a second photonic neural network. The second photonic neural network is a photonic neural network deployed in the local scheduler. The network input is a first allocation strategy, task-related information of each data processing task (such as task type, resource requirements, task priority), and second state information of each optical computing node. The output is a second allocation strategy to ensure that each data processing task is allocated to the first optical computing node in the most suitable first optical computing unit for execution.

[0060] In this embodiment, the second photonic neural network deployed in the local scheduler is similar to the first photonic neural network deployed in the global scheduler. Specifically, it may include an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a flattening layer, a fully connected layer, and an output layer. The training process of the second photonic neural network can be implemented on an FPGA.

[0061] The second allocation strategy includes the correspondence between each data processing task and the first optical computing node. That is, the second allocation strategy is used to indicate one or more first optical computing nodes that can execute each data processing task. Based on the global scheduler scheduling each data processing task to the corresponding first optical computing unit according to the first allocation strategy, the local scheduler can schedule each data processing task to the first optical computing node in the corresponding first optical computing unit for processing according to the correspondence between each data processing task and the first optical computing node in the second allocation strategy.

[0062] In some alternative implementations, the second photonic neural network utilizes a linear multiplier to perform linear computation, the linear multiplier comprising multiple interferometers connected in parallel.

[0063] In this embodiment, the second photonic neural network deployed in the local scheduler uses a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

[0064] The linear multiplier used in the second photonic neural network is similar to that used in the first photonic neural network. It can be represented by two unitary multipliers (i.e., unitary matrices) and a diagonal matrix. The diagonal matrix is ​​obtained through singular value decomposition. The multipliers and the diagonal matrix are constructed based on multiple interferometers connected in parallel, thus forming a linear multiplier. In this embodiment, no specific limitation is placed on the interferometers used to form the multipliers and the diagonal matrix. For example, they can be used... Figure 2 The Mach-Zehnder interferometer shown.

[0065] Furthermore, when performing complex calculations using a first or second photonic neural network, nonlinear computational units can be introduced into the nonlinear layer. These nonlinear computational units are composed of Rectified Linear Unit (ReLU) functions. For example, the expression for a ReLU function can be designed as follows: (1) Among them, the ReLU function, as a high-performance nonlinear function, outputs 0 when the input x < 0 and x when the input x ≥ 0. The ReLU function is characterized by its gradient being in the positive range, which can solve the gradient vanishing problem of other activation functions (such as the sigmoid function and the hyperbolic tangent function). Furthermore, since the ReLU function essentially involves simple maximum value calculation, it has lower computational energy consumption and faster computation speed compared to other activation functions.

[0066] In this embodiment, a linear multiplier based on multiple parallel interferometers is applied to the first and second photonic neural networks (for example, it can be applied to other layers such as fully connected layers) to achieve a large number of matrix-vector multiplications using the interferometer-based linear multiplier (i.e., optical matrix-vector multiplier). Matrix-vector multiplication is one of the most computationally intensive operations in data processing and deep learning. Photonic neural networks built using optical matrix-vector multipliers can exhibit extremely high energy efficiency, even exceeding the basic limit of irreversible digital computers. Its energy efficiency increases with the size of the matrix and the vector to be multiplied. Therefore, a large number of component scalar multiplication and accumulation operations can be executed in parallel in the optical domain to improve processing speed, parallelism, and scalability.

[0067] As an example, Figure 3 This is a flowchart illustrating an example of the application of this application to a task scheduling method; such as Figure 3 As shown, the specific process of task scheduling through the monitoring module, global scheduler, and local scheduler is as follows: Step 301: Obtain the task queue.

[0068] Specifically, by parsing the received task requests, a task queue is obtained. The task queue includes the data processing task corresponding to each task request, as well as task-related information for each data processing task. This task-related information includes one or more of the following: task type, resource requirements, and task priority. After obtaining the task queue, it is passed to the global scheduler and local scheduler, so that the global scheduler and local scheduler can obtain one or more data processing modules and the task-related information for each data processing task.

[0069] Step 302: Monitor and acquire the first state information of multiple optical computing units and the second state information of multiple computing nodes in each optical computing unit.

[0070] Specifically, the monitoring module monitors the status of multiple optical computing units deployed in the system, as well as the status of multiple computing nodes deployed in each optical computing unit, to obtain the first status information corresponding to each optical computing unit and the second status information corresponding to each computing node. The first status information is then provided to the global scheduler, and the second status information is provided to the local scheduler.

[0071] Step 303: Allocate the first optical computing unit to the data processing task according to the first state information.

[0072] Specifically, the global scheduler inputs the task-related information of each data processing task and the first state information of each optical computing unit obtained through the monitoring module into the first photonic neural network, uses the first photonic neural network to perform inference, outputs the first allocation strategy, and determines one or more first optical computing units to execute each data processing task according to the first allocation strategy.

[0073] Step 304: Allocate the first optical computing unit in the first optical computing node to the data processing task according to the second status information.

[0074] Specifically, the local scheduler inputs the first allocation strategy determined by the global scheduling module, the task-related information of each data processing task, and the second state information of each optical computing node obtained through the monitoring module into the second photonic neural network. The second photonic neural network is used for inference to output the second allocation strategy. Based on the second allocation strategy, one or more first optical computing nodes in each first optical computing unit are determined to be used to execute each data processing task. Based on the second allocation strategy, each data processing task is scheduled to the corresponding first computing node so that the corresponding data processing task can be executed by one or more first optical computing nodes in one or more first optical computing units.

[0075] Based on the above embodiments, this application also provides a task scheduling device. Figure 4 This is a schematic diagram of the composition structure of the task scheduling device provided in the embodiments of this application; as shown below. Figure 4 As shown, the device includes: a monitoring unit 41 and a scheduling unit 42, wherein: The monitoring unit 41 is used to monitor and acquire the first status information of multiple optical computing units and the second status information of multiple computing nodes in each optical computing unit. The scheduling unit 42 is configured to acquire one or more data processing tasks; allocate one or more first optical computing units to each data processing task according to the first status information; and allocate one or more first optical computing nodes in each first optical computing unit to each data processing task according to the second status information, so as to execute each data processing task through one or more first optical computing nodes in the one or more first optical computing units.

[0076] In one optional embodiment of this application, the data processing task includes task-related information, which includes one or more of the following: task type, resource requirements, and task priority; the scheduling unit 42 is configured to allocate one or more first optical computing units to each data processing task according to the task-related information and the first status information, and to allocate one or more first optical computing nodes in each first optical computing unit to each data processing task according to the task-related information and the second status information.

[0077] In one optional embodiment of this application, the scheduling unit 42 includes a first scheduling unit 421, which is used to process the task-related information and the first state information through a first photonic neural network to obtain a first allocation strategy, and determine one or more first optical computing units based on the first allocation strategy. The first allocation strategy includes the correspondence between each data processing task and the first optical computing unit.

[0078] In one optional embodiment of this application, the scheduling unit 42 includes a second scheduling unit 422, which is used to process the task-related information and the second state information corresponding to the first optical computing unit through a second photonic neural network to obtain a second allocation strategy, and determine one or more first optical computing nodes based on the second allocation strategy. The second allocation strategy includes the correspondence between each data processing task and the first optical computing node.

[0079] In one alternative embodiment of this application, the first photonic neural network utilizes a linear multiplier to perform linear calculations, wherein the linear multiplier includes multiple interferometers connected in parallel.

[0080] In one alternative embodiment of this application, the second photonic neural network utilizes a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

[0081] In the implementation of this application, the monitoring unit 41, scheduling unit 42, first scheduling unit 421 and second scheduling unit 422 in the device can be implemented by a CPU, digital signal processor (DSP), microcontroller unit (MCU) or FPGA in practical applications.

[0082] It should be noted that the task scheduling device described above is only illustrated by the division of the program modules. In practical applications, the processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the task scheduling device and task scheduling method embodiments provided above belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0083] This application also provides a computer device. Figure 5 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application; as shown below. Figure 5 As shown, the computer device includes a memory 52, a processor 51, and a computer program stored in the memory 52 and executable on the processor 51.

[0084] Optionally, the computer device also includes at least one communication component 54. The various components in the computer device can be coupled together via a bus system 53. It is understood that the bus system 53 is used to implement communication between these components. In addition to a data bus, the bus system 53 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general labeled all buses as Bus System 53.

[0085] It is understood that memory 52 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 52 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0086] The methods disclosed in the embodiments of this application can be applied to processor 51, or implemented by processor 51. Processor 51 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 51 or by instructions in the form of software. The processor 51 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 51 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 52. ​​Processor 51 reads the information in memory 52 and combines its hardware to complete the steps of the aforementioned method.

[0087] In an exemplary embodiment, the computer device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned methods.

[0088] This application also provides a computer-readable storage medium on which a computer program is stored.

[0089] Optionally, the computer-readable storage medium can be applied to the task scheduling apparatus of the present application embodiment; then when the program is executed by the processor, it implements the steps of the task scheduling method of the present application embodiment.

[0090] This application also provides a computer program product, including a computer program that can be executed by the processor 51 of a computer device to complete the steps of the task scheduling method described in this application.

[0091] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0092] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0093] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0095] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0097] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task scheduling method, characterized in that, The method includes: Monitor and acquire the first state information of multiple optical computing units and the second state information of multiple computing nodes in each optical computing unit; One or more data processing tasks are acquired, one or more first optical computing units are allocated to each data processing task according to the first status information, and one or more first optical computing nodes in each first optical computing unit are allocated to each data processing task according to the second status information, so as to execute the corresponding data processing tasks through one or more first optical computing nodes in the one or more first optical computing units.

2. The method according to claim 1, characterized in that, The data processing task includes task-related information, which includes one or more of the following: task type, resource requirements, and task priority; the step of allocating one or more first optical computing units to each data processing task based on the first status information, and allocating one or more first optical computing nodes in each first optical computing unit to each data processing task based on the second status information, includes: Based on the task-related information and the first status information, one or more first optical computing units are allocated to each data processing task, and based on the task-related information and the second status information, one or more first optical computing nodes are allocated to each first optical computing unit for each data processing task.

3. The method according to claim 2, characterized in that, The step of allocating one or more first optical computing units to each data processing task based on the task-related information and the first status information includes: The task-related information and the first state information are processed by a first photonic neural network to obtain a first allocation strategy. One or more first optical computing units are determined based on the first allocation strategy. The first allocation strategy includes the correspondence between each data processing task and the first optical computing unit.

4. The method according to claim 2, characterized in that, The step of allocating one or more first optical computing nodes in each first optical computing unit to each data processing task based on the task-related information and the second status information includes: The task-related information and the second state information corresponding to the first optical computing unit are processed by the second photonic neural network to obtain a second allocation strategy. One or more first optical computing nodes are determined based on the second allocation strategy. The second allocation strategy includes the correspondence between each data processing task and the first optical computing node.

5. The method according to claim 3, characterized in that, The first photonic neural network uses a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

6. The method according to claim 4, characterized in that, The second photonic neural network uses a linear multiplier to perform linear calculations. The linear multiplier includes multiple interferometers connected in parallel.

7. A task scheduling device, the device comprising: The monitoring unit and the scheduling unit, wherein: The monitoring unit is used to monitor and acquire the first status information of multiple optical computing units and the second status information of multiple computing nodes in each optical computing unit. The scheduling unit is configured to acquire one or more data processing tasks; allocate one or more first optical computing units to each data processing task according to the first status information; and allocate one or more first optical computing nodes in each first optical computing unit to each data processing task according to the second status information, so as to execute each data processing task through one or more first optical computing nodes in the one or more first optical computing units.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images