Optoelectronic interconnection cooperative training system and method for multi-modal intelligent agent network

Abstract

The application discloses a kind of photoelectric interconnection cooperative training systems and methods for multi-modal intelligent agent network, wherein, intelligent agent node includes photoelectric interface module, programmable optical switch unit is used to establish or disconnect the optical connection link between any two intelligent agent nodes;Training framework interface layer is used to receive inference task and task distribution according to inference task, obtain task distribution information;Control scheduling unit when obtaining task distribution information, control photoelectric interface module selects multiple first optical connection link, interaction between the intelligent agent node on first optical connection link, complete inference task.The present application utilizes the high bandwidth and low delay characteristics of optical communication, significantly improves the communication efficiency and energy consumption performance in multi-modal intelligent agent network training, while realizing dynamic high-speed optical interconnection between intelligent agent nodes, low-latency exchange of inter-modal features and gradients.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an optoelectronic interconnection collaborative training system and method for multimodal intelligent agent networks. Background Technology

[0002] With the deep integration of multimodal perception (vision, speech, infrared, radar, etc.), traditional centralized neural network computing architectures are gradually showing their bottlenecks when handling complex multi-source tasks. To improve the generalization and real-time performance of models, researchers have proposed a collaborative network structure composed of multiple "agents," where each node processes specific modal information and achieves information fusion and collaborative learning through network communication.

[0003] However, during the training of multi-agent systems, feature sharing and gradient synchronization between nodes become performance bottlenecks. Existing electrical interconnection methods have the following shortcomings: significant latency when the data volume is large, making it difficult to support high-frequency bidirectional communication; high copper cable link loss and power consumption, which are not conducive to system expansion; and slow routing switching speed, which cannot meet the requirements of dynamic task allocation.

[0004] Therefore, there is an urgent need for a short-range, low-power, high-bandwidth interconnect architecture to achieve efficient collaborative training and communication among multimodal agents. Summary of the Invention

[0005] This invention provides an optoelectronic interconnection collaborative training system and method for multimodal intelligent agent networks, which can improve the accuracy of road segmentation.

[0006] In a first aspect, the present invention provides a method for optoelectronic interconnection-based collaborative training of multimodal intelligent agent networks, comprising:

[0007] Multiple intelligent agent nodes, each intelligent agent node including a photoelectric interface module, are used to process data of multiple different modalities;

[0008] A programmable optical switching unit is provided, which is connected to the optoelectronic interface module in each of the intelligent agent nodes via optical fiber. The programmable optical switching unit is used to establish or disconnect the optical connection link between any two of the intelligent agent nodes.

[0009] An external light source is used to output a continuous light signal;

[0010] The beam splitter is connected to each of the intelligent agent nodes and the programmable optical switching unit respectively. The beam splitter is used to distribute the optical signal output by the external light source to each of the intelligent agent nodes and the programmable optical switching unit.

[0011] The training framework interface layer is connected to the beam splitter. The training framework interface layer is used to receive inference tasks and distribute tasks according to the inference tasks to obtain task distribution information. The task distribution information includes the task types of multiple sub-tasks, participating nodes, and the amount of data transmitted between participating nodes.

[0012] A control and scheduling unit is connected to the programmable optical switching unit, the training framework interface layer, and each of the agent nodes. When the control and scheduling unit obtains task distribution information, it acquires the agent modality mapping table of multiple agent nodes and the link status of each optical connection link. Based on the agent modality mapping table, the link status of each optical connection link, and the task distribution information, it controls the optoelectronic interface module to select multiple first optical connection links. The agent nodes on the first optical connection links interact with each other to complete the inference task.

[0013] In an optional embodiment, the programmable optical switching unit includes multiple MZI units, which are spliced ​​together in a matrix topology to form an optical matrix, thereby establishing or disconnecting optical connection links between any two of the intelligent agent nodes.

[0014] In an optional embodiment, the MZI unit is a 2×2 matrix MZI unit, which performs switching control between two input ports and two output ports.

[0015] In an optional embodiment, the optoelectronic interface module includes a short-pitch SerDes interface, and the optoelectronic interface module is connected to the programmable optical switching unit through the short-pitch SerDes interface.

[0016] In an optional embodiment, the optoelectronic interface module includes an electric drive amplifier, a modulation unit, a wavelength selection element, a photodetector, and a set of local control interfaces.

[0017] In an optional embodiment, the control scheduling unit detects the link status of the first optical connection link at a preset period; when the first optical connection link is abnormal, it switches the first optical connection link to the second optical connection link.

[0018] In an optional embodiment, when the load on the first optical connection link is less than a preset value, the control scheduling unit disconnects the first optical connection link.

[0019] In one alternative embodiment, the plurality of agent nodes include at least two of the following: voice-visual agents, voice agents, bioelectric agents, and radar agents.

[0020] In an optional embodiment, the interaction between the agent nodes on the first optical connection link includes:

[0021] Determine the source node and target node among the agent nodes on the first optical connection link;

[0022] The source node modulates local feature or gradient data into an optical signal via the photoelectric interface module and sends it to the target node.

[0023] The target node converts the optical signal back into an electrical signal through the photoelectric interface module and inputs it into the local accelerator for calculation.

[0024] Secondly, the optoelectronic interconnection collaborative training method for multimodal intelligent agent networks provided by the present invention includes:

[0025] When the system is powered on, the beam splitter distributes the optical signal output by the external light source to each of the intelligent agent nodes and the programmable optical switching unit. The programmable optical switching unit establishes an optical connection link between any two of the intelligent agent nodes.

[0026] The control and scheduling unit initializes the system, obtains the link status of multiple optical connection links, and enters standby mode.

[0027] After each of the intelligent agent nodes is started, the control and scheduling unit obtains the intelligent agent modality mapping table of multiple intelligent agent nodes;

[0028] After receiving the inference task, the training framework interface layer distributes the task according to the inference task to obtain task distribution information. The task distribution information includes the task type of multiple sub-tasks, participating nodes, and the amount of data transmitted between participating nodes.

[0029] The control and scheduling unit controls the photoelectric interface module to select multiple first optical connection links based on the intelligent agent modality mapping table, the link status of each optical connection link, and task distribution information.

[0030] The intelligent agent nodes on the first optical connection link interact to complete the sub-task;

[0031] When the subtask is completed, the control scheduling unit releases the first optical connection link corresponding to the subtask and reallocates it.

[0032] In this invention, compared to related technologies, the optoelectronic interconnection collaborative training system for multimodal agent networks includes: multiple agent nodes, each agent node including an optoelectronic interface module, the multiple agent nodes being used to process data from multiple different modalities; a programmable optical switching unit, the programmable optical switching unit being connected to the optoelectronic interface module in each agent node via optical fiber, the programmable optical switching unit being used to establish or disconnect an optical connection link between any two agent nodes; an external light source, the external light source being used to output a continuous optical signal; a beam splitter, the beam splitter being connected to each agent node and the programmable optical switching unit, the beam splitter being used to distribute the optical signal output from the external light source to each agent node and the programmable optical switching unit; and a training framework interface layer, the training framework interface layer being connected to the programmable optical switching unit. The system consists of a beamformer and a training framework interface layer. The training framework interface layer receives inference tasks and distributes them accordingly, obtaining task distribution information. This information includes the task types of multiple sub-tasks, participating nodes, and the amount of data transmitted between them. A control and scheduling unit is connected to the programmable optical switching unit, the training framework interface layer, and each agent node. Upon receiving task distribution information, the control and scheduling unit obtains the agent modality mapping tables of multiple agent nodes and the link status of each optical connection link. Based on the agent modality mapping tables, the link status of each optical connection link, and the task distribution information, it controls the optoelectronic interface module to select multiple first optical connection links. Agent nodes on these first optical connection links interact to complete the inference task. This invention utilizes the high bandwidth and low latency characteristics of optical communication to significantly improve communication efficiency and energy consumption performance in multimodal agent network training, while simultaneously achieving dynamic high-speed optical interconnection between agent nodes and low-latency exchange of features and gradients between modalities. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a scenario for an optoelectronic interconnection collaborative training system for multimodal intelligent agent networks provided in an embodiment of the present invention;

[0035] Figure 2 This is a schematic flowchart of an embodiment of the optoelectronic interconnection collaborative training method for multimodal intelligent agent networks provided by the present invention. Detailed Implementation

[0036] It should be noted that the principles of the present invention are illustrated by way of example implemented in a suitable computing environment. The following description is based on the specific embodiments of the invention illustrated, and should not be construed as limiting the invention to other specific embodiments not detailed herein.

[0037] In the following description of the present invention, references are made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description of the present invention, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0040] Please refer to Figure 1 , Figure 1 This is a schematic diagram of an embodiment of the optoelectronic interconnection collaborative training system for multimodal intelligent agent networks provided in this invention. Figure 1 As shown, the optoelectronic interconnection collaborative training system for multimodal intelligent agent networks provided by the present invention includes multiple intelligent agent nodes, a programmable optical switching unit, an external light source, a beam splitter, a training framework interface layer, and a control and scheduling unit.

[0041] In this embodiment of the invention, the intelligent agent node includes a photoelectric interface module. Multiple intelligent agent nodes are used to process data from multiple different modalities.

[0042] The programmable optical switching unit is connected to the optoelectronic interface module in each intelligent agent node via optical fiber. The programmable optical switching unit is used to establish or disconnect the optical connection link between any two intelligent agent nodes.

[0043] An external light source is used to output a continuous light signal.

[0044] The beam splitter is connected to each intelligent agent node and programmable optical switching unit. The beam splitter is used to distribute the optical signal output from the external light source to each intelligent agent node and programmable optical switching unit.

[0045] The training framework interface layer is connected to the beam splitter. The training framework interface layer is used to receive inference tasks and distribute tasks according to the inference tasks to obtain task distribution information. The task distribution information includes the task types of multiple sub-tasks, participating nodes, and the amount of data transmitted between participating nodes.

[0046] The control and scheduling unit is connected to the programmable optical switching unit, the training framework interface layer, and each agent node. When the control and scheduling unit obtains the task distribution information, it obtains the agent modality mapping table of multiple agent nodes and the link status of each optical connection link. Based on the agent modality mapping table, the link status of each optical connection link, and the task distribution information, the control and scheduling unit controls the optoelectronic interface module to select multiple first optical connection links. The agent nodes on the first optical connection links interact with each other to complete the inference task.

[0047] In this embodiment of the invention, the intelligent agent node includes: a local sensing / preprocessing unit, an AI accelerator, local storage, and an optoelectronic interface module (OE-lite). The AI ​​accelerator is such as a GPU / TPU / ASIC. The intelligent agent node is used to collect or receive data of a certain modality, such as visual frames, voice frames, radar echoes, etc., perform forward / backward computation (feature extraction, local model update, etc.) locally, and send features or gradients to other nodes or receive and fuse external features through the optoelectronic interface module when needed.

[0048] The data flow of agent nodes in collaborative training includes two types:

[0049] Feature dispatch: In the forward phase, each modality node dispatches its local feature vector. Send to the fusion or aggregation node.

[0050] Gradient / parameter exchange: During the reverse or synchronization phase, nodes need to exchange gradients or parameter update amounts.

[0051] Among them, multiple intelligent agent nodes include at least two of the following: voice and vision intelligent agents, voice intelligent agents, bioelectric intelligent agents, and radar intelligent agents.

[0052] In this embodiment of the invention, the optoelectronic interface module includes a very short-reach SerDes (VSR) interface. The optoelectronic interface module is connected to the programmable optical switching unit through the VSR interface of the control scheduling unit. VSR is a high-speed serial interconnect for short distances (typically ≤1m, commonly 10–100mm) within a board / package. By simplifying equalization and clock recovery, and optimizing encoding and driving, it achieves high-bandwidth point-to-point transmission with lower power consumption, smaller area, and lower latency.

[0053] A short-range SerDes interface is used between the intelligent agent node and the photoelectric interface module. To ensure system performance, the unidirectional transmission delay from the accelerator to the photoelectric interface module should meet the following requirements:

[0054]

[0055] in, For node internal I / O latency, such as serialization / parallelization, PCIe / PLLink, etc. For the overall latency budget, typically 0.1–0.5 µs is used. The node side should support data packing and small-batch aggregation to reduce overhead and improve optical link utilization.

[0056] The standardized data frame format between intelligent agent nodes and the optoelectronic interface module includes: frame header (source node ID, target node ID, mode type, sequence number), payload length, timestamp, and checksum. The control plane and data plane are separated; control messages are used for requesting path establishment, ACK / NAK, and link quality reporting.

[0057] In this embodiment of the invention, the first optical connection link includes a source node for transmitting data and a target node for receiving data.

[0058] Interactions between intelligent agent nodes on the first optical connection link include: determining the source node and the target node among the intelligent agent nodes on the first optical connection link; the source node modulating local feature or gradient data into an optical signal via a photoelectric interface module and sending it to the target node; and the target node restoring the optical signal into an electrical signal via the photoelectric interface module and inputting it into the local accelerator for computation.

[0059] The optoelectronic interface module is a key hardware module of this system, responsible for electro-optical and opto-electrical conversion, low-power modulation / detection, simple link monitoring, and local timing buffering.

[0060] In this embodiment of the invention, the optoelectronic interface module includes an electric drive amplifier, a modulation unit, a wavelength selection element, a photodetector, and a set of local control interfaces.

[0061] Specifically, the short-range SerDes supports NRZ or PAM4 formats, with modulation units consisting of a micro-ring modulator (MRM) or a Mach-Zehnder modulator (MZM). A local control interface is used for temperature control, tuning, and link measurement.

[0062] Furthermore, the optoelectronic interface module includes a FIFO (First In First Out) chip to resolve timing mismatch issues during the establishment of the optical switch between the transmitter and the receiver. FIFO is a fundamental data storage and transmission buffering mechanism, and also a type of dedicated hardware / software component. Its core principle is "the first data to enter is read / processed first," and it is widely used in digital circuits, computer systems, communication protocols, and other fields.

[0063] The relationship between the speed, bandwidth, and power consumption of the optoelectronic interface module is as follows:

[0064] Let the single-channel data rate of the optoelectronic interface module be R (Gb / s), and the number of wavelengths be... Then the theoretical bandwidth of a single port The calculation formula is as follows:

[0065]

[0066] If the system has N nodes, and each node uses P optical fibers / channels, then the total system bandwidth is... as follows,

[0067]

[0068] In terms of power consumption, the power consumption per bit of the optoelectronic interface module It can be approximated as:

[0069]

[0070] The items listed represent the power consumption per bit for the short-range SerDes interface, modulator, driver, and detector in the optoelectronic interface module. The system design goal is to... Lower than traditional modules, for example, a target of 0.5–1 pJ / bit, depending on the implementation process.

[0071] For a given rate R and transmission window, the required signal-to-noise ratio (SNR) and bit error rate (BER) can be estimated using the Q factor or Shannon approximation. If NRZ is used, the BER and the receiver Q value satisfy:

[0072]

[0073] Therefore, during the design phase, it is necessary to ensure that the BER corresponding to the Q value of the receiver meets the fault tolerance threshold of the training task.

[0074] Especially for MRMs (microrings), resonant frequency tuning requires thermoelectric or electric drive methods. The optoelectronic interface module includes local temperature sensing and closed-loop control, and the tuning algorithm runs periodically to maintain resonance alignment with the external laser wavelength.

[0075] In this embodiment of the invention, the programmable optical switching unit includes multiple MZI (Mach-Zehnder Interferometer) units. The multiple MZI units are spliced ​​together in a matrix topology, and the multiple MZI units are constructed into an optical matrix to establish or disconnect optical connection links between any two intelligent agent nodes.

[0076] Specifically, the control and scheduling unit MZI unit is a 2×2 matrix MZI unit, which performs switching control between two input ports and two output ports.

[0077] The programmable optical switching unit provides reconfigurable optical layer interconnection between nodes. It constructs an optical matrix based on 2×2 MZI switching units to achieve short-term establishment and disconnection of any port.

[0078] The programmable optical switching unit is composed of several MZI units assembled according to a matrix / network topology (e.g., Benes, Clos, or hierarchical crossover). For small-scale clusters, such as 4–16 nodes, a single-layer or two-layer MZI matrix can be used to reduce insertion loss and control complexity. Each 2×2 MZI unit is controlled by a local drive circuit for crossover / cut-through states.

[0079] Optical path establishment delay This includes control command issuance, MZI phase adjustment time, and settling time. The overall design should be as follows:

[0080]

[0081] in Based on the maximum transient latency allowed by the training framework, a target latency of less than 5ms (ideally sub-millisecond) is recommended. To achieve this, the controller of the programmable optical switching unit needs to support parallel configuration of multiple units and pre-programmed pipelines.

[0082] Each MZI unit has insertion loss, and the total path loss is the sum of the losses of each unit plus coupling / splitting loss. To ensure receiver SNR, the programmable optical switching unit should be coordinated with an external laser power budget and amplification strategy (using local amplification or higher external power if necessary). The programmable optical switching unit design should support multi-wavelength WDM, using filters or AWGs for wavelength division multiplexing / demultiplexing.

[0083] In this embodiment of the invention, an external light source is used to provide a stable continuous optical signal for the entire system. Unlike the traditional approach of configuring a laser for each node individually, this invention adopts a centralized light supply structure, where all nodes share one or more external light sources. The external light sources typically operate in the 1310 or 1550 nanometer wavelength range, outputting continuous wave light, which is then distributed to the modulation ports of each intelligent agent node via a beam splitter.

[0084] This centralized light supply method has three main advantages: reduced power consumption and thermal noise: the energy utilization rate of a single high-efficiency laser is higher, which can reduce the thermal load of each node. Unified wavelength management: it avoids detuning problems caused by laser drift at each node and simplifies optical path calibration; and easy maintenance: external light sources are easy to detect and replace uniformly, improving system reliability.

[0085] Beam splitters typically employ an optical fiber coupled structure, using power equalization and attenuation control to ensure each node receives a relatively consistent incident light intensity. For more complex tasks, wavelength division multiplexing channels can be added to achieve multi-wavelength allocation and support parallel communication.

[0086] In this embodiment of the invention, the control and scheduling unit is the core coordination module of the system, used to manage optical switching paths in real time, monitor link status, and coordinate multi-agent task communication. During operation, the control and scheduling unit continuously receives task requests from the training framework interface layer and dynamically allocates optical path resources according to the request content.

[0087] The control and scheduling unit has the following functions:

[0088] Optical path configuration and switching: The programmable optical switching unit is precisely driven through the electronic control interface to achieve rapid establishment or release of optical connection links between nodes.

[0089] Link status monitoring: Periodically detect the link status of each optical connection link, including optical power, bit error rate, and latency, to determine the health status of the link.

[0090] Fault tolerance and rerouting: When the signal of an optical link weakens or the connection becomes abnormal, the control and scheduling unit can automatically rebuild the path or switch to a backup channel to ensure uninterrupted communication.

[0091] Energy consumption and bandwidth optimization: Dynamically adjust the number of optical paths and connection methods according to the current load and task type to avoid energy waste caused by idle links.

[0092] In addition, the control and scheduling unit has hierarchical management capabilities, and can adopt a distributed structure of "global scheduling + local control" in larger-scale systems to balance configuration latency and system complexity.

[0093] In one specific embodiment, the control scheduling unit detects the link status of the first optical connection link at a preset period; when the first optical connection link malfunctions, it switches the first optical connection link to the second optical connection link. The second optical connection link is either an automatically rebuilt optical connection link or a backup optical connection link.

[0094] The preset period can be 1 minute, 2 minutes, etc., which can be set according to the specific situation.

[0095] Specifically, when the signal of the first optical connection link weakens or the connection becomes abnormal, the control and scheduling unit can automatically rebuild the second optical connection link or switch to the backup second optical connection link to ensure uninterrupted communication.

[0096] In this embodiment of the invention, when the load on the first optical connection link is less than a preset value, the control scheduling unit disconnects the first optical connection link to avoid energy waste caused by an idle link.

[0097] The preset value can be set according to specific circumstances, and the present invention does not limit it.

[0098] In this embodiment of the invention, the training framework interface layer is embedded within the deep learning training platform to connect the upper-layer algorithm with the underlying optoelectronic interconnect hardware. When the multimodal agent network needs to transmit features or gradients between different nodes during training, the interface layer is responsible for generating communication requests and initiating optical path establishment commands to the control and scheduling unit.

[0099] The main functions of the training framework interface layer include:

[0100] Task identification and routing request: Identify the parts of the current batch of tasks that require cross-modal communication and generate communication topology requirements.

[0101] Delay-aware scheduling: Based on the task size and real-time status, it chooses to immediately establish an optical path or temporarily store data for merged transmission.

[0102] Interface mapping: Maps upper-layer logical nodes (vision, voice, infrared, etc.) to physical node addresses, simplifying optical path control logic.

[0103] Feedback and optimization: Collect metrics such as link latency and transmission time during training for subsequent task scheduling optimization.

[0104] This module enables the entire system to remain compatible with existing distributed training frameworks at the software level, allowing it to leverage the high-speed advantages of optoelectronic interconnects without requiring large-scale modifications to the algorithm structure.

[0105] To ensure the real-time performance of multimodal collaborative training, this system has undergone comprehensive optimization in terms of bandwidth, latency, and energy consumption.

[0106] Optical interconnects consume significantly less power than electrical interconnects under the same bandwidth conditions, and single-hop communication latency is typically less than one microsecond. The control and scheduling unit completes path reconfiguration within milliseconds, thereby enabling near real-time feature interaction between different modalities.

[0107] The system employs a multi-layered optimization mechanism during its design:

[0108] At the hardware level, signal attenuation is reduced by using a low insertion loss MZI array and a high-sensitivity detector.

[0109] At the control level, a predictive scheduling algorithm is used to pre-configure the optical connection links that may be used based on historical traffic trends.

[0110] In terms of energy management, the system can automatically shut down idle modulators based on the training load, achieving dynamic energy saving.

[0111] Through these optimization measures, the system significantly reduces the latency of multimodal cooperative communication without increasing hardware complexity.

[0112] In this embodiment of the invention, after the system is initially deployed or reconfigured, a series of initialization operations, such as optical and electrical calibration, are required to ensure the performance stability of each module.

[0113] Initialization operations include link detection, modulator tuning, bit error testing, and automatic compensation.

[0114] Link detection: Confirm the channel connectivity between all nodes by scanning the test signal and measure the received power distribution.

[0115] Modulator tuning: Fine-tune the resonant frequency of each modulation unit (micro-ring or MZM modulator) to ensure matching with the external laser wavelength.

[0116] Bit error rate testing: The bit error rate is measured using a standard test sequence, and the results are uploaded to the control and scheduling unit to establish a health status table.

[0117] Automatic compensation: If a power imbalance is detected, the system will automatically adjust the splitting ratio or drive current to compensate.

[0118] This process can be completed automatically upon startup or run periodically to respond to temperature or load changes, ensuring long-term stable operation of the system.

[0119] To adapt to different application scenarios, this system possesses excellent scalability and replaceability: the number of nodes is scalable, and can be expanded to hundreds of nodes in the future through a hierarchical switching structure. The switching structure is replaceable: in addition to MZI arrays, MEMS micromirror arrays or photonic crystal switches can be used to achieve equivalent functionality, and the laser scheme is adjustable. In extremely low-power or portable scenarios, an independent laser can be used for each node, sacrificing some energy efficiency to simplify the architecture. It has strong algorithm compatibility: suitable for various training tasks such as multimodal perception, cross-domain feature learning, and edge intelligent collaboration. Both the system structure and control logic adopt a modular design, which can be tailored or reused according to requirements, exhibiting high engineering adaptability.

[0120] Please refer to Figure 2 , Figure 2 This is a schematic flowchart of an embodiment of the optoelectronic interconnection collaborative training method for multimodal intelligent agent networks provided by the present invention, as shown in the example. Figure 2 As shown, the flowchart of the optoelectronic interconnection collaborative training method for multimodal intelligent agent networks provided by this invention is as follows:

[0121] 201. When the system is powered on, the beam splitter distributes the optical signal output from the external light source to each intelligent agent node and the programmable optical switching unit. The programmable optical switching unit establishes an optical connection link between any two intelligent agent nodes.

[0122] In this embodiment of the invention, after the system is powered on, the control and scheduling unit first identifies and initializes all hardware modules. The external light source starts and outputs stable continuous wave light, which is distributed to the photoelectric interface ports of each intelligent agent node via a beam splitter. The programmable optical switching unit establishes an optical connection link between any two intelligent agent nodes.

[0123] 202. The control and scheduling unit performs system initialization, obtains the link status of multiple optical connection links, and enters standby mode.

[0124] The system automatically detects the link status of each optical connection link, including connectivity, optical power, and bit error rate, and establishes a link status table. The link status table includes the link status of each optical connection link. Automatic frequency calibration is performed on the micro-ring or MZM modulator to ensure that the modulation wavelength of each node matches the external laser wavelength. After this stage is completed, the system enters standby mode.

[0125] 203. After each agent node is started, the control and scheduling unit obtains the agent modality mapping table of multiple agent nodes.

[0126] After each agent node starts up, it registers its own modal information, computing power, and task type with the control and scheduling unit. The control and scheduling unit obtains the registration information of multiple agent nodes and generates an agent modality mapping table. The registration information includes its own modal information, computing power, and task type. For example, a visual agent is responsible for image recognition, a speech agent is responsible for voiceprint analysis, and a radar agent is responsible for target localization. The control and scheduling unit generates a modality mapping table based on the registration information, providing a basis for subsequent task allocation and optical path establishment.

[0127] 204. After receiving the inference task, the training framework interface layer distributes the task according to the inference task and obtains the task distribution information. The task distribution information includes the task type of multiple sub-tasks, the participating nodes, and the amount of data transmitted between the participating nodes.

[0128] When the training framework interface layer receives a new training batch or inference task, it distributes the task according to the inference task and obtains the task distribution information, which includes the task type of multiple sub-tasks, participating nodes, and the amount of data transmitted between participating nodes.

[0129] The inference task is divided into multiple subtasks, and each subtask is completed by data transmission between multiple participating nodes.

[0130] 205. The control and scheduling unit controls the photoelectric interface module to select multiple first optical connection links based on the intelligent agent modality mapping table, the link status of each optical connection link, and the task distribution information.

[0131] In this embodiment of the invention, the training framework interface layer first determines whether the task requires cross-modal communication. If so, the interface layer sends the task distribution information to the control and scheduling unit.

[0132] The training framework interface layer first parses the input modality type and output requirements of the task: if the task only requires single-modality data (such as pure visual image classification) and can be computed independently by a single agent node, it is directly determined that "no cross-modal communication is needed," and the task will be distributed to the corresponding single-modality agent node without enabling the optical path configuration of the programmable optical switching unit throughout the process; if the task input contains two or more modalities of data (such as "visual + radar" target tracking, "voice + bioelectric" emotion recognition), then cross-modal communication is required. If necessary, the interface layer sends the task distribution information to the control and scheduling unit.

[0133] Based on the agent modality mapping table, the link status of each optical connection link, and task distribution information, the control and scheduling unit selects multiple first optical connection links in the programmable optical switching unit and sends control signals to the corresponding MZI units to establish optical path connections between agent nodes. The optical path is established within milliseconds, realizing direct optical connections between nodes.

[0134] The control and scheduling unit retrieves the modality mapping table and locks down the cooperative node pairs according to the modality requirements of the task. The cooperative node pairs include source nodes and target nodes. For example, the "vision + radar" task will match the "vision node" and "radar node" in the modality mapping table, and at the same time confirm that the OE-lite interface of the two nodes is available, excluding nodes with computing power overload or interface failure.

[0135] For example, the delay of an optical link is 0.3µs, and the bit error rate is... With an optical power of -2dBm and a bandwidth of 10Gbps, its overall score will be higher than that of a link with a latency of 0.6µs, making it the preferred choice for the first optical connection link.

[0136] 206. The intelligent agent nodes on the first optical connection link interact with each other to complete the sub-task.

[0137] After the optical path is established, the source node modulates its local feature or gradient data into an optical signal via the optoelectronic interface module and transmits it. The target node then converts the signal back into an electrical signal via the optoelectronic interface module and inputs it into its local accelerator for computation. The entire data transmission process is point-to-point or multi-point parallel optical communication, eliminating the need for intermediate buffers or repetitive encoding, thus significantly reducing latency. During training, multiple agents can share features and synchronize gradients in parallel, achieving true multimodal collaborative learning.

[0138] 207. When a subtask is completed, the control scheduling unit releases the first optical connection link corresponding to the subtask and reallocates it.

[0139] When a subtask is completed or communication demand decreases, the control and scheduling unit automatically releases the corresponding optical path resources. If a new high-priority task is detected or the link quality deteriorates, the system can switch to an alternative path or adjust the wavelength channel in real time without affecting the current training. Through this mechanism, the system can maintain high throughput and stability over a long period of time.

[0140] During system operation, the control and scheduling unit continuously monitors optical power, temperature, and bit error rate. When the load is low, it can automatically shut down some idle channels or modulators to reduce power consumption. When a link anomaly or a drop in optical power is detected, the system will automatically reconfigure the optical path or fall back to the electrical channel to ensure communication continuity and correct task execution. Intelligent agent nodes communicate via both optical and electrical links. When a link anomaly or a drop in optical power is detected, the system falls back to the electrical channel to ensure communication continuity and correct task execution.

[0141] Through the above process, this system can achieve efficient collaborative training among multimodal intelligent agents and maintain stable operation under various loads and environmental changes.

[0142] Compared with existing multimodal training systems, the present invention has the following significant advantages:

[0143] Communication latency is significantly reduced: By establishing direct optical paths between agent nodes, the serial transmission and multi-level switching of traditional electrical interconnection are avoided, and the single-hop transmission latency can be controlled at the microsecond level, which greatly improves the training synchronization efficiency.

[0144] Significantly reduced power consumption: By adopting a centralized external laser light supply and short-distance optoelectronic interface module, the overall power consumption of the system is reduced by more than 50% compared with traditional optical modules or high-speed cables, meeting the low power consumption requirements of edge computing and mobile platforms.

[0145] Strong reconfigurability and adaptability: The programmable optical switching unit can dynamically adjust the connection mode according to the task, realize fast rerouting between any nodes, and support multi-task collaboration and resource reuse between different modes.

[0146] Excellent multimodal compatibility: The system supports the parallel access of multiple modal nodes such as vision, voice, infrared, and radar, and can flexibly adapt to multimodal fusion and cross-domain collaborative tasks, making it suitable for complex application scenarios such as intelligent sensing, unmanned systems, and security monitoring.

[0147] High reliability and maintainability: The centralized light supply structure simplifies light source management, and the control and scheduling unit has real-time monitoring and fault tolerance mechanisms, which can quickly restore communication in the event of a link failure, significantly improving the stability of system operation.

[0148] Easy to expand and integrate: The system adopts a modular design and can be linearly expanded in the range of 4–32 nodes. It can also be integrated with on-chip optoelectronic computing chips, optical neural networks or edge sensing units to form a reconfigurable intelligent optical computing platform.

[0149] In summary, this invention outperforms existing technologies in terms of energy consumption, bandwidth, latency, and flexibility, and can provide a highly efficient underlying communication and collaborative computing solution for multimodal artificial intelligence systems.

[0150] The foregoing has provided a detailed description of the optoelectronic interconnection collaborative training system and method for multimodal intelligent agent networks provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0151] It should be noted that when the above embodiments of the present invention are applied to specific products or technologies, and user-related data is involved, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A photoelectric interconnection collaborative training system for multimodal intelligent agent networks, characterized in that, include: Multiple intelligent agent nodes, each intelligent agent node including a photoelectric interface module, are used to process data of multiple different modalities; A programmable optical switching unit is provided, which is connected to the optoelectronic interface module in each of the intelligent agent nodes via optical fiber. The programmable optical switching unit is used to establish or disconnect the optical connection link between any two of the intelligent agent nodes. An external light source is used to output a continuous light signal; The beam splitter is connected to each of the intelligent agent nodes and the programmable optical switching unit respectively. The beam splitter is used to distribute the optical signal output by the external light source to each of the intelligent agent nodes and the programmable optical switching unit. The training framework interface layer is connected to the beam splitter. The training framework interface layer is used to receive inference tasks and distribute tasks according to the inference tasks to obtain task distribution information. The task distribution information includes the task types of multiple sub-tasks, participating nodes, and the amount of data transmitted between participating nodes. A control and scheduling unit is connected to the programmable optical switching unit, the training framework interface layer, and each of the agent nodes. When the control and scheduling unit obtains task distribution information, it obtains the agent modality mapping table of multiple agent nodes and the link status of each optical connection link. Based on the agent modality mapping table, the link status of each optical connection link, and the task distribution information, it controls the optoelectronic interface module to select multiple first optical connection links. The agent nodes on the first optical connection links interact with each other to complete the inference task. After each agent node is started, it registers its own modal information, computing power and task type with the control and scheduling unit. The control and scheduling unit obtains the registration information of multiple agent nodes and generates an agent modal mapping table. The registration information includes its own modal information, computing power and task type.

2. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 1, characterized in that, The programmable optical switching unit includes multiple MZI units, which are spliced ​​together in a matrix topology to form an optical matrix, thereby establishing or disconnecting optical connection links between any two intelligent agent nodes.

3. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 2, characterized in that, The MZI unit is a 2×2 matrix MZI unit, which switches between two input ports and two output ports.

4. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 1, characterized in that, The optoelectronic interface module includes a short-range SerDes interface, and the optoelectronic interface module is connected to the programmable optical switching unit through the short-range SerDes interface.

5. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 4, characterized in that, The optoelectronic interface module includes an electric drive amplifier, a modulation unit, a wavelength selection element, a photodetector, and a set of local control interfaces.

6. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 1, characterized in that, The control and scheduling unit detects the link status of the first optical connection link at a preset period; when the first optical connection link is abnormal, it switches the first optical connection link to the second optical connection link.

7. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 1, characterized in that, When the load on the first optical connection link is less than a preset value, the control scheduling unit disconnects the first optical connection link.

8. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 1, characterized in that, Multiple agent nodes include at least two of the following: voice and vision agents, voice agents, bioelectric agents, and radar agents.

9. The optoelectronic interconnection collaborative training system for multimodal intelligent agent networks according to claim 1, characterized in that, The interaction between the intelligent agent nodes on the first optical connection link includes: Determine the source node and target node among the agent nodes on the first optical connection link; The source node modulates local feature or gradient data into an optical signal via the photoelectric interface module and sends it to the target node. The target node converts the optical signal back into an electrical signal through the photoelectric interface module and inputs it into the local accelerator for calculation.

10. A method for optoelectronic interconnection collaborative training of multimodal agent networks, based on the optoelectronic interconnection collaborative training system for multimodal agent networks as described in any one of claims 1-9, characterized in that, include: When the system is powered on, the beam splitter distributes the optical signal output by the external light source to each of the intelligent agent nodes and the programmable optical switching unit. The programmable optical switching unit establishes an optical connection link between any two of the intelligent agent nodes. The control and scheduling unit initializes the system, obtains the link status of multiple optical connection links, and enters standby mode. After each of the intelligent agent nodes is started, the control and scheduling unit obtains the intelligent agent modality mapping table of multiple intelligent agent nodes; After receiving the inference task, the training framework interface layer distributes the task according to the inference task to obtain task distribution information. The task distribution information includes the task type of multiple sub-tasks, participating nodes, and the amount of data transmitted between participating nodes. The control and scheduling unit controls the photoelectric interface module to select multiple first optical connection links based on the intelligent agent modality mapping table, the link status of each optical connection link, and task distribution information. The intelligent agent nodes on the first optical connection link interact to complete the sub-task; When the subtask is completed, the control scheduling unit releases and reallocates the first optical connection link corresponding to the subtask; After each agent node is started, it registers its own modal information, computing power and task type with the control and scheduling unit. The control and scheduling unit obtains the registration information of multiple agent nodes and generates an agent modal mapping table. The registration information includes its own modal information, computing power and task type.

Images