Construction method and apparatus for optical-electrical hybrid data center network, and medium

Abstract

The present invention relates to the technical field of digital information transmission, and in particular, to a construction method and apparatus for an optical-electrical hybrid data center network, and a medium. The method comprises: generating an optical-electrical hybrid network topology for a large model on the basis of a tree topology, an optical circuit switch, and an electrical packet switch; acquiring a target topology allocation scheme on the basis of target communication requirements and a target scenario, and using the target topology allocation scheme to reconfigure the network topology to obtain a reconfigured network topology; and optimizing a network transport layer on the basis of the reconfigured network topology, an artificial intelligence task, and characteristics of an optical-electrical hybrid cluster to obtain an optimization result, and constructing an optical-electrical hybrid intelligent computing network on the basis of the optimization result. Thus, the problems in the related art that using an optical-electrical hybrid networking approach for large model training requires additional control measures for scheduling, leading to reduced communication efficiency, diminished flexibility in network topology adjustment, and an inability to meet the use requirements of users are solved.

Description

Method and device for constructing optoelectronic hybrid data center network and medium

[0001] Cross-reference to related applications

[0002] The present application is based on and claims priority to Chinese patent application No. 202410467177.6, filed on April 18, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of digital information transmission, in particular to a method and device for constructing an optoelectronic hybrid data center network and a medium. BACKGROUND

[0004] Model training is a complex task that requires a large amount of computing resources and efficient data transmission. Currently, the main large model training schemes include using electrical packet switches and optical path switches, i.e., network topologies of electrical switches and optical switches for training, and these training clusters usually use collection communication libraries for data synchronization and communication.

[0005] Electrical switches use a traditional packet-forwarding architecture, which is a widely used device. The switching function of the electrical switch is provided by its core switching chip, and the capacity of the switching chip determines the capacity of the electrical switch. For example, NVLink and NVSwitch of NVIDIA use an electrical switch architecture, which optimizes data transmission and communication efficiency through advanced chip manufacturing technology, enabling efficient training of large models. Optical switches are different from electrical switches. Optical switches do not have a packet-forwarding process, but directly perform one-to-one mapping between ports. Typically, optical switches use mechanical elements to control a number of small mirrors to complete the switching process using light reflection. Compared with electrical switches, optical switches do not require ultra-high performance switching chips, which can reduce production barriers and operation and maintenance costs. However, optical switches do not provide many-to-many communication between ports and require additional control measures for scheduling. Therefore, a combination of optical switches and electrical switches can be used to construct an efficient and low-cost data center cluster to support upper-layer artificial intelligence training and inference tasks.

[0006] However, the related art uses an optoelectronic hybrid networking method for large model training, which lacks an adaptive software system and related scheduling algorithms to support large model training, requiring additional control measures for scheduling, which reduces communication efficiency and reduces the flexibility of adjusting the network topology, failing to meet the user's use requirements and reducing the user's use experience, which needs to be solved urgently.

[0007] SUMMARY

[0008] The application provides a method and device for constructing an optoelectronic hybrid data center network and a medium to solve the problem that in related technologies, large model training is performed in an optoelectronic hybrid networking manner, additional control measures are required for scheduling, communication efficiency is reduced, flexibility of adjusting network topology is reduced, and user use requirements cannot be met.

[0009] The first aspect of the application provides a method for constructing an optoelectronic hybrid data center network, including the following steps: generating a network topology of a large model optoelectronic hybrid based on a tree topology, an optical path switch and an electrical packet switch; obtaining a target topology distribution scheme according to a target communication requirement and a target scene, and adjusting the network topology by using the target topology distribution scheme to obtain an adjusted network topology; and optimizing a network transmission layer according to the adjusted network topology, an artificial intelligence task and an optoelectronic hybrid cluster feature to obtain an optimization result, and constructing an optoelectronic hybrid data center network according to the optimization result.

[0010] Optionally, in an embodiment of the application, the adjusting of the network topology by using the target topology distribution scheme includes: generating the network topology by using a user requirement based on a static topology distribution scheme of the target topology distribution scheme; or adjusting the network topology by using a traffic load of the target scene based on a dynamic adaptability topology distribution scheme of the target topology distribution scheme.

[0011] Optionally, in an embodiment of the application, the adjusting of the network topology by using the dynamic adaptability topology distribution scheme of the target topology distribution scheme includes: constructing an optical path-aware RDMA (Remote Direct Memory Access) collective communication library according to the network topology; judging whether a current network topology meets a communication condition requested by a user based on the RDMA collective communication library; if the communication condition is not met, performing a target strategy corresponding to the current network topology to perform large-scale artificial intelligence model training according to the target strategy to obtain the adjusted network topology; and if the communication condition is met, performing large-scale artificial intelligence model training to obtain the adjusted network topology.

[0012] Optionally, in an embodiment of the application, the large-scale artificial intelligence model training according to the target strategy to obtain the adjusted network topology includes: calculating an optimal topology of the current network topology based on a hybrid cluster controller in the network topology; and downloading the optimal topology to the optical path switch to perform large-scale artificial intelligence model training by using the optical path switch to obtain the adjusted network topology.

[0013] The second aspect embodiment of the present application provides a device for constructing an optoelectronic hybrid data center network, comprising: a generating module configured to generate a large model optoelectronic hybrid network topology based on a tree topology, an optical path switch and an electrical packet switch; a determining module configured to obtain a target topology distribution scheme according to a target communication requirement and a target scene, and to adjust the network topology by using the target topology distribution scheme to obtain an adjusted network topology; and a constructing module configured to obtain an optimization result according to the adjusted network topology, an artificial intelligence task and an optoelectronic hybrid cluster feature optimization network transmission layer, and to construct an optoelectronic hybrid data center network according to the optimization result.

[0014] Optionally, in an embodiment of the present application, the determining module comprises: a generating unit configured to generate the network topology by using a user requirement based on a static topology distribution scheme of the target topology distribution scheme, or to adjust the network topology by using a traffic load of the target scene based on a dynamic adaptability topology distribution scheme of the target topology distribution scheme.

[0015] Optionally, in an embodiment of the present application, the generating unit comprises: an accessing subunit configured to construct an optical path aware remote direct memory access (RDMA) collective communication library according to the network topology; a judging subunit configured to judge whether a current network topology satisfies a communication condition requested by a user based on the RDMA collective communication library; a first processing subunit configured to execute a target strategy corresponding to the current network topology to perform large-scale artificial intelligence model training according to the target strategy to obtain the adjusted network topology if the communication condition is not satisfied; and a second processing subunit configured to perform large-scale artificial intelligence model training to obtain the adjusted network topology if the communication condition is satisfied.

[0016] Optionally, in an embodiment of the present application, the first processing subunit is further configured to calculate an optimal topology of the current network topology based on a hybrid cluster controller in the network topology, to issue the optimal topology to the optical path switch, and to perform large-scale artificial intelligence model training by using the optical path switch to obtain the adjusted network topology.

[0017] The third aspect embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for constructing an optoelectronic hybrid data center network as described in the above embodiments.

[0018] The fourth aspect embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the method for constructing an optoelectronic hybrid data center network as described above.

[0019] The fifth aspect of the present application provides a computer program, which is executed to implement the method for constructing the optoelectronic hybrid data center network as above.

[0020] The embodiment of the present application can obtain a target topology distribution scheme according to target communication requirements and target scenarios, deploy the network topology of the optoelectronic hybrid network using the target topology distribution scheme, optimize the network transmission layer according to the deployed network topology, artificial intelligence tasks, and optoelectronic hybrid cluster characteristics to obtain an optimization result, and construct the optoelectronic hybrid data center network according to the optimization result, thereby effectively improving the communication efficiency and the flexibility of adjusting the network topology, and meeting the use requirements of users. Thus, the problem that in the related art, the optoelectronic hybrid networking method is used for large model training, and additional control measures are required for scheduling, thereby reducing the communication efficiency and the flexibility of adjusting the network topology, and failing to meet the use requirements of users, is solved.

[0021] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0023] Fig. 1 is a flowchart of a method for constructing an optoelectronic hybrid data center network according to an embodiment of the present application;

[0024] Fig. 2 is an architecture diagram of an optoelectronic hybrid data center network according to an embodiment of the present application;

[0025] Fig. 3 is a schematic diagram of the principle of constructing an optoelectronic hybrid network according to an embodiment of the present application;

[0026] Fig. 4 is a schematic diagram of a commonly used topology of an optical switch according to an embodiment of the present application;

[0027] Fig. 5 is a schematic diagram of a static topology distribution scheme according to an embodiment of the present application;

[0028] Fig. 6 is a schematic diagram of a dynamic adaptive topology distribution scheme according to an embodiment of the present application;

[0029] Fig. 7 is a schematic diagram of an optimized RDMA scheduler structure according to an embodiment of the present application;

[0030] Fig. 8 is a schematic diagram of a structure of a device for constructing an optoelectronic hybrid data center network according to an embodiment of the present application;

[0031] FIG. 9 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or similar components have the same or similar reference numbers throughout the drawings and a detailed description of the same or similar components is not repeated. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be understood as limiting the present application.

[0033] A method, device and medium for constructing an optoelectronic hybrid data center network are described below with reference to the accompanying drawings. In order to solve the problem that the related art uses an optoelectronic hybrid networking method for large model training, requires additional control measures for scheduling, reduces communication efficiency, reduces the flexibility of adjusting the network topology, and cannot meet the user's use requirements, the present application provides a method for constructing an optoelectronic hybrid data center network. In the method, a target topology distribution scheme can be obtained according to target communication requirements and target scenarios, the network topology of the optoelectronic hybrid large model can be adjusted using the target topology distribution scheme, an optimized result can be obtained by optimizing the network transmission layer according to the adjusted network topology, artificial intelligence tasks, and optoelectronic hybrid cluster characteristics, and an optoelectronic hybrid data center network can be constructed according to the optimized result, thereby effectively improving the communication efficiency and the flexibility of adjusting the network topology, and meeting the user's use requirements. Thus, the problem that the related art uses an optoelectronic hybrid networking method for large model training, requires additional control measures for scheduling, reduces communication efficiency, reduces the flexibility of adjusting the network topology, and cannot meet the user's use requirements is solved.

[0034] Specifically, FIG. 1 is a flowchart of a method for constructing an optoelectronic hybrid data center network according to an embodiment of the present application.

[0035] As shown in FIG. 1, the method for constructing an optoelectronic hybrid data center network includes the following steps:

[0036] In step S101, a network topology of an optoelectronic hybrid large model is generated based on a tree topology, an optical path switch, and an electrical packet switch.

[0037] It can be understood that the embodiment of the application can generate a large model optical-electric hybrid network topology based on a tree topology, an optical path switch and an electric packet switch. For example, as shown in FIG. 2, the embodiment of the application can design a large model optical-electric hybrid network topology based on a tree topology, using an optical path switch in the core layer and an electric packet switch in the edge layer. The optical-electric hybrid cluster in FIG. 2 is composed of one cluster controller, a plurality of optical path switching devices, a plurality of electric packet switching devices and a plurality of hosts. The cluster controller can be indirectly or directly connected with all the hosts, the electric packet switching devices and the optical path switching devices. The cluster controller can be responsible for tasks such as traffic demand collection, dynamic topology calculation, optical switch control and electric switch control. The hosts have a set communication library and an RDMA communication library, which are connected with the electric switch through a wired link. The electric switch is connected with the optical switch through a wired link, thereby effectively improving the executability of constructing an optical-electric hybrid data center network.

[0038] Specifically, the embodiment of the application can take a double-layer 1024-card cluster as an example. Four servers are placed in one cabinet. The four servers can provide 16 acceleration cards and access to the same top switch (access layer switch). The top switch has 8 downlink ports and 8 uplink ports, a total of 16 ports. The single port bandwidth matches the network port bandwidth on the server. The top switch also has a control port connected to the control switch. The embodiment of the application deploys a total of 64 cabinets, a total of 64x16=1024 acceleration cards, and each cabinet has 8 uplink ports waiting to be connected. Further, as shown in FIG. 3, the embodiment of the application deploys 8 optical path switches. Each optical path switch is connected to the ports of each cabinet with a fixed offset. That is, the first optical path switch is connected to the first port of all top switches, the second optical path switch is connected to the second port of all top switches, and so on. Each optical path switch occupies 64 ports. Through the design of the large model optical-electric hybrid network topology, an efficient and scalable large model training solution can be provided, effectively improving the user experience.

[0039] For example, the large-scale optical-electric hybrid network topology deployed by the embodiment of the application can at least support the large model training of a thousand-card network and apply a hybrid cluster controller. The hybrid cluster controller can collect traffic demand and notify the cluster controller through an out-of-band network. In addition, the embodiment of the application can implement the wrapped communication library and topology request interface in the server software layer through the hybrid cluster controller, support dynamic topology adjustment, and reduce the topology scheduling time in the training process.

[0040] In step S102, a target topology allocation scheme is obtained according to a target communication demand and a target scene. The network topology is adjusted using the target topology allocation scheme, and the adjusted network topology is obtained.

[0041] It can be understood that the embodiment of the application obtains a target topology distribution scheme according to target communication requirements and target scenarios, for example, a target topology distribution scheme can be obtained according to different tenant scenarios and different traffic requirements, and the network topology is deployed by using the target topology distribution scheme in the following steps to obtain a deployed network topology, which effectively improves the communication efficiency and improves the flexibility of adjusting the network topology.

[0042] For example, in large-scale artificial intelligence model training, the topology control method of optical path switching and the selection of the topology distribution scheme are crucial to improving the training efficiency and performance of large models. Therefore, the embodiment of the application can use an optical-electric hybrid topology calculation and control scheme based on GenModel for network topology control.

[0043] In the embodiment of the application, still taking the double-layer 1024-card cluster in the above steps as an example, in a single-tenant scenario, the embodiment of the application can connect all 64 cabinets using an optical path switch, according to different traffic requirements of the upper-layer artificial intelligence task, and according to the suggestions given by GenModel, the optical switch can select different connection modes, for example, as shown in FIG. 4, for a set communication operator in which the main communication occurs between neighbors, such as Ring-AllReduce, a ring topology of FIG. 4(a) can be used; for a set communication operator involving a large number of many-to-many traffic, such as AlltoAll, a Chord topology of FIG. 4(b) can be used; for a scenario in which the communication traffic is complex and irregular and changes frequently, a dynamic hybrid topology, such as the TopoOpt topology of FIG. 4(c), can be used.

[0044] In a multi-tenant scenario, the embodiment of the application can use the characteristics of the optical path switch to physically isolate different tenants, and the cabinets in each tenant are connected to each other, and the connection mode in the tenant is consistent with that in the single-tenant scenario. Since the optical path switch is physically equivalent to the operation of “pulling and plugging a network cable”, the routing table and the MAC address table of the set-top switch will change with the switching of the optical path switch. In order to ensure the correctness of the routing and avoid the time of automatic detection of the MAC address table, the routing and MAC table on all set-top switches will be controlled by the hybrid cluster controller via the control port of the switch. After the connection of the optical path switch is changed, the hybrid cluster controller issues relevant data to the set-top switch, thereby effectively improving the training efficiency of the large model.

[0045] Therefore, the embodiment of the application can flexibly adjust and optimize the network topology according to different communication requirements and scenarios based on the static and dynamic topology control schemes of the optical switch, thereby improving the communication efficiency.

[0046] Further, in an embodiment of the present application, the network topology is deployed using the target topology allocation scheme, including: generating the network topology using user demand based on the static topology allocation scheme of the target topology allocation scheme; or adjusting the network topology using the traffic load of the upper artificial intelligence task of the target scene based on the dynamic adaptability topology allocation scheme of the target topology allocation scheme.

[0047] For example, the target topology allocation scheme in the embodiment of the present application can include both static topology allocation scheme and dynamic adaptability topology allocation scheme, wherein, as shown in FIG. 5(a), the static topology allocation scheme can deploy a fixed topology in advance, such as ring topology or Chord topology, according to the traffic characteristics of the task before running the task, i.e., before allocating resources to the user, and control the optical path switch to switch to the specified state, and then the topology will not be changed during the task execution, i.e., the user can directly run the large model training without any operation, achieving transparent and non-perception, in addition, part of the topology control API (Application Programming Interface) can be opened to advanced users, so that they can change the network topology during the task, so that advanced users can customize the network topology according to their own needs.

[0048] In addition, as shown in FIG. 5(b), the dynamic adaptability topology allocation scheme can execute judgment whether to change the topology every time the upper layer calls the communication demand during the task execution, when the topology needs to be changed, the topology is changed, when the topology does not need to be changed, the topology remains unchanged, the dynamic adaptability topology allocation scheme can modify and package the communication library such as NCCL (NVIDIA Collective Communication Library) and HCCL (Huawei Collective Communication Library), and can construct an optical path-aware RDMA collective communication library, so that the network topology can be dynamically and adaptively adjusted and optimized according to the traffic load of the target scene.

[0049] In an embodiment of the present application, the dynamic adaptive topology allocation scheme based on the target topology allocation scheme adjusts the network topology according to the traffic load of the target scene, including: constructing a remote direct memory access (RDMA) collective communication library with light path awareness according to the network topology; determining whether the current network topology meets the communication condition requested by the user based on the RDMA collective communication library; if the communication condition is not met, performing the target strategy corresponding to the current network topology to train the large-scale artificial intelligence model according to the target strategy, and obtaining the adjusted network topology; if the communication condition is met, training the large-scale artificial intelligence model, and obtaining the adjusted network topology.

[0050] For example, in combination with FIGS. 2 and 6, the embodiment of the present application can construct a remote direct memory access (RDMA) collective communication library with light path awareness according to the network topology, and when the user calls the interface of the RDMA collective communication library, first determine whether the current network topology supports the communication requested by the user, when the requested communication is not supported, initiate a request to the hybrid cluster controller, perform the target strategy corresponding to the current network topology to train the large-scale artificial intelligence model according to the target strategy in the following steps, and obtain the adjusted network topology; when the requested communication is supported, directly train the large-scale artificial intelligence model, and obtain the adjusted network topology, effectively improving the communication efficiency.

[0051] In an embodiment of the present application, the large-scale artificial intelligence model is trained according to the target strategy to obtain the adjusted network topology, including: calculating the optimal topology of the current network topology based on the hybrid cluster controller in the network topology; and downloading the optimal topology to the light path switch to train the large-scale artificial intelligence model using the light path switch to obtain the adjusted network topology.

[0052] For another example, the embodiment of the present application can calculate the optimal topology based on the hybrid cluster controller in the network topology, download the optimal topology to the optical switch, and adjust the topology of the optoelectronic hybrid cluster to the optimal state, and then train the large-scale artificial intelligence model using the optoelectronic hybrid cluster, wherein the collective communication involved in the artificial intelligence model training will benefit from the optimized topology, thereby obtaining the adjusted network topology. In addition, in order to avoid additional waiting time, the embodiment of the present application can also provide an interface for the user to inform the next communication characteristics in advance, and schedule the network topology in advance, thereby improving the communication efficiency.

[0053] In step S103, the optimized result is obtained by optimizing the network transmission layer according to the adjusted network topology, the artificial intelligence task, and the optoelectronic hybrid cluster characteristics, and the optoelectronic hybrid data center network is constructed according to the optimized result.

[0054] It can be understood that the embodiment of the present application can be combined with the characteristics of the artificial intelligence task and the optoelectronic hybrid cluster according to the deployed network topology, wherein the artificial intelligence task refers to the training and inference of the artificial intelligence model, and the characteristics of the optoelectronic hybrid cluster refer to the features of the optoelectronic hybrid cluster. The switches in the optoelectronic hybrid cluster include optical path switches and traditional packet forwarding switches. The optical path switch is a new type of switching device realized by one-to-one reflection between ingress and egress ports, usually controlled by MEMS (Micro Electromechanical System), further optimizes the network transmission layer, and constructs an optoelectronic hybrid data center network according to the optimization result. The optimization of the network transmission layer includes RDMA optimization mechanism, congestion control system and load balancing system, effectively improves the communication efficiency, and improves the flexibility of adjusting the network topology, and meets the use requirements of users.

[0055] For example, as shown in FIG. 7, the embodiment of the present application can optimize the RDMA scheduler. Mainly by implementing software-defined RDMA (sRDMA) in the user-mode driver RDMA scheduler to improve data transmission performance. sRDMA is a technology that realizes software control on RDMA hardware, which can provide more flexible and efficient data transmission services.

[0056] In actual execution process, the embodiment of the present application can encapsulate scheduling information on the user-mode application program interface (APIs) of RDMA, in order to facilitate the transmission of application scheduling requirements. This encapsulation method enables the application program to directly control the scheduling of RDMA, thereby meeting the specific needs of different applications.

[0057] In addition, in order to realize more efficient data transmission scheduling, the embodiment of the present application can design a scheduler. The scheduler can modify the transmission request (WR) according to the transmission priority of the application to realize scheduling. This priority-based scheduling method can ensure that high-priority data transmission requests are processed first, thereby improving the response speed and performance of the system. In addition, the embodiment of the present application also designs a splitter. The splitter can split data transmission exceeding a preset threshold, such as 1MB, to alleviate the head-of-line blocking caused by RDMA long data transmission, and further improve the scheduling space. This splitting method can effectively solve the queue blocking problem caused by too large data transmission request, and further improve the efficiency of data transmission.

[0058] The setting of the preset threshold is an empirical result. For example, the maximum message transmission allowed by RDMA is 2GB. Therefore, the threshold can be set to 1MB. In this way, the scheduling can be fully performed, and unnecessary processing overhead caused by too small splitting can be avoided.

[0059] In order to improve the performance and scalability of the distributed application, the embodiment of the present application constructs an efficient communication library, optimizes the RDMA mechanism, and realizes a congestion control system and a load balancing system. These optimizations and improvements can not only improve the efficiency of data transmission, but also improve the stability and scalability of the optoelectronic hybrid data center network system.

[0060] In the embodiment of the present application, the transmission process of RDMA mainly includes three steps of processing transmission requests, data transmission and delivering transmission completion notification. In the transmission process after adding the scheduler, steps of checking transmission requests, dividing transmission requests, scheduling transmission requests and recombining transmission completion notification are added. These newly added steps can make the data transmission process more flexible and efficient.

[0061] In summary, in the optimization of the RDMA scheduler, by realizing sRDMA, optimizing the data transmission process and improving the scheduling efficiency, the use efficiency of RDMA and the network performance are effectively improved, the data transmission is optimized, and the operation efficiency of the optoelectronic hybrid data center network system and the performance and scalability of the distributed application are improved.

[0062] For example, the computing power required for training a large model increases exponentially, and increases by 750 times every two years, far exceeding Moore's law. It is impossible to meet the demand by improving the performance of a single chip or a single card. Therefore, distributed parallel training is required for large models, which puts high requirements on the performance and size of the network. The optoelectronic hybrid networking scheme has the advantages of high performance, low cost and flexible topology, and can effectively solve the above problems.

[0063] The construction method of the optoelectronic hybrid data center network according to the embodiment of the present application can obtain a target topology distribution scheme according to target communication requirements and target scenarios, deploy the network topology of the optoelectronic hybrid of the large model by using the target topology distribution scheme, optimize the network transmission layer according to the deployed network topology, artificial intelligence task and optoelectronic hybrid cluster characteristics to obtain an optimization result, and construct the optoelectronic hybrid data center network according to the optimization result, effectively improving the communication efficiency and the flexibility of adjusting the network topology, meeting the use requirements of users. Thus, the problem that in the related art, the optoelectronic hybrid networking scheme is used for large model training, and additional control measures are required for scheduling, reducing the communication efficiency and the flexibility of adjusting the network topology, and failing to meet the use requirements of users is solved.

[0064] Secondly, the construction device of the optoelectronic hybrid data center network according to the embodiment of the present application is described with reference to the accompanying drawings.

[0065] FIG. 8 is a block schematic diagram of the construction device of the optoelectronic hybrid data center network according to the embodiment of the present application.

[0066] As shown in FIG. 8, the construction device 10 of the optoelectronic hybrid data center network includes a generation module 100, a determination module 200, and a construction module 300.

[0067] Specifically, the generation module 100 is configured to generate a large model optoelectronic hybrid network topology based on a tree topology, an optical path switch, and an electrical packet switch.

[0068] The determination module 200 is configured to obtain a target topology allocation scheme according to a target communication requirement and a target scene, and adjust the network topology by using the target topology allocation scheme to obtain an adjusted network topology.

[0069] The construction module 300 is configured to optimize a network transmission layer according to the adjusted network topology, an artificial intelligence task, and optoelectronic hybrid cluster characteristics to obtain an optimization result, and construct the optoelectronic hybrid data center network according to the optimization result.

[0070] Optionally, in an embodiment of the present application, the determination module 200 includes a generation unit.

[0071] The generation unit is configured to generate the network topology based on a static topology allocation scheme of the target topology allocation scheme, or adjust the network topology based on a dynamic adaptability topology allocation scheme of the target topology allocation scheme according to a traffic load of the target scene.

[0072] Optionally, in an embodiment of the present application, the generation unit includes an access subunit, a judgment subunit, a first processing subunit, and a second processing subunit.

[0073] The access subunit is configured to construct a remote direct memory access (RDMA) collective communication library with optical path awareness according to the network topology.

[0074] The judgment subunit is configured to judge whether the current network topology meets a communication condition requested by a user based on the RDMA collective communication library.

[0075] The first processing subunit is configured to execute a target strategy corresponding to the current network topology to perform large-scale artificial intelligence model training according to the target strategy to obtain the adjusted network topology if the communication condition is not met.

[0076] The second processing subunit is configured to perform large-scale artificial intelligence model training to obtain the adjusted network topology if the communication condition is met.

[0077] Optionally, in an embodiment of the present application, the first processing subunit is further configured to calculate an optimal topology of the current network topology based on a hybrid cluster controller in the network topology, and downlink the optimal topology to the optical path switch to perform large-scale artificial intelligence model training by using the optical path switch to obtain the adjusted network topology.

[0078] It should be noted that the foregoing explanation of the embodiment of the method for constructing the optoelectronic hybrid data center network is also applicable to the apparatus for constructing the optoelectronic hybrid data center network of the embodiment, which will not be described here again.

[0079] The apparatus for constructing the optoelectronic hybrid data center network according to the embodiment of the present application can obtain a target topology distribution scheme according to target communication requirements and target scenarios, use the target topology distribution scheme to adjust the network topology of the optoelectronic hybrid data center, optimize the network transmission layer according to the adjusted network topology, artificial intelligence tasks and optoelectronic hybrid cluster characteristics to obtain an optimization result, and construct the optoelectronic hybrid data center network according to the optimization result, which effectively improves the communication efficiency and improves the flexibility of adjusting the network topology, meeting the use requirements of users. Thus, the problem that in the related art, the optoelectronic hybrid networking method is used for large model training, and additional control measures are required for scheduling, which reduces the communication efficiency and reduces the flexibility of adjusting the network topology, and cannot meet the use requirements of users, is solved.

[0080] FIG. 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device can include:

[0081] The memory 901, the processor 902 and the computer program stored in the memory 901 and executable on the processor 902.

[0082] The processor 902 implements the method for constructing the optoelectronic hybrid data center network provided in the above embodiments when executing the program.

[0083] Further, the electronic device further includes:

[0084] The communication interface 903 is used for communication between the memory 901 and the processor 902.

[0085] The memory 901 is used to store the computer program executable on the processor 902.

[0086] The memory 901 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0087] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in FIG. 9, but it does not mean that there is only one bus or only one type of bus.

[0088] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can complete communication between each other through an internal interface.

[0089] The processor 902 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

[0090] The embodiment further provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the method for constructing an optoelectronic hybrid data center network as above.

[0091] The embodiment further provides a computer program, which, when executed by a processor, is used to implement the method for constructing an optoelectronic hybrid data center network as above.

[0092] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. In addition, the usage of "first", "second" and the like does not indicate any order but rather serves merely to name various components. Moreover, the usage of "top", "bottom", and the like is made for the purpose of illustration only and does not indicate any orientation. The terms "coupled" and "connected", along with their derivatives, can be used. It should be understood that these terms are not intended as synonyms for each other. Rather, particular features are described as being coupled or connected where the feature is in some way present, for example through shared use of one or more components, and can be communicatively, electrically, structurally, and / or mechanically connected, for example. Similarly, "coupled" or "connected" can be used to indicate that two or more members are either directly in contact or indirectly in contact through one or more intermediate members.

[0093] Furthermore, the terms "first", "second", and the like, merely denote different categories, and do not imply a relative importance or a specific order. Thus, features defined with "first", "second" and the like can include at least one of the features, either explicitly or implicitly. In the description of the application, the term "N" means at least two, for example two, three, etc., unless explicitly specified otherwise.

[0094] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments of modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps, and alternate implementations are possible. In some embodiments, the processes or methods described can be accomplished with one or more hardware items, for example, hardwired circuits, memory, logic circuits, look-up tables, microcode or the like, software programs, firmware programs, microcode routines, embedded logic, embedded software, or any combination thereof, which work together to cause a general purpose computer, a special purpose computer, or both, to perform the processes or methods described. The various embodiments further can interact with a user through one or more computer programs, software applications, firmware applications, operating systems, or the like, which interact with a user. Such software can be written in any of a variety of suitable programming languages and can be executed using a variety of suitable hardware and software configurations. It will be appreciated that computer programs, software applications, firmware applications, operating systems, or the like, can be written in any combination of one or more suitable programming languages, and that such software can be executed using one or more computing devices capable of netlist generation as described herein.

[0095] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy, flexible or other), a machine-readable storage card (e.g., ROM, EEPROM, flash memory or other), a machine- readable storage tape (e.g., magnetic, optical or other), a machine-readable storage medium (e.g., a portable electronic device, a computer diskette, a computer memory stick, a computer hard drive, a computer tape, a computer readable storage medium, or other), or a machine-readable wireless transmission (e.g., a radio frequency signal, an infrared signal, a microwave signal, or other). Further, the computer-readable medium can even be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, via optical scanning of the paper or other suitable medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory to execute the computer program.

[0096] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies, alone or in any combination, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0097] Those of skill in the art could readily implement the above described example methods with all or a subset of the recited steps carried out with a program of instructions directed to the relevant hardware, the program of instructions being stored in a computer readable storage medium which, when executed, includes one or a combination of the steps of the example methods.

[0098] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0099] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for constructing an optoelectronic hybrid data center network, characterized in that, The method comprises the following steps: Based on the tree topology, optical path switch and electrical packet switch, a network topology of large model optical-electric hybrid is generated; According to the target communication demand and the target scene, a target topology allocation scheme is obtained, and the network topology is adjusted by using the target topology allocation scheme to obtain an adjusted network topology; According to the adjusted network topology, the artificial intelligence task and the optical-electric hybrid cluster feature, an optimization result of the network transmission layer is obtained, and an optical-electric hybrid data center network is constructed according to the optimization result.

2. The method of constructing an opto-electric hybrid data center network according to claim 1, wherein, The network topology is adjusted by using the target topology allocation scheme, comprising: Based on the static topology allocation scheme of the target topology allocation scheme, the network topology is generated by using the user demand; Or, based on the dynamic adaptability topology allocation scheme of the target topology allocation scheme, the network topology is adjusted by using the traffic load of the target scene.

3. The method of constructing an opto-electric hybrid data center network of claim 2, wherein, The network topology is adjusted by using the dynamic adaptability topology allocation scheme of the target topology allocation scheme, comprising: According to the network topology, a remote direct memory access (RDMA) collective communication library with optical path awareness is constructed; Based on the RDMA collective communication library, it is judged whether the current network topology meets the communication condition requested by the user; If the communication condition is not met, the target strategy corresponding to the current network topology is executed to perform large-scale artificial intelligence model training according to the target strategy to obtain the adjusted network topology; If the communication condition is met, large-scale artificial intelligence model training is performed to obtain the adjusted network topology.

4. The method of constructing an opto-electric hybrid data center network of claim 3, wherein, The large-scale artificial intelligence model training according to the target strategy is performed to obtain the adjusted network topology, comprising: Based on the hybrid cluster controller in the network topology, the optimal topology of the current network topology is calculated; The optimal topology is sent to the optical path switch, and large-scale artificial intelligence model training is performed by using the optical path switch to obtain the adjusted network topology.

5. A device for constructing an optoelectronic hybrid data center network, characterized in that, Comprise: A generation module is configured to generate a network topology of large model optical-electric hybrid based on a tree topology, an optical path switch and an electrical packet switch; A determination module is configured to obtain a target topology allocation scheme according to a target communication demand and a target scene, and adjust the network topology by using the target topology allocation scheme to obtain an adjusted network topology; A construction module is configured to obtain an optimization result of a network transmission layer according to the adjusted network topology, an artificial intelligence task and an optical-electric hybrid cluster feature, and construct an optical-electric hybrid data center network according to the optimization result.

6. The apparatus according to claim 5, wherein, The determination module comprises: A generation unit is configured to generate the network topology by using a user demand based on a static topology allocation scheme of the target topology allocation scheme, or adjust the network topology by using a traffic load of the target scene based on a dynamic adaptability topology allocation scheme of the target topology allocation scheme.

7. The apparatus according to claim 6, wherein, The generation unit comprises: An access subunit is configured to construct a remote direct memory access (RDMA) collective communication library with optical path awareness according to the network topology; A judgment subunit is configured to judge whether the current network topology meets the communication condition requested by a user based on the RDMA collective communication library. A first processing subunit is configured to execute a target strategy corresponding to the current network topology to perform large-scale artificial intelligence model training according to the target strategy and obtain the adjusted network topology if the communication condition is not met. A second processing subunit is configured to perform large-scale artificial intelligence model training to obtain the adjusted network topology if the communication condition is met.

8. An electronic device, comprising: Comprise: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for constructing an optoelectronic hybrid data center network according to any one of claims 1-4.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method for constructing an optoelectronic hybrid data center network according to any one of claims 1-4.

10. A computer program, characterized in that, The computer program is executed to implement the method for constructing an optoelectronic hybrid data center network according to any one of claims 1-4.

Images