A method and apparatus for operating routing of requests
By adding routing units to the die and utilizing resource mapping relationships and address offsets, the problem of inaccurate operation request routing in multi-die cascade structures is solved, thereby improving chip performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI BIREN TECH CO LTD
- Filing Date
- 2026-04-10
- Publication Date
- 2026-07-21
AI Technical Summary
In AI chips with a multi-die cascaded structure, operation requests are difficult to route accurately to the correct die, leading to performance degradation.
By adding routing units to the die, the access address of the operation request is updated based on the preset resource mapping relationship and address offset to ensure accurate routing to the physical resources of the target die.
It improves the routing accuracy of operation requests, enhances the performance of artificial intelligence chips, and is simple to implement with low resource consumption.
Smart Images
Figure CN122431787A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a method and device for routing operation requests. Background Technology
[0002] The Central Processing Unit (CPU) uses a hypervisor to run multiple operating systems (OS) on a single physical device, providing simultaneous access for multiple users. When an AI chip is connected to the CPU via the PCIe interface, the AI chip must also be virtualized to ensure that each user perceives themselves as using a real AI chip peripheral.
[0003] In related technologies, when an AI chip needs to be used by N users simultaneously, the AI chip is virtualized into N virtual functions (VFs). Each VF corresponds to a physical resource on the AI chip, where N is greater than 1. In practical applications, the CPU sends operation requests to the AI chip via the PCIe interface. These requests carry a VFID indicating the virtual instance. The AI chip determines the corresponding virtual instance based on the VFID and routes the operation request to the physical resource corresponding to that virtual instance for processing.
[0004] In some scenarios, AI chips employ a multi-die cascaded architecture, meaning the AI chip comprises multiple dies with identical structures. Consequently, the physical resources corresponding to N virtual instances may reside on different dies. When the AI chip receives an operation request, it struggles to determine the correct die from among the multiple dies, potentially leading to routing errors and impacting chip performance. Summary of the Invention
[0005] This application provides a method and apparatus for routing operation requests, which improves routing accuracy and thus enhances chip performance when routing operation requests for an AI chip with a multi-die cascaded structure.
[0006] On one hand, embodiments of this application provide a routing method for operation requests, utilizing a first die in an artificial intelligence chip, wherein the first die is associated with a first die identifier, and the method includes: Receive a first operation request, the first operation request carrying a virtual instance identifier and a first access address, the first access address being within the full address range of the first bare die; Based on the preset resource mapping relationship, the second bare disk identifier corresponding to the virtual instance identifier is obtained; When the first die identifier is different from the second die identifier, obtain the address offset between the first die and the second die associated with the second die identifier; Based on the first access address and the address offset, the second access address is obtained; The first access address carried in the first operation request is updated to the second access address to obtain the second operation request, and the second operation request is routed to the second bare die, so that the second bare die routes the second operation request to the corresponding second physical resource in the second bare die based on the second access address and the virtual instance identifier.
[0007] On one hand, embodiments of this application provide a method for routing operation requests, utilizing a second die in an artificial intelligence chip, the method comprising: The system receives a second operation request sent by a first raw die, the second operation request carrying a virtual instance identifier and a second access address; the second operation request is obtained by the first raw die in the following manner: obtaining a first operation request carrying the virtual instance identifier and a first access address, and obtaining a second raw die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship; when the first raw die identifier is different from the second raw die identifier, obtaining the address offset between the first raw die and the second raw die associated with the second raw die identifier, and obtaining the second access address based on the first access address and the address offset; updating the first access address carried in the first operation request to the second access address, and obtaining the second operation request; Based on the virtual instance identifier and the second access address, the second operation request is routed to the corresponding second physical resource in the second die.
[0008] On one hand, embodiments of this application provide a routing device for operation requests, which utilizes a first die in an artificial intelligence chip, wherein the first die is associated with a first die identifier, and the device includes: The first receiving module is configured to receive a first operation request, the first operation request carrying a virtual instance identifier and a first access address, the first access address being located within the full address range of the first die; The discrimination module is used to obtain the second bare die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship; when the first bare die identifier is different from the second bare die identifier, obtain the address offset between the first bare die and the second bare die associated with the second bare die identifier; obtain the second access address based on the first access address and the address offset; update the first access address carried by the first operation request to the second access address to obtain the second operation request; The sending module is configured to route the second operation request to the second die, so that the second die routes the second operation request to the corresponding second physical resource in the second die based on the second access address and the virtual instance identifier.
[0009] Optionally, the discrimination module is specifically used for: When the first die identifier is different from the second die identifier, determine the identifier difference between the first die identifier and the second die identifier; The address offset is obtained based on the identification difference and the configuration space size of a single die.
[0010] Optionally, the sending module is specifically used for: The second operation request is routed to the die interconnect channel via the first bus of the first die, based on the second access address; The second operation request is routed to the second die via the die interconnect channel.
[0011] Optionally, the discrimination module is further configured to: When the first die identifier is the same as the second die identifier, the first operation request is routed to the corresponding first physical resource in the first die based on the first access address and the virtual instance identifier.
[0012] Optionally, the first physical resource includes: a first execution unit in the first die; The discrimination module is specifically used for: When the first die identifier is the same as the second die identifier, the first operation request is routed to the first computing unit corresponding to the first access address in the first die through the first bus of the first die; The first execution unit associated with the virtual instance identifier is determined from among the plurality of first execution units contained in the first computing unit, and the first operation request is routed to the first execution unit.
[0013] Optionally, the first die is the master die among the plurality of dies included in the artificial intelligence chip.
[0014] On one hand, embodiments of this application provide a routing device for operation requests, utilizing a second die in an artificial intelligence chip, the device comprising: The second receiving module is configured to receive a second operation request sent by the first die, the second operation request carrying a virtual instance identifier and a second access address; the second operation request is obtained by the first die in the following manner: obtaining a first operation request carrying the virtual instance identifier and a first access address, and obtaining a second die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship; when the first die identifier is different from the second die identifier, obtaining the address offset between the first die and the second die associated with the second die identifier, and obtaining the second access address based on the first access address and the address offset; updating the first access address carried in the first operation request to the second access address, thereby obtaining the second operation request; The processing module is configured to route the second operation request to the corresponding second physical resource in the second die based on the virtual instance identifier and the second access address.
[0015] Optionally, the second receiving module is specifically used for: The second operation request sent by the first die is received through the die interconnect channel.
[0016] Optionally, the second physical resource includes: a second execution unit in the second die; The processing module is specifically used for: The second operation request is routed to the second bus of the second die; The second operation request is routed to the second computing unit corresponding to the second access address in the second die via the second bus; The second execution unit associated with the virtual instance identifier is determined from among the plurality of second execution units contained in the second computing unit, and the second operation request is routed to the second execution unit.
[0017] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored on the memory and running on the artificial intelligence chip, wherein the artificial intelligence chip executes the computer program to implement the steps of the routing method for the above-mentioned operation request.
[0018] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the routing method for the aforementioned operation request.
[0019] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the routing method for the above-mentioned operation request.
[0020] In this embodiment, in a scenario where the AI chip has a multi-die cascaded structure, the first die receives a first operation request carrying a virtual instance identifier and a first access address. Based on the virtual instance identifier and a preset resource mapping relationship, it obtains the corresponding second die identifier. Then, based on the consistency between the first and second die identifiers, it determines the die that the first operation request actually needs to access, effectively distinguishing operation requests accessing different dies. When it is determined that the first operation request needs to access a second die other than the first die, the first access address carried by the first operation request is updated to a second access address associated with the second die, obtaining a second operation request. The second operation request is then successfully routed to the corresponding second physical resource in the second die, thereby ensuring the accuracy of cross-die routing and improving the performance of the AI chip. Furthermore, the solution of this application only requires adding a routing unit to the die, making it simple to implement and requiring less resources. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an artificial intelligence chip and CPU provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a bare die provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application; Figure 5 A flowchart illustrating an operation request routing method provided in an embodiment of this application; Figure 6 This application provides a schematic diagram of the structure of a GPU and a CPU according to an embodiment of the present application. Figure 7 A schematic diagram of the structure of an operation request routing device provided in an embodiment of this application; Figure 8 A schematic diagram of another operation request routing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of another computer device provided in an embodiment of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0024] It should be noted that in this application, "first" and "second" are used for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated.
[0025] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to an embodiment of this application. The artificial intelligence chip 100 includes at least a plurality of dies 101. This application does not specifically limit the number of dies included in the artificial intelligence chip 100. The plurality of dies 101 have the same structure, and any two dies 101 can communicate.
[0026] In some embodiments, see Figure 2 One of the multiple dies 101 can be designated as the master die, and the others as slave dies. The master die's PCIe interface is connected to the CPU 201. Thus, when the CPU 201 accesses the master die, it sends the operation request to the master die for processing via the PCIe interface. When the CPU 201 accesses any slave die, it first sends the operation request to the master die via the PCIe interface, and then the master die routes the operation request to the corresponding slave die for processing.
[0027] The PCIe interface uses SR-IOV technology (i.e., hardware virtualization technology), which allows a single physical PCIe device (such as a network card, an AI chip, or a storage controller) to be used by multiple virtual machines or applications to achieve efficient sharing.
[0028] In this embodiment, multiple dies 101 have the same structure. The following description uses one die 101 as an example. See [link to relevant documentation]. Figure 3 The bare chip 101 includes: a routing unit 301, a bus 302, a computing unit 303, and an execution unit 304.
[0029] The die 101 includes one or more computing units 303, each computing unit including multiple execution units 304; the bus 302 is connected to the routing unit 301 and the computing unit 303 respectively. The die 101 is connected to other dies through a die interconnect channel (D2D) 305.
[0030] In the embodiments of this application, the first die can be any die in the artificial intelligence chip 100, and the second die can be any die in the artificial intelligence chip 100 other than the first die. It should be noted that the artificial intelligence chip 100 may include other dies besides the first and second dies, and this application does not specifically limit this.
[0031] The following section uses the first and second raw dies as examples to illustrate the routing method for operation requests in this application. (See also...) Figure 4 This is a schematic diagram of a computer device applicable to an embodiment of this application, including: CPU 201 and artificial intelligence chip 100. The artificial intelligence chip 100 includes: a first die 401 and a second die 402. The PCIe interface of the first die 401 is connected to the CPU 201.
[0032] The first die 401 includes a first routing unit 4011, a first bus 4012, a first computing unit 4013, and a first execution unit 4014. The second die 402 includes a second routing unit 4021, a second bus 4022, a second computing unit 4023, and a second execution unit 4024. The first die 401 and the second die 402 are connected via a die interconnect channel 305.
[0033] The first die 401 receives a first operation request sent by the CPU 201. The first operation request carries a virtual instance identifier and a first access address, which is within the full address range of the first die 401. The first routing unit 4011 obtains the second die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship.
[0034] When the first die identifier is different from the second die identifier, the first routing unit 4011 obtains the address offset between the first die 401 and the second die 402 associated with the second die identifier; then, based on the first access address and the address offset, it obtains the second access address, updates the first access address carried by the first operation request to the second access address, obtains the second operation request; and then routes the second operation request to the first bus 4012.
[0035] The first bus 4012 routes the second operation request to the die interconnect channel 305 based on the second access address. The die interconnect channel 305 then routes the second operation request to the second bus 4022 of the second die. The second bus 4022 routes the second operation request to the second computing unit 4023 corresponding to the second access address. The second computing unit 4023 then routes the second operation request to the second execution unit 4024 corresponding to the virtual instance identifier.
[0036] When the first die identifier and the second die identifier are the same, the first routing unit 4011 routes the first operation request to the first bus 4012, and the first bus 4012 routes the first operation request to the first computing unit 4013 corresponding to the first access address. The first computing unit 4013 routes the first operation request to the first execution unit 4014 associated with the virtual instance identifier.
[0037] In this embodiment, in a scenario where the AI chip has a multi-die cascaded structure, the first die receives a first operation request carrying a virtual instance identifier and a first access address. Based on the virtual instance identifier and a preset resource mapping relationship, it obtains the corresponding second die identifier. Then, based on the consistency between the first and second die identifiers, it determines the die that the first operation request actually needs to access, effectively distinguishing operation requests accessing different dies. When it is determined that the first operation request needs to access a second die other than the first die, the first access address carried by the first operation request is updated to a second access address associated with the second die, obtaining a second operation request. The second operation request is then successfully routed to the corresponding second physical resource in the second die, thereby ensuring the accuracy of cross-die routing and improving the performance of the AI chip. Furthermore, the solution of this application only requires adding a routing unit to the die, making it simple to implement and requiring less resources.
[0038] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.
[0039] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-Purpose Graphics Processing Units (GPGPU), Domain Specific Architecture (DSA), etc.
[0040] The following is based on Figure 4 The diagram shown illustrates the architecture of an AI chip and details the flow of a routing method for operation requests. (See also...) Figure 5This method is executed by the interaction of a first die and a second die in an artificial intelligence chip. The first die is associated with a first die identifier, and the second die is associated with a second die identifier. The method includes the following steps: Step 501: The first die receives the first operation request.
[0041] The operation request routing method in this application can be applied to various scenarios, such as image processing, speech processing, and text processing. In different application scenarios, the physical meaning of the tensor processed by the first operation request can be different.
[0042] For example, in a text processing scenario, the tensor processed by the first operation request can be text data used in tasks such as text generation and text recognition.
[0043] For example, in a speech processing scenario, the tensor processed by the first operation request can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.
[0044] For example, in an image processing scenario, the tensor that the first operation requests to process can be image data used in tasks such as image preprocessing, image segmentation, and object detection.
[0045] In this embodiment of the application, the first die may be the master die among a plurality of dies included in the artificial intelligence chip, and the dies other than the first die among the plurality of dies are slave dies.
[0046] The PCIe interface of the first die is connected to the CPU, while the PCIe interfaces of the other dies do not need to be connected to the CPU. The CPU sends a first operation request to the first die through the PCIe interface. The first operation request may be to access the first die or to access another die. If the first operation request is to access another die, the first die will route the first operation request to that die.
[0047] The first operation request carries a virtual instance identifier and a first access address, which is located within the full address range of the first bare chip. For example, if the full address range of the first bare chip is 0 to (addr-1), then the first access address is located within this address range. Specifically, the first access address can be the address of a certain computing unit in the first bare chip.
[0048] The virtual instance identifier carried in the first operation request indicates the virtual instance accessed by the user. The physical resource corresponding to this virtual instance may be located on the first die or on the second die.
[0049] Step 502: The first bare die obtains the second bare die identifier corresponding to the virtual instance identifier based on the preset resource mapping relationship.
[0050] In practical applications, for the first operation request to access different raw dies, the first access address carried is the same, but the virtual instance identifier carried is different.
[0051] For example, the AI chip is configured to be allocated to four users, named U0, U1, U2, and U3. The AI chip is virtualized into four virtual instances, identified as VF0, VF1, VF2, and VF3.
[0052] The AI chip is configured to consist of two dies, labeled die0 and die1. die0 contains two execution units: die0_EU0 and die0_EU1; die1 contains two execution units: die1_EU0 and die1_EU1. die0_EU0 belongs to the physical resource of VF0, die0_EU1 belongs to the physical resource of VF1, die1_EU0 belongs to the physical resource of VF2, and die1_EU1 belongs to the physical resource of VF3.
[0053] When performing virtualization resource allocation, each user can use any one of the physical resources die0_EU0, die0_EU1, die1_EU0, and die1_EU1, as long as multiple users do not share the same physical resource. Users are unaware of which physical resource they are actually using; that is, the addresses presented to the four users are the same for die0_EU0, die0_EU1, die1_EU0, and die1_EU1. Therefore, when U0 accesses die0_EU0 via PCIe and U2 accesses die1_EU0 via PCIe and die0, the access address is the same; the difference lies in the virtual instance identifier.
[0054] Because the operation requests issued by U0 and U2 carry the same access address, but U0 actually needs to access EU0 in die0, while U2 actually needs to access EU0 in die1, relying solely on the bus in die0 is insufficient to make this distinction, resulting in a routing error for U2's operation request.
[0055] Based on this, in this embodiment, the first routing unit in the first die stores basic configuration information for the virtualization of the artificial intelligence chip through a register. This basic configuration information is pre-configured by the CPU. Specifically, the basic configuration information includes: the number of virtual instances virtualized in the artificial intelligence chip, and the resource mapping relationship between each virtual instance and the die, wherein the resource mapping relationship includes: multiple virtual instance identifiers and their corresponding die identifiers.
[0056] For example, an AI chip includes two dies, labeled die0 and die1. die0 includes two execution units, die0_EU0 and die0_EU1; die1 includes two execution units, die1_EU0 and die1_EU1.
[0057] If the AI chip only needs to be allocated to one user, then the AI chip is virtualized as a single virtual instance, identified as VF0. die0 and die1 are physical resources belonging to VF0. Accordingly, the resource mapping relationship configured by the CPU in the routing unit of die0 includes the association between VF0 and die0 and die1.
[0058] If the AI chip needs to be allocated to two users, it is virtualized into two virtual instances, identified as VF0 and VF1. In this case, die0_EU0 and die0_EU1 are physical resources belonging to VF0, and die1_EU0 and die1_EU1 are physical resources belonging to VF1. Accordingly, the resource mapping relationships configured by the CPU in the routing unit of die0 include the association between VF0 and die0_EU0 and die0_EU1, and the association between VF1 and die1_EU0 and die1_EU1.
[0059] If the AI chip needs to be allocated to four users, it is virtualized into four virtual instances, identified as VF0, VF1, VF2, and VF3. In this case, die0_EU0 is a physical resource belonging to VF0, die0_EU1 is a physical resource belonging to VF1, die1_EU0 is a physical resource belonging to VF2, and die1_EU1 is a physical resource belonging to VF3. Correspondingly, the resource mapping relationships configured by the CPU in the routing unit of die0 include: the association between VF0 and die0_EU0, the association between VF1 and die0_EU1, the association between VF2 and die1_EU0, and the association between VF3 and die1_EU1.
[0060] Of course, the AI chip can also be allocated to more than four users, which will not be elaborated here.
[0061] In this embodiment, after the first die receives the first operation request sent by the CPU, the first operation request first enters the first routing unit in the first die. The first routing unit queries a preset resource mapping relationship based on the virtual instance identifier carried by the first operation request to obtain the second die identifier corresponding to the virtual instance identifier.
[0062] For example, the resource mapping relationships stored in the routing unit of die0 can be configured to include: the association between VF0 and die0_EU0, the association between VF1 and die0_EU1, the association between VF2 and die1_EU0, and the association between VF3 and die1_EU1.
[0063] When the first die receives the operation request from U0, it queries the resource mapping relationship based on the virtual instance identifier VF0 carried in the operation request to obtain the die identifier die0.
[0064] When the first die receives the operation request from U2, it queries the resource mapping relationship based on the virtual instance identifier VF2 carried in the operation request to obtain the die identifier die1.
[0065] Step 503: When the first die identifier is different from the second die identifier, the first die obtains the address offset between the first die and the second die associated with the second die identifier.
[0066] Step 504: The first die obtains the second access address based on the first access address and the address offset.
[0067] Specifically, when the first die identifier and the second die identifier are different, it indicates that the operation request received by the first die is not to access the first die, but to access another die. Considering that the bus usually routes based on the access address, and the first access address carried by the first operation request is within the full address range of the first die (i.e., the physical address range of the first die), in this case, the first bus of the first die will only route the first operation request to the first computing unit corresponding to the first access address, and will not route it to other dies, thus causing a routing error.
[0068] Based on this, in this embodiment of the application, when the second bare die identifier corresponding to the virtual instance identifier is found and it is determined that the first bare die identifier and the second bare die identifier are different, the address offset between the first bare die and the second bare die associated with the second bare die identifier is first determined; then, according to the address offset, the first access address is converted into the second access address, wherein the second access address is located in the full address range of the second bare die. In this way, the first bus of the first bare die can route the operation request to the second bare die according to the second access address.
[0069] In some embodiments, when the first die identifier is different from the second die identifier, the identifier difference between the first die identifier and the second die identifier is determined; then, based on the identifier difference and the configuration space size of a single die, the address offset is obtained.
[0070] Specifically, the configuration space size of multiple dies in an artificial intelligence chip is the same. To avoid address duplication among multiple dies, during the chip design phase, each die can be assigned a corresponding full address range based on its own die identifier, and the full address ranges of different dies do not overlap.
[0071] For example, the configuration space size of a single die is addr; the full address range corresponding to die0 is 0 to (addr-1), and the full address range corresponding to die1 is (addr ~ addr + addr – 1).
[0072] The difference between the first and second die identifiers represents the number of die segments between them. This difference is then multiplied by the configuration space size of a single die segment to obtain the address offset. Next, the address offset is added to the first access address to obtain the second access address.
[0073] For example, let the first access address be x, which is located within the full address range of die0, i.e., the range from 0 to (addr-1). The difference between the identifiers of die0 and die1 is 1, therefore, the address offset between die0 and die1 is addr. Accordingly, the address offset is added to the first access address to obtain the second access address, i.e., x+addr; this second access address is located within the full address range of die1 (addr to addr+addr-1).
[0074] Step 505: The first die updates the first access address carried in the first operation request to the second access address, and obtains the second operation request.
[0075] Step 506: The first die routes the second operation request to the second die.
[0076] Specifically, the second operation request carries a virtual instance identifier and a second access address. Thus, the first die routes the second operation request to the die interconnect channel via the first bus, based on the second access address; then, the second operation request is routed to the second die via the die interconnect channel.
[0077] Step 507: The second die routes the second operation request to the corresponding second physical resource in the second die based on the second access address and the virtual instance identifier.
[0078] Specifically, after the second die receives the second operation request sent by the first die through the die interconnect channel, it routes the second operation request to the second bus of the second die, and then the second bus routes the second operation request to the corresponding second physical resource in the second die.
[0079] In some embodiments, the second die includes a plurality of second computing units, each second computing unit including a plurality of second execution units; the second physical resource includes one or more second execution units.
[0080] The second operation request is routed to the second computing unit corresponding to the second access address via the second bus; then, the second execution unit associated with the virtual instance identifier is determined from the multiple second execution units contained in the second computing unit, and the second operation request is routed to that second execution unit.
[0081] Specifically, the CPU can pre-configure internal routing information in the second computing unit. This internal routing information includes virtual instance identifiers associated with each second execution unit within the second computing unit. Thus, when the second bus routes a second operation request to the second computing unit, the second computing unit first obtains the virtual instance identifier carried in the second operation request, then queries the internal routing information based on this identifier to determine the corresponding second execution unit, and finally routes the second operation request to that second execution unit for processing.
[0082] In this embodiment, in a scenario where the AI chip has a multi-die cascaded structure, the first die receives a first operation request carrying a virtual instance identifier and a first access address. Based on the virtual instance identifier and a preset resource mapping relationship, it obtains the corresponding second die identifier. Then, based on the consistency between the first and second die identifiers, it determines the die that the first operation request actually needs to access, effectively distinguishing operation requests accessing different dies. When it is determined that the first operation request needs to access a second die other than the first die, the first access address carried by the first operation request is updated to a second access address associated with the second die, obtaining a second operation request. The second operation request is then successfully routed to the corresponding second physical resource in the second die, thereby ensuring the accuracy of cross-die routing and improving the performance of the AI chip. Furthermore, the solution of this application only requires adding a routing unit to the die, making it simple to implement and requiring less resources.
[0083] In some embodiments, when the first die identifier is the same as the second die identifier, the first operation request is routed to the corresponding first physical resource in the first die based on the first access address and the virtual instance identifier.
[0084] Specifically, when the first die identifier is the same as the second die identifier, it means that the first operation request received by the first die needs to access the first die. The first access address carried by the first operation request is within the full address range of the first die. Therefore, there is no need to perform address translation. The first operation request can be directly routed to the corresponding first physical resource in the first die through the first bus.
[0085] In some embodiments, the first die includes a plurality of first computing units, each first computing unit including a plurality of first execution units; the first physical resource includes one or more first execution units.
[0086] When the first die identifier is the same as the second die identifier, the first operation request is routed to the first computing unit corresponding to the first access address through the first bus; then, the first execution unit associated with the virtual instance identifier is determined from the multiple first execution units contained in the first computing unit, and the first operation request is routed to that first execution unit.
[0087] Specifically, the CPU can pre-configure internal routing information in the first computing unit. This internal routing information includes virtual instance identifiers associated with each first execution unit within the first computing unit. Thus, when the first bus routes a first operation request to the first computing unit, the first computing unit first obtains the virtual instance identifier carried in the first operation request, then queries the internal routing information based on this identifier to determine the corresponding first execution unit, and finally routes the first operation request to that first execution unit for processing.
[0088] For example, see Figure 6 The GPU is configured to be allocated to four users, named U0, U1, U2, and U3. The GPU is virtualized into four virtual instances using a hypervisor, identified as VF0, VF1, VF2, and VF3.
[0089] The GPU is configured to consist of two dies, labeled die0 and die1, which are connected via a die-to-die interconnect (D2D) channel.
[0090] die0 includes: a routing unit, a bus, and two computing units. The two computing units are IP0 and IP1, and each computing unit includes two execution units, EU0 and EU1.
[0091] die1 includes: a routing unit, a bus, and two computing units. The two computing units are IP0 and IP1, and each computing unit includes two execution units, EU0 and EU1.
[0092] EU0 in die0 belongs to the physical resource of VF0, EU1 in die0 belongs to the physical resource of VF1, EU0 in die1 belongs to the physical resource of VF2, and EU1 in die1 belongs to the physical resource of VF3. The routing unit of die0 stores the resource mapping relationship between the above virtual instance identifiers and the corresponding physical resources.
[0093] Assume that U0 uses EU0 in die0, U1 uses EU1 in die0, U2 uses EU0 in die1, and U3 uses EU1 in die1.
[0094] The routing unit of die0 receives an operation request sent by U0 through the CPU. This operation request carries the address of VF0 and IP0 in die0. The routing unit queries the resource mapping relationship based on VF0 and determines that the operation request needs to access die0, so it directly routes the operation request to the bus of die0. The bus of die0 routes the operation request to IP0 in die0 based on the address of IP0 in die0 carried in the operation request. The internal routing information in IP0 includes the association information between VF0 and EU0, and the association information between VF1 and EU1. Thus, based on VF0 and the internal routing information carried in the operation request, IP0 routes the operation request to EU0 in that IP0.
[0095] The routing unit of die0 receives an operation request sent by U1 via the CPU. This operation request carries VF1 and the address of IP0 in die0. The routing unit queries the resource mapping relationship based on VF1 and determines that the operation request needs to access die0, so it directly routes the operation request to the bus of die0. The bus of die0 routes the operation request to IP0 in die0 based on the address of IP0 in the operation request; IP0 then routes the operation request to EU1 in that IP0 based on VF1 and internal routing information carried in the operation request.
[0096] The routing unit of die0 receives an operation request sent by U2 via the CPU. This operation request carries VF2 and the address of IP0 in die0. Based on VF2, the routing unit queries the resource mapping relationship and determines that the operation request needs to access die1. Therefore, it adds addr (i.e., the configuration space size of a single die) to the address of IP0 in die0 to obtain the address of IP0 in die1, and updates the address carried in the operation request to the address of IP0 in die1.
[0097] The updated operation request is routed to the die0 bus. The die0 bus, based on the address of IP0 in die1 carried in the operation request, routes the operation request to the die interconnect channel. The die interconnect channel routes the operation request to the die1 bus. The die1 bus, based on the address of IP0 in die1 carried in the operation request, routes the operation request to IP0 in die1; the internal routing information in this IP0 includes the association information between VF2 and EU0, and the association information between VF3 and EU1. Thus, based on VF2 and the internal routing information carried in the operation request, this IP0 routes the operation request to EU0 in this IP0.
[0098] The routing unit of die0 receives an operation request sent by U3 via the CPU. This operation request carries VF3 and the address of IP0 in die0. Based on VF3, the routing unit queries the resource mapping relationship and determines that the operation request needs to access die1. Therefore, it increments addr by the address of IP0 in die0 to obtain the address of IP0 in die1, and updates the address carried in the operation request to the address of IP0 in die1.
[0099] The updated operation request is routed to the die0 bus. The die0 bus, based on the address of IP0 in die1 carried in the operation request, routes the operation request to the die interconnect channel. The die interconnect channel routes the operation request to the die1 bus. The die1 bus, based on the address of IP0 in die1 carried in the operation request, routes the operation request to IP0 in die1; this IP0, based on the VF3 and internal routing information carried in the operation request, routes the operation request to EU1 within that IP0.
[0100] In this embodiment, firstly, in a scenario where the AI chip has a multi-die cascaded structure, the first die receives a first operation request carrying a virtual instance identifier and a first access address. Based on the virtual instance identifier and a preset resource mapping relationship, it obtains the corresponding second die identifier. Then, based on the consistency between the first and second die identifiers, it determines the die that the first operation request actually needs to access, thus effectively distinguishing operation requests accessing different dies. When it is determined that the first operation request needs to access a second die other than the first die, the first access address carried by the first operation request is updated to a second access address associated with the second die, obtaining a second operation request. Then, the second operation request is successfully routed to the corresponding second physical resource in the second die, thereby ensuring the accuracy of cross-die routing and improving the performance of the AI chip.
[0101] Secondly, when it is determined that the first operation request needs to access the first die, there is no need to perform address translation. The first operation request is directly routed to the corresponding first physical resource in the first die through the first bus, thereby ensuring the accuracy of the internal routing of the die.
[0102] In addition, the solution in this application only requires adding a routing unit to the bare die to achieve accurate routing of operation requests in a multi-die cascaded structure, which is simple to implement and consumes less resources.
[0103] Based on the same technical concept, this application provides a schematic diagram of the structure of an operation request routing device, which utilizes a first die in an artificial intelligence chip, the first die being associated with a first die identifier, such as... Figure 7 As shown, the routing device 700 for the operation request includes: The first receiving module 701 is used to receive a first operation request, wherein the first operation request carries a virtual instance identifier and a first access address; The discrimination module 702 is used to obtain the second bare die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship; when the first bare die identifier is different from the second bare die identifier, obtain the address offset between the first bare die and the second bare die associated with the second bare die identifier; obtain the second access address based on the first access address and the address offset; update the first access address carried by the first operation request to the second access address to obtain the second operation request; The sending module 703 is configured to route the second operation request to the second die, so that the second die routes the second operation request to the corresponding second physical resource in the second die based on the second access address and the virtual instance identifier.
[0104] Optionally, the discrimination module 702 is specifically used for: When the first die identifier is different from the second die identifier, determine the identifier difference between the first die identifier and the second die identifier; The address offset is obtained based on the identification difference and the configuration space size of a single die.
[0105] Optionally, the sending module 703 is specifically used for: The second operation request is routed to the die interconnect channel via the first bus of the first die, based on the second access address; The second operation request is routed to the second die via the die interconnect channel.
[0106] Optionally, the discrimination module 702 is further configured to: When the first die identifier is the same as the second die identifier, the first operation request is routed to the corresponding first physical resource in the first die based on the first access address and the virtual instance identifier.
[0107] Optionally, the first physical resource includes: a first execution unit in the first die; The discrimination module 702 is specifically used for: When the first die identifier is the same as the second die identifier, the first operation request is routed to the first computing unit corresponding to the first access address in the first die through the first bus of the first die; The first execution unit associated with the virtual instance identifier is determined from among the plurality of first execution units contained in the first computing unit, and the first operation request is routed to the first execution unit.
[0108] Optionally, the first die is the master die among the plurality of dies included in the artificial intelligence chip.
[0109] Based on the same technical concept, this application provides a schematic diagram of the structure of an operation request routing device, which utilizes a second die in an artificial intelligence chip, such as... Figure 8 As shown, the routing device 700 for the operation request includes: The second receiving module 801 is configured to receive a second operation request sent by the first die, the second operation request carrying a virtual instance identifier and a second access address; the second operation request is obtained by the first die in the following manner: obtaining a first operation request carrying the virtual instance identifier and a first access address, and obtaining a second die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship; when the first die identifier is different from the second die identifier, obtaining the address offset between the first die and the second die associated with the second die identifier, and obtaining the second access address based on the first access address and the address offset; updating the first access address carried in the first operation request to the second access address, thereby obtaining the second operation request; The processing module 802 is used to route the second operation request to the corresponding second physical resource in the second die based on the virtual instance identifier and the second access address.
[0110] Optionally, the second receiving module 801 is specifically used for: The second operation request sent by the first die is received through the die interconnect channel.
[0111] Optionally, the second physical resource includes: a second execution unit in the second die; The processing module 802 is specifically used for: The second operation request is routed to the second bus of the second die; The second operation request is routed to the second computing unit corresponding to the second access address in the second die via the second bus; The second execution unit associated with the virtual instance identifier is determined from among the plurality of second execution units contained in the second computing unit, and the second operation request is routed to the second execution unit.
[0112] In this embodiment, in a scenario where the AI chip has a multi-die cascaded structure, the first die receives a first operation request carrying a virtual instance identifier and a first access address. Based on the virtual instance identifier and a preset resource mapping relationship, it obtains the corresponding second die identifier. Then, based on the consistency between the first and second die identifiers, it determines the die that the first operation request actually needs to access, effectively distinguishing operation requests accessing different dies. When it is determined that the first operation request needs to access a second die other than the first die, the first access address carried by the first operation request is updated to a second access address associated with the second die, obtaining a second operation request. The second operation request is then successfully routed to the corresponding second physical resource in the second die, thereby ensuring the accuracy of cross-die routing and improving the performance of the AI chip. Furthermore, the solution of this application only requires adding a routing unit to the die, making it simple to implement and requiring less resources.
[0113] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0114] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 9 As shown, it includes at least one artificial intelligence chip 100 and a memory 901 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 901 is not limited. Figure 9 Taking the connection between the AI chip 100 and the memory 901 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.
[0115] In this embodiment, the memory 901 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 901, the at least one artificial intelligence chip 100 can perform the steps of the routing method for the above-mentioned operation request.
[0116] The artificial intelligence chip 100 serves as the control center of the computer device. It connects to various parts of the computer device via various interfaces and lines, and routes operation requests by running or executing instructions stored in the memory 901 and accessing data stored in the memory 901. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 901 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.
[0117] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0118] Memory 901, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 901 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 901 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 901 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0119] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the routing method for the aforementioned operation request.
[0120] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the routing method for the above-mentioned operation request.
[0121] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0122] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0124] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0125] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0126] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for routing operation requests, characterized in that, The method includes: using a first die in an artificial intelligence chip, wherein the first die is associated with a first die identifier; Receive a first operation request, the first operation request carrying a virtual instance identifier and a first access address, the first access address being within the full address range of the first bare die; Based on the preset resource mapping relationship, the second bare disk identifier corresponding to the virtual instance identifier is obtained; When the first die identifier is different from the second die identifier, obtain the address offset between the first die and the second die associated with the second die identifier; Based on the first access address and the address offset, the second access address is obtained; The first access address carried in the first operation request is updated to the second access address to obtain the second operation request, and the second operation request is routed to the second bare die, so that the second bare die routes the second operation request to the corresponding second physical resource in the second bare die based on the second access address and the virtual instance identifier.
2. The method as described in claim 1, characterized in that, When the first die identifier and the second die identifier are different, obtaining the address offset between the first die and the second die associated with the second die identifier includes: When the first die identifier is different from the second die identifier, determine the identifier difference between the first die identifier and the second die identifier; The address offset is obtained based on the identification difference and the configuration space size of a single die.
3. The method as described in claim 1, characterized in that, The step of routing the second operation request to the second die includes: The second operation request is routed to the die interconnect channel via the first bus of the first die, based on the second access address; The second operation request is routed to the second die via the die interconnect channel.
4. The method as described in claim 1, characterized in that, Also includes: When the first die identifier is the same as the second die identifier, the first operation request is routed to the corresponding first physical resource in the first die based on the first access address and the virtual instance identifier.
5. The method as described in claim 4, characterized in that, The first physical resource includes: the first execution unit in the first die; When the first die identifier is the same as the second die identifier, the first operation request is routed to the corresponding first physical resource in the first die based on the first access address and the virtual instance identifier, including: When the first die identifier is the same as the second die identifier, the first operation request is routed to the first computing unit corresponding to the first access address in the first die through the first bus of the first die; The first execution unit associated with the virtual instance identifier is determined from among the plurality of first execution units contained in the first computing unit, and the first operation request is routed to the first execution unit.
6. The method according to any one of claims 1 to 5, characterized in that, The first die is the master die among the multiple dies included in the artificial intelligence chip.
7. A method for routing operation requests, characterized in that, The method of applying a second die in an artificial intelligence chip includes: The system receives a second operation request sent by a first raw die, the second operation request carrying a virtual instance identifier and a second access address; the second operation request is obtained by the first raw die in the following manner: obtaining a first operation request carrying the virtual instance identifier and a first access address, and obtaining a second raw die identifier corresponding to the virtual instance identifier based on a preset resource mapping relationship; when the first raw die identifier is different from the second raw die identifier, obtaining the address offset between the first raw die and the second raw die associated with the second raw die identifier, and obtaining the second access address based on the first access address and the address offset; updating the first access address carried in the first operation request to the second access address, and obtaining the second operation request; Based on the virtual instance identifier and the second access address, the second operation request is routed to the corresponding second physical resource in the second die.
8. The method as described in claim 7, characterized in that, The receiving of the second operation request sent by the first die includes: The second operation request sent by the first die is received through the die interconnect channel.
9. The method as described in claim 8, characterized in that, The second physical resource includes: the second execution unit in the second die; The step of routing the second operation request to the corresponding second physical resource in the second die based on the virtual instance identifier and the second access address includes: The second operation request is routed to the second bus of the second die; The second operation request is routed to the second computing unit corresponding to the second access address in the second die via the second bus; The second execution unit associated with the virtual instance identifier is determined from among the plurality of second execution units contained in the second computing unit, and the second operation request is routed to the second execution unit.
10. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored on the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1 to 9.