Computing device connection and computing device mapping method based on bus page table

Abstract

A bus connection in which bus elements (3000) are operably connected to each other in a computing device according to the present invention may comprise: a bus master (3100) in charge of controlling the bus (3010) by a predetermined implementation or communication method; a bus slave (3200) controlled by the bus (3010) by means of the predetermined implementation or communication method; and an internal bus (3010) that physically implements between the bus elements. In addition, the present invention comprises an internal physical bus address unit (3320) that divides slaves (3200) of the internal bus (3010) in a physical form in the bus master (3100) that physically controls the internal bus (3010), and another arbitrary virtual bus address unit (3310), and may comprise a bus address page table (3300) which can be recorded or set (5000) so as to enable mapping or address conversion (4000) and helps the address conversion (4000) operation of the bus master (3100). In addition, the predetermined internal bus (3010) implementation and communication method may be a method based on a bus physical protocol such as AXI / AHB or UPI / QPI. The bus master (3100) and the bus address page table (3300) according to the present invention may comprise a chiplet connection (3020) that connect chips which are other computing devices as well as an operation in a chip (200) which is the computing device; and a chiplet connection (3020), and the chiplet connection (3020) is connected between the bus masters (3100) and may include an address conversion (4000), or a mapping and setting method (5000) by being organically coupled to the bus address page table (3300) even between the plurality of computing devices through a bus address conversion (4000) of the bus address page table (3300) or mapping operation. In addition, the predetermined chiplet connection (3020) communication method may be a method based on a chiplet connection physical protocol such as UCIe. In addition, the predetermined chiplet connection (3020) communication method may be a method based on a conventional bus physical protocol such as AXI / AHB or a non-standard bus physical protocol such as UPI / QPI. A method for implementing the predetermined chiplet connection (3020) may be a method of implementing a serial bus structure, a parallel bus structure, and a tree bus structure that applies same in various ways, and may be a method including various implementation methods such as a typical ring bus structure. In a method for implementing the bus master (3100) in the present invention, an interconnect link unit (400) is included; and the present invention can have an organic operation with a bus address page table (3300) through a predetermined interconnect link (400) implementation or a communication method without directly passing through the bus (3010) or the chiplet connection (3020) structure, and may be a method for having limited mapping coherence between computing devices and organically combining the virtual bus address unit (3310) inside the bus address page table (3300) partially through the interconnect link (400) through an operation of remote bus interception (7000) between computing devices. In addition, the communication method of the predetermined interconnect may be a method based on a PCle / CXL physical protocol.

Description

Bus page table-based computing device connection and computing device mapping method

[0001] The present invention relates to a computing device connection and a computing device mapping method based on a bus page table, and to a computing device connection and a computing device mapping method that can organically operate processors, devices, computers, etc. inside / outside a chip by utilizing an open interconnect such as CXL and an open chiplet connection bus page table such as UCIe.

[0002] In hardware, which is a computing device, the concept of a bus address is something that has not been easily visible because it is overshadowed by the concept of a conventional memory virtual / physical page address, and is an address concept that those skilled in the art encounter only when designing hardware, such as a computing device, such as a processor.

[0003] A bus address, typically a memory physical page address, is an address that the memory controller sets as a physical bus address on the bus and that is accessed by a computing device such as a processor. This bus address concept can be said to be the most fundamental physical root address, extended to the memory paging address concept, or memory virtual address concept, by computing devices such as processors and memory, and software such as operating systems running on them.

[0004] The concept of the above-mentioned fixed and unchangeable physical bus address in hardware ultimately relied on the concept of the virtual paging address in the memory for connection and mapping between computing devices, and this was also the reason for problems such as NUMA / SMP between two computing devices, i.e. processors, in the past. In addition, the above-mentioned problems between the host and the device have been very long-standing, and the problem of having to rely on unreliable, old, and inefficient hardware, such as DMA / IOMMU (DTLB) designed for it, and the hardware interrupt problem, such as APIC derived from it, have also continued without being easily replaced.

[0005] The present invention is intended to solve the above-described problems, and borrows the memory paging concept used in conventional operating systems, and presents an alternative to the above-described problems by remapping the concept of virtual bus address to the internal physical bus address through the page table and bus paging concept for the bus address.

[0006] The bus page table is a method devised in the present invention, which is derived from the conventional operating system's practice of allocating and accessing memory in paging units, and enables the paging address system to be implemented not only in memory addresses but also in physical internal bus address values, thereby enabling mapping and address translation of virtual bus addresses to conventional physical bus addresses.

[0007] In addition, it does not stop at simply mapping and translating the address of an internal physical bus address that cannot be physically changed into a paging unit, but alleviates the above concepts and problems common in conventional unreliable hardware through interaction between the bus page table and a reliable operating system.

[0008] However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

[0009] In a bus connection that operably connects bus elements (3000) within a computing device according to the present invention, the computing device may include a bus master (3100) that controls the bus (3010) by a predetermined implementation or communication method; a bus slave (3200) that receives control of the bus (3010) by the predetermined implementation or communication method; and an internal bus (3010) that physically implements between such bus elements.

[0010] In addition, the bus master (3100) that physically controls the internal bus (3010) includes an internal physical bus address section (3320) that distinguishes slaves (3200) of the internal bus (3010) in a physical form, and may include another arbitrary virtual bus address section (3310); and may be recorded and set (5000); to enable mapping or address conversion (4000); and may include a bus address page table (3300) that assists the address conversion (4000) operation of the bus master (3100).

[0011] In addition, the above-described internal bus (3010) implementation and communication method may be a method based on a bus physical protocol such as AXI / AHB or UPI / QPI.

[0012]

[0013] In the bus master (3100) and bus address table (3300) according to the present invention, the operation is not limited to the operation within the chip (200) which is the computing device, but may include a chiplet connection (3020) which connects between chips which are between other computing devices; and the chiplet connection (3020) is connected between bus masters (3100), and through the bus address conversion (4000) of the bus table (3300); or the mapping operation, it may be organically combined with the bus table (3300) between a plurality of computing devices to include an address conversion (4000), or a mapping, setting method (5000);

[0014] Additionally, the above-described chiplet connection (3020) communication method may be a method based on a chiplet connection physical protocol such as UCIe.

[0015] In addition, the above-described chiplet connection (3020) communication method may be a method based on a conventional bus physical protocol such as AXI / AHB, or a non-standard bus physical protocol such as UPI / QPI.

[0016] In addition, the above-described chiplet connection (3020) implementation method may be an implementation method of a serial bus structure; a parallel bus structure; a tree bus structure that is variously applied thereto; and may include various implementation methods such as a typical ring bus structure.

[0017]

[0018] In the method for implementing a bus master (3100) in the present invention, the interconnect link unit (400) may be included, and may have organic operation with the bus page table (3300) through a predetermined interconnect link (400) implementation or communication method, without directly passing through the bus (3010) or chiplet connection (3020) structure, and may be a method for organically combining the virtual bus address unit (3310) inside the bus page table (3300) between computing devices through an operation of mapping (7000) between computing devices, although limited, through the interconnect link (400).

[0019] Additionally, the communication method of the above-determined interconnect may be a method based on the PCIe / CXL physical protocol.

[0020] By utilizing the computing device and computing device design methods based on the bus page table according to the present invention, it is possible to achieve innovation not only within a single computing device, i.e., a chip, but also between multiple computing devices while taking advantage of the conventional memory paging structure, and alleviate problems derived from the memory paging structure through the bus page table structure.

[0021] Additionally, in addition to the SMP / NUMA issues between conventional computing devices, it helps to alleviate the conventional DMA / IOMMU (DTLB) issues through organic coupling with the interconnect by having a bus page table structure between two computing devices, and helps to transfer all hardware interrupts such as APIC derived from it back to reliable software.

[0022] In conclusion, this open bus / chiplet connection structure and bus page table enable us to enjoy reliable computing methods both inside and outside the chip on untrusted computers.

[0023]

[0024] However, the effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.

[0025] FIG. 1 is a drawing illustrating a portion of the present invention from the perspective of a chip internal bus section (3000) and an application related to the bus page table (3300) between the chip internal bus (3010) and the chip external chiplet connection (3020).

[0026] FIG. 2 is a drawing showing the application of the bus page table (3300) technology shown in FIG. 1, the application in the chiplet connection (3020) during bus address paging, and the crossbar-type structure.

[0027] The example and application of Fig. 2a are examples of a method for alleviating the problem of allowing only symmetrical processors, which are symmetrical computing devices (200) in an SMP (Shared Memory Processing / Symmetric Multi Processing) structure, and the NUMA (Non Unified Memory Access) problem through a bus page table (3300) structure.

[0028] FIG. 2b is a diagram showing the relationship with the bus page table (3300) from the perspective of a TLB (260) / cache (240) that has the task of converting a conventional virtual paging address into a physical paging address.

[0029] FIG. 2c is a drawing showing an example of a case in which the bus address table (3300) shown in FIG. 2a and FIG. 2b is not present.

[0030] FIG. 3 is an example diagram showing the operation of the chip internal bus (3010), chiplet connection (3020), and TLB (260) inside the cache (240) through the open interconnect link (400) presented in FIGS. 1 and 2.

[0031] Figure 3a is a diagram schematically illustrating the operation of the bus page table (3300) in Figure 3 with the TLB (260) / cache (240) and bus address conversion (4000) within the chip (200).

[0032] FIG. 3b is a diagram schematically illustrating the initialization process and the bus address paging (3300) setting procedure when the chip (200) mentioned in FIGS. 1 and 2 is internally powered on, and the method of sharing a secure bus address table (3300) with the kernel (500) after nonlinear protection mode / Protection SWAP (5000).

[0033] FIG. 3c is a diagram showing various calculations derived from a virtual bus address (3310) in the bus paging table (3300) of FIG. 3, FIG. 3a, and FIG. 3b, and Protection SWAP (5000), which is a table sharing / protection technique with an operating system (500) using the bus page table (3300) of FIG. 3b.

[0034] FIG. 4 is a drawing showing a part of a hardware interrupt replacement such as a conventional APIC, which is one of the advantages obtained when having a bus page table (3300) as proposed by the present invention.

[0035] FIG. 4a is a diagram illustrating a method for mitigating a bus intercept (6000) that replaces the above hardware interrupt element so that it cannot be exploited by untrusted hardware or conventional malicious software (900).

[0036] FIG. 4b is an example of a method for enabling interconnection (6700) between the areas that cause (6200) the bus intercept (6000) and the bus intercept through cache (240) to cache (240) access between processors (200) in a bus (3010) / chiplet connection (3020) shared structure, such as a bus intercept (6000) that replaces an interrupt.

[0037] Figure 5 is a drawing showing the direct bus access part that replaces the conventional DMA / IOMMU (DTLB), etc. in the bus page table (3300) structure.

[0038] FIG. 5a is a drawing that more clearly shows Direct Bus Access, which replaces DMA / IOMMU (DTLB) shown in FIG. 5 above, from the internal perspective of the bus master controller (3110) and the bus page table controller (3120).

[0039] FIG. 6 is an example of a part that can be applied to a local switch (410) or a network such as the Internet through application with a conventional NTB when a bus paging table (3300) is applied to an interconnect link (400) such as CXL, as in the above FIG. 5.

[0040] 10: Bus / chiplet connection connector / socket

[0041] 11: Bus / Chiplet Connection Conductor

[0042] 12: Bus / chiplet connection optical fiber section

[0043] 13: Auxiliary Conductor Unit (Power / Spare)

[0044]

[0045] 100: Bus / Chiplet Connection Imagination Area

[0046] 110: Cache thought area

[0047] 120: Memory (buffer) mapping area

[0048] 130: Register mapping area

[0049] 140: Interconnect link thought domain

[0050] 170: Variables mapped / allocated via bus / chiplet connection

[0051] 171: Variable X mapped / allocated via bus / chiplet connection

[0052] 172: Variable Y mapped / allocated via bus / chiplet connection

[0053] 173: Variable Z mapped / allocated via bus / chiplet connection

[0054]

[0055] 200: Computing device / processor

[0056] 201: First computing device

[0057] 202: Second computing device

[0058] 203: Third computing device

[0059] 210: Processor core

[0060] 220: Processor Memory Controller

[0061] 230: Processor physical wiring

[0062] 240: Processor cache

[0063] 241: L1 cache

[0064] 242: L2 cache

[0065] 243: L3 cache

[0066] 244: L4 cache

[0067] 250: Terminal Cache / LLC (Last Level Cache)

[0068] 260: Address Translation Preview Block / TLB (Translate Lookup Block)

[0069]

[0070] 300: Interconnect Device

[0071] 310: Device Physical Driver

[0072] 311: Physical LVDS

[0073] 312: Physical ADC / DAC

[0074] 314: Physical GPIO

[0075] 320: Modem device

[0076]

[0077] 400: Interconnect link section

[0078] 401: Interconnect first connector

[0079] 402: Interconnect second connector

[0080] 410: Interconnect Switch

[0081] 411: Interconnect 1st NTB Section

[0082] 412: Interconnect 2nd NTB Section

[0083] 413: Interconnect 3rd NTB Division

[0084] 420: Interconnect Root Complex

[0085] 430: Interconnect endpoint

[0086] 440: Interconnect Upstream

[0087] 450: Interconnect downstream

[0088] 460: CXL.io

[0089] 470: CXL.memory

[0090] 480: CXL.cache

[0091]

[0092] 500: Kernel section

[0093] 501: First operating system

[0094] 502: Second operating system

[0095] 510: Interconnect kernel module (system driver)

[0096]

[0097] 600: Department of Infrastructure

[0098] 610: CXL Framework

[0099] 611: PCIE Framework

[0100] 612: USB Framework

[0101] 620: Unix socket 630: CGI socket

[0102] 640: Graphics Framework 650: Audio Framework

[0103] 660: Printer Framework 670: Accelerator Framework

[0104] 680: Cluster Orchestration Framework

[0105]

[0106] 700: Interconnect Virtual Device

[0107]

[0108] 900: User Application Department

[0109] 901: First Application

[0110] 902: Second Application

[0111] 920: User Driver

[0112]

[0113] 3000: Chip bus section

[0114] 3010: On-chip switch bus / fabric matrix

[0115] 3020: Chiplet Connection

[0116]

[0117] 3100: Bus Master

[0118] 3101: First Bus Master

[0119] 3102: Second Bus Master

[0120] 3110: Bus Master Controller

[0121] 3120: Bus Address Table Controller

[0122] 3130: Bus Synchronous / Asynchronous Allocator

[0123] 3140: Bus Data Mover

[0124] 3150: Bus data transfer channel section

[0125] 3151: Bus data transfer channel 1

[0126] 3152: Bus Data Mover Channel 2

[0127] 3160: TLB / Cache Status Checker

[0128]

[0129] 3200: Bus Slave

[0130] 3210: Bus read / write section

[0131] 3220: Bus Stream Department

[0132] 3230: Busstream input / output section

[0133] 3231: Bus stream input

[0134] 3232: Bus stream output

[0135]

[0136] 3300: Bus Address Paging Table

[0137] 3301: First bus address table 3302: Second bus address table

[0138] 3310: Virtual / global bus address 3320: Internal physical bus address

[0139] 3330: Bus address paging entry

[0140] 3340: Bus address paging entry setup bit (SET) / spare (ETC, precision setting)

[0141] 3500: Bus table entry line / count calculation formula

[0142] 3600: Bus table calculation formula

[0143] 3700: Calculation of the number of page entries required to hide the bus table in the operating system virtual memory page entry and the number of bootstrap initial random numbers N.

[0144]

[0145] 4000: Chip internal bus address paging and operation method using bus page table

[0146] 5000: Protection Swap / Setup Method through Bus Page Table Virtual Address Change

[0147]

[0148] 6000: Bus Interception Method Using Bus Paging Table

[0149] 6010: Local bus intercept inbound entire memory area

[0150] 6020: Global bus intercept inbound full memory area

[0151] 6030: Global bus intercept outbound entire memory area

[0152] 6100: Cache / LLC residency method and assembly / bytecode for bus interception

[0153] 6200: Cache / LLC resident area trigger / trigger for causing bus intercepts

[0154]

[0155] 6500: Operating system APIC / interrupt handler using bus intercept

[0156] 6510: OneTime Vector for Bus Intercept Operating System Handler Entry

[0157] 6600: Bus Intercept Operating System Handler Protection Method and Graph

[0158] 6650: Bus Intercept Operating System Handler Protection Graph Vertex

[0159] 6700: Involved Region and Protection Method Using Bus Interception

[0160]

[0161] 7000: Remote bus interception method between computing devices using interconnect protocol and bus interception

[0162]

[0163] Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention will be able to easily propose other inventions that are retrograde or other embodiments included within the scope of the spirit of the present invention by adding, modifying, or deleting other components within the scope of the same spirit. However, this will also be considered to be included within the scope of the spirit of the present invention.

[0164]

[0165] A means for connecting internal elements of a computing device according to the present invention may include an internal physical bus unit (3010) that is communicatively connected under the control of the bus master (3100) by a predetermined implementation or communication method in a bus unit (3000) that communicatively connects a bus master (3100) and a bus slave (3200) in a computing device (200); and a bus slave (3200) that is controlled and driven by receiving control and signals of the internal physical bus unit (3010) by a predetermined implementation or communication method.

[0166] In addition, the above-described bus (3010) implementation and communication method may be a method based on an open standard physical protocol such as AXI / AHB.

[0167] Additionally, the above-described implementation and communication method may be a method based on a non-standard physical protocol such as QPI / UPI.

[0168] In addition, the above-described bus (3010) implementation and communication method may be a method including a bus read / write protocol that recognizes, sets, enumerates, manipulates, reads, writes, and implements error processing in the bus slave (3200) based on the internal physical bus address (3320) of at least one bus slave (3200) of the read / write signal of the bus master (3100).

[0169] In addition, the above-described bus (3010) implementation and communication method may be a method including a bus stream protocol that recognizes, sets, enumerates, manipulates, reads, writes, and handles errors in the bus slave (3200) without an internal physical bus address (3320) by transmitting the read / write signal of the bus master (3100).

[0170] However, without limitation thereto, the above-described communication method can be modified in various ways at a level obvious to a person skilled in the art.

[0171]

[0172] In the bus master (3100) according to the present invention, the bus master (3100) may include an internal physical bus address section (3320) of the internal bus (3010), which is an address that physically controls the internal bus (3010); an arbitrary virtual bus address section (3310) set (5000) / allocated by the operating system (500); and a bus address page table (3300) that helps to record and set (5000); so that mapping or address conversion (4000) can be performed.

[0173] The implementation of the bus page table (3300) located within the above bus master (3100) is similar to the conventional cache (240) in that it records and stores the internal physical bus address section (3320) and the virtual bus address (3310) in the form of a register such as SRAM, and can perform operations such as retrieval and recording for address conversion (4000), or it can physically have the form of an eDRAM, DRAM, GDDR, HBM, or another form of storage recording medium.

[0174] In addition, the operation of the bus table (3300) within the bus master (3100) may have a structure similar to that of the cache (240) in the example of this patent, but may not have a structure similar to that of the cache (240), and may have a completely different page precision and bus page entry (3330) structure, and the method of traversing and querying it may not be the same as that of the example of this patent, and may have some completely different cache traversal, query, or bus address compare logic, or some lookup elements such as LRU.

[0175] However, without limitation thereto, the above-described predetermined bus page table and the method of applying and implementing the bus page table can be modified in various ways at a level obvious to a person skilled in the art.

[0176]

[0177] The bus master (3100) and bus address table (3300) according to the present invention not only operate within the chip (200), which is the computing device, but may also include a chiplet connection (3020) connecting between chips (201, 202), which are other computing devices.

[0178] The above chiplet connection (3020) may be connected between bus masters (3100), and may include a bus address translation (4000) of a bus table (3300); or a mapping operation; and may be organically connected to a bus table (3300) between a plurality of computing devices to include an address translation (4000), or a mapping, setting method (5000);

[0179] In addition, the above-described chiplet connection (3020) implementation and communication method may be a method based on an open chiplet connection physical protocol such as UCIe that allows a bus master (3100) to connect and communicate with another bus master (3100) of an external computing device.

[0180] In addition, the above-described chiplet connection (3020) implementation and communication method may be based on a bus communication standard such as AXI / AHB, which is a conventional bus physical protocol, or a non-standard bus communication physical protocol such as UPI / QPI.

[0181] However, without limitation thereto, the implementation and communication method of the above-described chiplet connection (3020) can be modified in various ways at a level obvious to a person skilled in the art.

[0182]

[0183] In the present invention, the bus master (3100) may have an organic operation with the bus page table (3300) through an interconnect link section (400); including a predetermined interconnect link (400) implementation or communication method, without directly passing through the bus (3010) or chiplet connection (3020) structure, and may have a mapping coherence between computing devices, albeit limited, and may be a method of organically combining the virtual bus address section (3310) within the bus page table (3300) between computing devices through an operation of mapping (7000); partially through the interconnect link (400).

[0184] Additionally, the communication method of the above-determined interconnect may be a method based on the PCIe / CXL physical protocol.

[0185] In addition, the above-described interconnect communication, implementation method may be a method in which the first computing device (201) operates as an interconnect endpoint (430) as a device (300), the first connection unit (401) operates as an interconnect downstream (450) that mediates it, the second computing device (202) operates as an interconnect root complex (420) that allows it to access the first computing device (201) as a host, and the second connection unit (402) operates as an interconnect upstream (440) that mediates it.

[0186] The above first computing device (201) and second computing device (202) can coexist in both the root complex (420) mode as a host and the endpoint (430) mode as a device. When operating (7000) by interconnection and bus page table, one of the two must be determined to play the role of the root complex (420) or the other must be determined to play the role of the endpoint (430).

[0187] Additionally, the first computing device (201) and the second computing device (202) may have an interconnect switch (410) or at least one interconnect upstream (440) and interconnect downstream (450), or may not be directly connected.

[0188] Additionally, it may be directly connected to a root complex (420) through a conventional NTB and interconnect switch (410).

[0189] However, without limitation thereto, the above-described interconnect link (400) implementation and communication method can be modified in various ways at a level obvious to a person skilled in the art.

[0190] In order to connect and map the first computing device (201) and the second computing device (202), for example, the area of ​​the bus page table (3300) within the first computing device (201) must be disclosed by the manufacturer of the first computing device (201) and recognized / set through a protocol such as CXL.io (460), and this protocol may be a negotiation (handshaking) method including error handling, DMA, interrupt, and enumeration.

[0191] In addition, the above-described predetermined implementation and application method may be a method used for manipulating the bus table (3300) in this patent, including the CXL.cache (460) protocol used for implementing cache coherence with the memory of the second computing device (201) when accessing the second computing device (202) from the first computing device (201).

[0192] In addition, the above-described predetermined implementation and application method may be a method used for manipulating the bus table (3300) in this patent, including the CXL.memory (470) protocol used for implementing non-coherence access when accessing the first computing device (201) from the second computing device (202).

[0193] However, without limitation to this, the specific methods of the predetermined bus page table and interconnect implementation using the bus page table, and communication methods can be modified in various ways at a level that is obvious to a person skilled in the art.

[0194]

[0195] FIG. 1 is a drawing illustrating a portion of the present invention from the perspective of a chip internal bus section (3000) and an application related to the bus page table (3300) between the chip internal bus (3010) and the chip external chiplet connection (3020).

[0196]

[0197] Referring to Figure 1, this is a part that briefly shows the inside of a single computing device (200), which is a CPU or application processor.

[0198] The core (210), which is the element responsible for the most basic operation and control of the computing device (200), which is a processor, is located in one or more numbers, and each core (210) has its own L1 (241), L2 (242), L3 (243) or more caches (240), and only through this cache (240) can the cores (210) exchange data to be operated / controlled internally within the chip with the bus (3010).

[0199] Specifically, assuming that the LLC (250) at the very end of each core (210) is an L3 cache (243), the L3 cache (243) at the very end (250) simultaneously provides a shared connection between the L2 caches (242) of each core (210) and has the role of a master (3100) that actually controls the bus (3010) located at the center of FIG. 1 within the conventional processor chip (200).

[0200] In addition, the bus (3010) controls the IP elements inside the chip (200), i.e., the Peripherals section connected to peripheral devices such as the LTE modem (320) or GPIO (314), and the controller (220) connected to the memory as a bus slave (3200) according to the command of the master (3100). When the operation / control related results from the core (210) pass through the core (210) and reach the L3 (243) / LLC (250), which is the last end after L1 (241) and L2 (242), the chip internal bus (3010) is controlled to put the data back into the cache (240) according to the results.

[0201] Controlling the above buses (3010) as a master (3100) is the role of the conventional processor (200) or CPU, and is the entirety of a computing device. Through the bus (3010) within the chip, the master (3100) inputs an address to the bus (3010), and reads and writes data.

[0202] However, as technology advances, the L1 cache (241) takes over what a single core (210) used to do, the L2 cache (242) takes over what the L1 cache (241) did, and with the emergence of multiple cores (210), the shared cache, the L3 cache (243), takes over and controls the bus (3010) to read and write. As a result, the processor's core (210) can be seen as having returned to the original computing device, like a DSP (digital signal processor) or accumulator, which mindlessly calculates data read and written by the L3 cache (243) and the address translation device, such as the TLB (260) linked to it, like a dumb pipe.

[0203] Here, the address within the chip bus (3010) is the address assigned to the assigned IP elements within the chip, i.e., the bus slaves (3200). In general, the address area (4 GB / 16 GB) of the volatile memory / memory controller (220) assigned to the computer is not entirely used by the bus (3010) within the chip, but it can be seen that an address area (64 bits) of that size exists.

[0204] However, due to the hardware-invariant bus system (3000) and address system, it became impossible to guarantee verifiability and reliability.

[0205]

[0206] More specifically, the authority of the master (3100) of the bus is taken back from the cache (240) that controls the bus (3010) as a master (3100) by receiving the delegation of the core (210) / core within the processor, which is a conventional computing device (200), and the core (210) / cache (240) is driven as a slave (3200) of the bus, like a DSP or an accumulator, with a dump pipe.

[0207] As a result, the core (210) / cache (240) delegated by the core can make a request to control the bus (3010), but cannot perform any internal bus control as a slave (3200) without permission from the open interconnect root complex (420) of FIG. 1, which is the sole master (3100) of the processor chip internal bus (3010), and delegates the request to the chip internal interconnect root complex (420) or, in the case of a device (300), to the endpoint (430).

[0208] Of course, in order to maintain the existing architecture or performance, the internal bus master (3100) of the chip also performs most of the requests pouring in from the dump pipe of the core (210) or the L3 cache unit (243), just like the L3 cache (243) / LLC (250) delegated to the core (210). To elaborate, the interconnect link unit (400), which is the internal master (3100), receives the input / output of the L3 cache (243) for each operating clock of the processor, and reads and writes the bus (3010) on its behalf to the slave (3200) L3 cache (243).

[0209] In addition, the address translation look-ahead part, such as the conventional TLB (260), still exists in the end cache (250) such as L3 (243), and does not compromise the conventional architecture and performance. Furthermore, the address inside the bus (3010) presented in FIG. 1 of the present invention is divided into paging units and allocated, and secondary address translation look-ahead elements, such as another TLB (260), are not placed in the chip's internal interconnect root complex (420) / endpoint (430).

[0210] The TLB (260) exists only in the core (210) / cache (240) and is passed through only once, thereby maintaining the conventional architecture and performance. In addition, the only master (3100) on the bus (3010) is the open interconnect root complex (420) / endpoint (430), which is verifiable and has no other components.

[0211] In modern processor architectures, in addition to the core (210) / L3 cache (243), the slave (3200) elements inside the chip are controlled through a crossbar (a technology in which multiple bus masters (3100) share the same bus slaves (3200), or more precisely, cross over and simultaneously control them). Things like DMA, which were introduced for performance, have the authority of another master (3100) of the internal bus of the chip, so they can transparently bypass the verifiable software kernel (500) or trusted elements once again. Therefore, in the above-mentioned Fig. 1, only the interconnect root complex (420) / endpoint (430) inside the chip has the authority of the sole master (3100) of the bus.

[0212] In conclusion, all things that had conventional master authority (3100), such as DMA, are demoted to the memory controller (220) or other slaves (3200) that are bus slaves (3200) within the chip, and the L3 cache (243) of the core (210) is also demoted, with control of the bus (3010) only possible through the delegated chip interconnect root complex (420) / endpoint (430).

[0213] Moreover, the purpose of introducing an interconnect such as the above CXL is to process a CXL packet that replaces the DMA of the conventional PCIe with a protocol such as mapping coherence, and as will be described later in FIG. 5, the root complex (420) / endpoint (430) operates internally as a bus master (3100) by linking with the bus table (3300) in advance by the kernel (500).

[0214]

[0215] Referring to FIG. 1, this is a part showing how to link the chip internal bus (3010) with the kernel section (500) to enable verification and make it reliable through the intervention of software (500) as an alternative suggested by the present invention.

[0216] The origin of the address translation device, such as the conventional TLB (260) located above the L3 cache (243) in the drawing, is the same as the history of the operating system (500). When there was no address translation element, such as the previous TLB (260), when allocating memory, the operating system (500) allocated the memory of the application (900) in order. The problem with that method is that, no matter how large the memory is, the application (900) does not return the memory in order. Just like the disk defragmenter of the conventional Windows, the memory that is terminated and returned out of order causes fragmentation between the memory allocatable areas, which ultimately leads to the problem that the memory area of ​​the application (900) to be executed next cannot be allocated to any area of ​​memory.

[0217] In order to solve the above problem, the conventional method of dividing memory into paging units, such as 4KB, and allocating the same, and when applications (900) are allocated memory by the operating system (500), they feel that they are occupying the entire memory by themselves in a straight line, but in reality, the allocated areas are divided into 4KB page units, typically by the operating system kernel (500), and are randomly fragmented into 4KB units and allocated to the physical address in the actual memory.

[0218] In addition, as the above memory allocation fragmentation problem was solved, the simple desire to solve this memory allocation fragmentation problem and the address translation element also unintentionally became the basis for solving the security problem of modern operating systems. However, the process of converting the paged virtual address of the application (900) into an actual physical address (Physical Address) once caused a tremendous performance drop, and the address translation table that manages between the virtual address and the physical address must be managed somewhere in the operating system (500).

[0219] Of course, fortunately, such address translation problem has been transferred to the processor (200) such as the CPU in the process of reading / writing the memory (220) directly in the L3 cache (243) / terminal (250), and there is no performance degradation. Furthermore, for security purposes, ASLR (Address Space Layout Randomization), which is a memory paging area randomization that was previously performed only in the application (900) area, has been applied to the operating system's own kernel (500) (KASLR), thereby further strengthening security without performance degradation.

[0220] However, what is presented in Fig. 1 is a method of similarly applying such paging to the bus area (3000) inside the chip.

[0221]

[0222] More specifically, the bus (3010) area inside the conventional processor (200) is where the master (3100) that inputs the address value of the bus and performs read / write, and the slaves (3200) that are subject to read / write control by the bus (3010) are located. The bus slaves (3200) are set to the address area of ​​the bus (3010), i.e., the start address and end address values, to the size that can be accommodated during chip manufacturing / design.

[0223] The reason why the physical bus address (3320) area inside the processor is also used in conjunction with the memory physical page address (Physical Address) that we use in the program (900) is that the memory controlled by the processor, which is a computing device (200), is also ultimately controlled and read / written through this bus address area, so for design convenience, the processor internal bus address is usually designed to match the physical address area of ​​the memory.

[0224] For example, typically, within the application processor (200) bus (3010), the L3 (243) / L2 cache (242) is set as a master (3100) through a bus such as AXI, and each peripheral element such as GPIO (314) or SPI, I2C controlled within it is typically allocated 4KB to 64KB.

[0225]

[0226] However, unfortunately, if the bus address is divided into 4KB units, approximately 4.2 million entries (3330) are required to accommodate all 16 GB, and if each entry (3330) has a 64-bit address system / 48-bit address system, it is 4.2 million times 64 bits (8 bytes), or 4.2 million times 48 bits (6 bytes), so it requires approximately 32MB / 24MB.

[0227] This not only requires an enormous amount of cost in SRAM, but even if it is considered a secondary issue, as will be described later, in the core (210) dependent on the proposed bus in the form of a slave (3200), most of the cycle speed of the bus (3010) must be consumed in traversing the bus table (3300), so the bandwidth of the bus (3010) is drastically reduced.

[0228] Also, since the speed of the bus (3010) determines the speed of the entire chiplet connection (3020) and all elements inside the chip, as well as the core (210), if the bus table (3300) is not HIT and converted within 1 cycle, or 2 to 3 cycles, the speed of the entire system may be slowed down exponentially in inverse proportion, although it may solve the NUMA / SMP problem.

[0229] Moreover, the problem of how many cycles it takes for an address to be translated (HIT) within the bus table (3300) is also a major problem within the cache (240), and if the table (3300) is segmented into 4KB units such as memory, even if the table with 4.2 million entries is relaxed in a 2-way, 4-way, or 8-way manner, it still takes approximately 2.1 million / 1.05 million cycles to traverse the table (3300).

[0230] However, this is one of the biggest problems that the cache (240) has suffered by introducing the concept of virtual addresses, and there is still no solution to it even if a lookahead address concept such as TLB (260) is used. The above reason is also the reason why Intel used a method such as random speculative execution inside the cache (240) despite causing several TLB bugs and hardware security bugs. No matter how much the bus (3010) / core (210) speed is increased, it is difficult to recover the performance degradation caused by a cache (240) miss under a virtual address such as TLB (260). Since the above part is difficult to alleviate even if various techniques are introduced, the reason why TLB (260) passes through L3 (243) / LLC (250) only once is precisely for this reason.

[0231] However, just as the benefits of the paging concept are much greater, the significance of segmenting bus addresses is also significant, so for now, it is meaningful to start with a bus address granularity of about 4 MB.

[0232] Although the precision of the above 4MB table (3300) does not have the security of 4KB in the memory paging concept, it alleviates the NUMA / SMP problem, and in the case of 16GB, the table entry (3330) of the bus master (3100) only needs to have about 4096, and 64bits (8bytes) require a total capacity of 32KB, and 48bits (6bytes) require a total of 24KB, and the internal conversion requires only a maximum of 4096 times in the case of 1 way and a maximum of 2048 times in the case of 2 ways to traverse the entire thing, so it does not have a large impact on the entire bus (3010) / cache (240) system. In addition, since the precision is also reduced, the degree required for address translation is also alleviated. Detailed examples are additionally described in FIGS. 3A and 3B.

[0233]

[0234] Referring back to Figure 1, if the first bus slave (3200) element is 4MB, the bus address values ​​that can access this slave (3200) element on the bus (3010) start from 0x00 and end at 0x3FFFFF. Of course, the bus address can also start from an offset or a higher address, and there can be various variations.

[0235] Likewise, if the second slave (3200) element, Periphereals, is 4MB as in the example above, its bus address starts at 0x400000 and ends at 0x7FFFFF, and the third element, Modem (320), is also 4MB and starts at 0x800000 and ends at 0xBFFFFF. However, if the next slave (3200) is a memory controller (220), its address size is different from that of the peripheral or modem (320) elements, and if it is a controller (220) with 16GB as in the example, its starting address on the bus (3010) will start at 0xC00000 and end at 0x400BFFFFF.

[0236] However, for design convenience, the address lines inside the processor are accessible only as wide as the address width of the chip's internal bus (3010) such as AXI.

[0237] For example, if the bus address line inside the chip is 32 bits, the 32-bit address area of ​​the bus (3010) that the slave (3200) can have ends from 0x00 to 0xFFFFFFFF, which is the conventional 4 GB limit. However, in the above example, since the hardware of the slave (3200) ends with 0x400BFFFFF, the remaining area cannot be accessed by the master (3100), or even if it is accessed, it cannot be read or written normally.

[0238] Likewise, for modern processors, for design convenience in 64-bit, the internal bus address (3320) can have as many as 0xFFFFFFFFFFFFFFFF, which is 2^64, but such an address cannot have actual physical capacity, so the remaining area after 0x400BFFFFF cannot be accessed normally for reading or writing. However, for the above-mentioned memory paging and security reasons, it exists in the form of a virtual bus (3310) / physical memory page address.

[0239] Also, as with the 32-bit or 64-bit internal bus address lines, the bus data lines also have the same width for design convenience / advantage, and this is generally used in most processors (200), but this method may not be used.

[0240] However, it should be noted that the address area and allocation of the bus (3010) inside the chip above are not virtual bus addresses (3310) that are finely divided into paging units such as 4 MB as a virtual address concept as used in the operating system (500), but rather are internal physical addresses (3320) of the actual bus (3010) that have not yet passed through the bus table (3300), and were the physical page addresses of the conventional memory.

[0241] In conclusion, inside the processor (200), the virtual page memory address is converted into the physical paging address of the memory and accessed by the L3 cache (243) and TLB (260). The physical address area of ​​the memory is typically an internal bus address (3320), which means that it is physically wired during chip design / manufacturing and cannot be changed, and is divided into paging units like software memory, meaning that it cannot be protected or verified.

[0242] Of course, if you do not trust the processor itself, which is the computing device (200), then it is right not to use the computer in the first place. However, if it is not simply a problem of the computing device (200) being the processor, but a problem of sharing memory or address concepts with computing devices from multiple vendors, such as in a company or a cluster, the situation is different.

[0243]

[0244] Referring to FIG. 1, to alleviate this problem, as described above, the core (210) and L3 (243) / LLC (240) are simply demoted from bus master (3100) to bus slave (3200) elements, and the master (3100) of the chip's internal bus is transferred to a trusted interconnect root complex (420) / endpoint (430). This transfer is essential not only for the hardware reliability / verification presented in the present inventions, but also for alleviating the SMP / NUMA problem presented in FIG. 2.

[0245] In addition, it does not end there, and the address values ​​of the internal bus slave (3200) elements of the chip are set without going through the address translation table of the core (210) such as the TLB (260), but rather the bus paging table (3300) is physically located within the interconnect root complex (420) / endpoint (430).

[0246] However, the reason why it does not have a bus address translation lookahead section of an address translation lookahead structure, such as the TLB (260) of the conventional L3 cache (243) / LLC (250), is to avoid damaging the conventional architecture and performance. However, as shown in FIG. 1 of the present invention, the bus address area inside the chip is not always internally constant, but is internally segmented into paging units as soon as the power is turned on to the processor chip, and the address assignment of the bus slave (3200) is always changed, thereby minimizing the intervention of unreliable hardware such as the TLB (260) or the core (210).

[0247] For example, as suggested above, in order to maintain compatibility with the existing architecture, if the L3 cache (243) still has a TLB (260), and the interconnect root complex (420), which is the only bus master (3100) inside the chip, has one more bus address translation lookahead element similar to the TLB (260), it can be integrated into the TLB (260) or cause another modification by manufacturers or architecture companies such as ARM.

[0248] Moreover, since the TLB (260) itself is a concept that was derived from the existing operating system (500), was located within software, and was then transferred to hardware, the operating principle of the TLB (260) is that when the CPU tries to read a specific virtual address, if it does not know the actual physical address where it is located in advance due to the limitation of cache capacity, it must eventually go to an interrupt or a specific page directory, and then to the area where the promised address translation table of the kernel (500) is located.

[0249] References to the kernel area through hardware interrupts such as TLB (260) or fixed address values ​​may not be available when the bus slave (3200) elements within memory or other chips have not yet been initialized. Furthermore, even if the initial address is fixed and the bus address is paged thereafter, references to the bus address table due to cache misses lead to greater security issues that render the above-mentioned inventions ineffective and to performance degradation.

[0250] In addition, the concept of the conventional TLB (260) itself, when the operating system (500) creates a virtual address, requires each application (900) or kernel (500) area to have an address translation table, and must cover the entire virtual address area that is not actually used, so cache missing is inevitable. However, in the bus inside the chip that knows the number and size of the bus slaves (3200), the internal bus paging address can be sufficiently mapped with only a small area (tens of KB) inside the interconnect root complex (420) / endpoint (430).

[0251] More specifically, as illustrated in the above drawing 1, if there are three bus slave (3200) elements, since the L3 (243), which is a shared cache between the cores (210), is 4 MB, when the paging size of the example is divided into 4 MB units, it has a value of 4 MB divided by 4 MB, that is, 1 bus paging count (3330). Next, the peripheral register (230), which is an example, has a paging count of 1, which is 4 MB, due to internal paging fragmentation when the hardware address size is 4 KB, for example. In addition, even if the modem (320), which is an SOC within the chip, is a slightly larger 64 KB, it is also 1 4 MB. In addition, the memory controller (220), if the processor (200) of the example supports a maximum of 16 GB, is 16 x 1024 divided by 4 MB, which is 4,096.

[0252] In the above example, since a virtual bus address (3310) space corresponding to a total of approximately 4099 64-bit internal physical bus addresses (3320) is required, it can be inferred that a total of 64 bits, or 8 bytes, is required for 4099. This is a very small amount, slightly over 32 KB in a computing device, so the interconnect root complex (420) / endpoint (430), which is the internal bus master (3100) of the processor chip, can maintain the information without sharing it entirely in the form of SRAM or registers. Of course, since it is not necessary to have the entire 64-bit virtual bus address value (3310), it occupies a smaller capacity, approximately 21.5 KB, and this calculation method is expressed in FIGS. 3A and 3B.

[0253] However, due to the cycle problem of traversing the bus table (3300) rather than the above capacity problem, it is currently impossible to increase the granularity more precisely because it would cause not only the bus speed but also the entire core (210) to halt (HALT).

[0254] Also, even in the case of a memory controller (220) requiring the largest bus address space (3300), even if the internal chip bus address space (3300) of 32 GB or hundreds of gigabytes is required in the future, the size of the bus address mapping table (3300) increases in direct proportion to the increase, and it still requires a small capacity for the interconnect root complex (420) / endpoint (430).

[0255] The reason why the above bus page table (3300) has a very small capacity is that the paging concept of address translation required by the operating system kernel (500) or the TLB (260) of the L3 cache (243) / LLC (250) is not simply a 1:1 conversion of the virtual bus address (3310) / internal physical bus address (3320) in hardware, but also goes beyond the cache validity or actual address translation required by modern processor cores (210). Of course, the precision (granularity) of 4KB is the biggest factor.

[0256] In addition, the paging size in the bus address, although not recommended, can be segmented into various units such as 64 MB, which is larger than 4 MB, unlike the method of the above example in the kernel section (500), so the value required for the bus address paging table (3300) can be greatly reduced and is flexible depending on the capacity of the memory controller (220).

[0257] For example, the interconnect root complex (420) / endpoint (430), which is the master (3100) of the chip internal bus, only needs to have 1:1 mapping space / entries (3330) for a total of 4,096 + 3 64-bit address values ​​in the above example, but if segmented into 8MB units, which is twice that, it only needs to have half of that, or roughly 2,048 (16KB) + 3 (24Byte).

[0258] However, the above bus table (3300) does not necessarily need to exist to solve some (NUMA) problems, as in the example in Fig. 2c. Of course, since the bus address (3320) is not segmented, it must be linear, and Symmetric; SMP still remains.

[0259] In addition, since the bus paging table (3300) is used when completely replacing the conventional DMA / IOMMU (DTLB) described later in FIGS. 5 with an interconnect mapping coherence maintenance protocol, it can be said that having a bus table (3300) is much more advantageous.

[0260] However, each entry (3330) of this mapping table (3300) is never changed until the processor is reset once the value is set when the initial lightweight kernel / kernel unit (500) creates the virtual paging memory, and although it can be set when going through the bootstrap with the CXL Wake On / Power On protocol, or when the operating system (500) in non-linear protection mode writes (write-through) a table area somewhere in the memory that is arbitrarily hidden among the conventional virtual page tables, the value cannot be queried unless the virtual bus address (3310) is directly known in the protection mode. If the virtual bus address (3310) inside the chip is not known, the process of initializing the chip (200) must be repeated.

[0261] More specifically, according to the volatile virtual bus address (3310) mapping within the interconnect root complex (420) / endpoint (430) which is the bus master (3100), the slaves (3200) are allocated virtual bus addresses (3310) in segmented units and converted into internal bus addresses (3320). At this time, in the protected mode, the bus master (3100) is not the same element as the TLB (260), so if the core (210) or the L3 cache (243) / LLC (250) delegated by the core internally in the chip requests an incorrect (non-existent) chip internal bus address (3310) to the bus master table (3300) by a malicious application (900) / hardware, unlike the conventional TLB (260), it cannot and should not go to the fixed area of ​​the operating system (500) via an interrupt, so a NULL value or a garbage value in the form of a random number may be returned to the core (210) / L3 cache (243), or the same exception handling (Exception) bit as before may be loaded together in the control cache area and then uploaded back to the L3 (243) / LLC (250).

[0262] However, as recommended above, in cases where confidential computing, such as in enterprises, is required, malicious applications (900) / hardware are ignored and mishandled to maintain hardware integrity without handling failures / exceptions. This handling method may lead to similar results as conventional TLB bugs when normal processing fails, but under ECC memory and normal operating system (500) / process (900) routines, such cases are rare enough to cause hash collisions.

[0263] In the case of a malicious routine, it is beneficial to have the operating system restart or the chip reset by the interconnect cluster / Kubernetes (680) to prevent incorrect access from propagating throughout the interconnect link (400) while maintaining the compatibility and performance of the existing architecture. Furthermore, under the interconnect, the death of one computing device node or one or two kernels does not affect the entire cluster system, such as Kubernetes (680).

[0264] Of course, for general users who do not require confidential corporate computing, it is sufficient to simply return to the bus intercept chain section (5600) of the kernel section (500) in the event of a failure / exception by using various protection techniques presented in FIGS. 3 and 4.

[0265]

[0266] Referring to FIG. 1, the bus address paging (3300) setting of the internal interconnect link section (400) of the chip is initialized together with the processor (200) chip when it is reset, and this bus address conversion table (3300) is never shared with other cores (210) or L3 (243) / LLC (250), which are slave (3200) elements within the chip.

[0267] As explained in more detail in Figures 3, in the Wake On / Power On of the initially open interconnect protocol or in the non-linear protection mode, the setting is only possible when the kernel (500) writes (writes through) the bus address table area (3330) hidden somewhere in the memory that the previous kernel arbitrarily set during the initial bootstrap process.

[0268] In addition, the interconnect root complex (420) / endpoint (430), which is the bus master (3100) inside the chip, is in the form of a simple bus controller (3110) when the SRAM or interconnect link unit (400) is removed, so it is composed of very few elements, and even if this is difficult to trust, the area it occupies is very small compared to other slaves (3200) in the chip, and it can be easily verified using an electron microscope or the like.

[0269] In addition, during the chip initialization process, the bus address paging (3300) can be completely verified, and the process difficulty and unit cost of other slave (3200) areas can be manufactured separately and then connected / replaced with an open chiplet connection (3020) / bus (3010) element such as UCIe inside the chip (200), and the physical bus address and data signals going back and forth between the UCIe chiplet connection (3020) / bus (3010) can be easily verified with a UCIe signal reflector, etc. during the non-protected mode bootstrap before the initial protected mode. The subsequent process can be easily protected in 1 or 2 cycles without memory copy / migration, using various techniques such as applying Protection SWAP (5000) in the bus address table (3300), as will be described in the following FIGS.

[0270] Moreover, the paging of these internal chip bus addresses has meaning not only in having a paged bus address (3310) within the chip, but also as a virtual bus address / global bus address (3310) linked to the kernel's bus paging table (3300) outside the chiplet connection (3020), even when connected to an open chiplet connection (3020) such as UCIe in the above-described FIG. 2.

[0271] For example, slaves (3200) between multiple chipsets do not overlap at all, are assigned a 64-bit virtual bus address (3310) of the above example, and are managed as a paging unit by one node and one kernel (500), so that bus signals / memory controllers (220) can be easily controlled / shared like a conventional crossbar format as suggested in the present invention.

[0272] Although the interconnect link (400) of this patent is not limited to CXL, in the conventional CXL, when memory is expanded throughout the CXL through a memory expander, the memory paging table management problem that occurs between kernels (500) has been a factor that reduces the performance and security of CXL.

[0273] More specifically, when expanding memory (220) through an interconnect link (400) such as CXL, a problem arises in that table information managing paging addresses and references for CXL shared memory across the cluster must be shared by multiple other nodes and kernels (500). This may cause security issues to be transferred to all interconnect nodes, not to mention performance issues.

[0274] In addition, contrary to the purpose of introducing CXL memory, there is also a problem that it is difficult to easily expand / replace. Unlike the above CXL memory expander, what is presented in the patent, Fig. 2, is that the open chiplet connection (3020) / bus (3010) technology such as UCIe can be gradually used to expand the bus area paged by the interconnect root complex (420) / endpoint (430) and the single kernel unit (500) to other chips outside the chip through the open chiplet connection (3020) / bus (3010) such as UCIe through the bootstrap process presented in the above Figs.

[0275] Of course, the alternative presented in the present invention does not solve the problem in a situation where many nodes and kernels (500) share the actual virtual memory paging concept across the CXL, such as the CXL memory expander or DirectCXL (a concept developed by KAIST), but as described above, it is not recommended in terms of performance or security for many nodes and kernels (500) to share the virtual memory paging address and TLB (260) used by the kernel part (500) across the interconnect.

[0276] In conclusion, it is advantageous for memory (220) to be managed separately by a single node and a single operating system kernel (500) from both a performance and security / management perspective. However, the significance of CXL is significant simply as a mapping coherence protocol that replaces the existing PCIe DMA.

[0277] In addition, it is recommended to share resources in the form of a virtual memory device (820) by appropriately utilizing the above elements and virtual device (700) to share the virtual device (700) across Kubernetes (680) nodes connected by an interconnect link (400).

[0278] However, the bus address paging (3300) between chiplet connections (3020) presented in FIGS. 2 segments the already determined internal bus address (3310) values ​​into paging units, and since only one kernel (500) is present within one node composed of one / multiple computing devices (200), there are no security or performance issues associated with sharing a bus table (3300) between nodes / kernels (500) across an interconnect link (400) such as CXL.

[0279]

[0280] FIG. 2 is a drawing showing the application of the bus page table (3300) technology shown in FIG. 1, the application in the chiplet connection (3020) during bus address paging, and the crossbar-type structure.

[0281]

[0282] The painstaking efforts to verify and secure the hardware described above do not necessarily mean sacrificing anything for the sake of security. Just as the memory paging and address translation elements of the kernel (500) were initially designed to prevent memory fragmentation, but unintentionally became a crucial cornerstone of modern operating system security, paging the bus address space has significant benefits beyond security.

[0283] Moreover, the biggest secondary benefit, other than security, is the prevention of fragmentation of internal bus addresses (3320), which is the initial reason for designing the paging table, and the significance of preventing fragmentation at the internal physical bus address (3320) has great implications in the open chiplet connection (3020) / bus (3010) structure.

[0284] Specifically, preventing fragmentation of the internal bus address (3320) is very useful for memory expanders using chiplet connections (3020) or expansion of processors (200) using chiplet connections (3020) within a single node / single kernel (500), other than interconnect link (400) methods such as CXL.

[0285]

[0286] Referring to FIG. 2, as an example of connecting a single computing device (200) described in the above FIGS. 1 to a plurality of devices, the first computing device (201) and the second computing device (202) are shown, and share a common chiplet connection (3020) line through an open chiplet connection (3020) such as UCIe within the chipset.

[0287] However, the above-described FIG. 2a only shows a computing device (200) having the same processor for illustrative purposes, and in the same manner as the open chiplet connection (3020) / bus (3010) such as the conventional UCIe, the first computing device (201) may be a processor, and the second computing device (202) may be a memory controller (220) or a memory expander.

[0288] In addition, a plurality of third computing devices (203), fourth computing devices (204), etc., which share the same chiplet connection (3020) / bus (3010) line, may be differently modified, such as a computing device (200) of a modem chip (320) or other peripheral devices.

[0289] Moreover, the wiring method of the above chiplet connection (3020) / bus (3010) line may be such that one chiplet connection (3020) line may be serially shared like a typical serial bus structure, or, differently from that, may be such that one / multiple parallel structures like a parallel bus structure, a tree structure, or various modifications and applications may be possible. In addition, one chiplet connection (3020) / bus (3010) line may be connected in a circulating manner between the computing devices (200) without being disconnected at any point like a ring bus structure.

[0290]

[0291] In Fig. 2, it does not simply show the connection of the chiplet connection (3020) presented in the past, but rather shows a part applied so that multiple computing devices (200) can share the same bus address table (3300) during such connection.

[0292] In the above example, there is a first computing device (201) and a second computing device (202), and a single serial type chiplet connection (3020) line that mediates between them. When the two computing devices (200), i.e., the first computing device (201) and the second computing device (202), are connected to the bus (3010) inside the two computing devices (200) through the chiplet connection (3020), the internal bus lines (3010) between the two computing devices (200) operate as if they are connected by a single bus line (3010).

[0293] For the above reasons, the open chiplet connection (3020) such as UCIe proposed by Intel is typically made to be used only when directly connecting one element within the chip (200), for example, between a modem chip (320) and a core (210), or between a core (210) and an external graphics core.

[0294] Moreover, for the above reasons, when directly connected 1:1 or when a bus signal such as AXI is mediated by UCIe, the master-slave relationship between the two chips / processors (200) must be clear, just like a normal master and slave.

[0295] However, in addition to such a method, FIG. 2 and FIG. 2 described below map the bus physical address (3320) presented in FIG. 1 to a bus virtual address (3310) using a conventional AXI crossbar-like method, thereby enabling the buses (3010) between two or multiple chips (200) to operate as if they were one chip (200) originally designed to be connected by one bus (3010).

[0296] Of course, as described in the above-described Fig. 1, a bus address table (3300) must be managed as one node and one kernel (500) between multiple chips (200) mediated by a shared chiplet connection (3020). Unfortunately, in order to be managed by multiple kernels (500) / nodes across the interconnect link (400), such as the CXL memory expander, the above-described TLB (260) or the method suggested by DirectCXL must be used, and the introduction of this concept in the virtual bus address (3310) is not appropriate.

[0297] However, in the case of a single node / single kernel unit (500), with only one bus address table (3300) as described in the above-described FIG. 1, software elements such as cores (210) / cache (240) do not pass through elements such as conventional TLBs (260) for virtual bus addresses (3310), so security is a secondary benefit, and in terms of performance and expandability, the bus (3010) can be shared as if it were within one chip (200).

[0298] The reason why the above elements are possible is that, although not open between conventional chips and chips, they were also used by mediating a bus (3010) such as AXI through an independent chiplet connection (3020). However, here, the AXI bus mediated by the chiplet connection (3020) cannot have multiple bus masters (3100) similar to the above drawing, and in order to access each other's bus slaves (3200) in the form of a crossbar, one of them must be a subordinate of another bus master (3100) and relay its own bus slaves (3200).

[0299] The reason is that the physical bus address (3320) area between the two chiplet connections (3020) must be hardware-fixed, whether it is ARM's AXI, Intel's UPI, QPI, or whatever. This also means that the two bus address areas (3320) must not physically overlap and must be linearly added to each other.

[0300] Of course, even if the address (3320) area between buses (3010) is reconfigurable for expandability, such as chiplet connection (3020), similar to the case inside an FPGA, and crossbar-type bus sharing is possible, the problem derived from SMP / NUMA, which is the memory controller (220) access in software, still cannot be avoided.

[0301] Specifically, when two computing devices, processors (200), are connected in a form such as a chiplet connection (3020) to share a bus (3010), the two memories must be shared as a single virtual memory address area by the memory controller (220) inside the two processors (200). This is SMP (Shared Memory Processing / Symmetric Multi Processing), and it means to create a single memory address area through the virtual memory address between the two processors (200).

[0302] Also, this structure is used when connecting CPUs in units of two or four in a server. As the name suggests, the processors of the two computing devices (200) must have the same specifications, and the size of the memory connected to the memory controllers (220) of the two processors (200) must also be symmetrical. Unfortunately, the integration of conventional asymmetric processors (200) or asymmetric memory sizes exists only in theory as MMP (Multiple Memory Processing) and is not actually used in the field. In MMP, since the same virtual memory address is not used between multiple memories like in SMP, the memory (220) between the two processors, that is, the two computing devices (200), cannot be moved without any virtual address->physical address->physical address->virtual address conversion work.

[0303] In addition, when SMP moves between two processors (200) or between the memory areas of two memory controllers (220), not only does the performance of elements such as the TLB (260) deteriorate, but a thread being processed in the cache (240) of the other processor (200) has to maintain coherence between the caches (240) of the other processors (200), unlike when accessing its own local memory controller (220), and a much larger and more complex problem occurs along with a very slow access time. For that reason, in SMP, the operating system kernel (500) usually considers NUMA (Non Unified Memory Access) for the initial memory allocation so that it accesses only its own local memory (220) as much as possible. Therefore, there was a problem in which process occupancy was asymmetrically concentrated on only one processor (200) among the two processors (200).

[0304] In conclusion, the SMP concept is inherent in the structure of modern processors (200) / applications (900), and since memory is shared through the virtual address concept, it means that it cannot help but depend on the kernel (500) and the memory address paging table.

[0305]

[0306] The example and application of Fig. 2a are examples of a method for alleviating the problem of allowing only symmetrical processors, which are symmetrical computing devices (200) in an SMP (Shared Memory Processing / Symmetric Multi Processing) structure, and the NUMA (Non Unified Memory Access) problem through a bus page table (3300) structure.

[0307]

[0308] Referring to FIG. 2a, in the case of a conventional SMP processor, very differently from FIG. 1, FIG. 2, and FIG. 2a, the core (210) / L3 cache (243) is a bus master (3100) and not a slave (3200), and therefore is not visible anywhere in the slave (3200) address area. In addition, even though the peripheral areas such as the conventional SPI, I2C, interrupt areas such as the APIC, power elements, or the conventional BIOS area should be in the bus address (3320) as bus slaves (3200), they are hidden in the x86 protected mode and cannot even be accessed.

[0309] In the conventional x86, considering the inherent limitations of SMP (Shared Memory Processing / Symmetric Multi Processing), only the memory address area of ​​the memory controller (220) can be used as the bus slave (3200) address area in the inherent protected mode. However, such an action eliminates the possibility of hardware verification and reliability, and does not resolve the inherent limitations of SMP.

[0310]

[0311] Referring to FIG. 2a, since there is no fragmentation of the physical bus (3320) from the conventional SMP processor (200) structure, there is no need to have a completely different structure between slaves (3200), for example, there is no need to have an unconditionally symmetrical bus address (3320), and there is no need to expand to only a bus (3010) of multiples of 2, and it does not matter if there are three asymmetric structures.

[0312] Also, it does not have to be the exact same type of processor (200), and even if a processor (200) and a memory expander, or again a completely different type of processor (200) come to the chiplet connection (3020) and the bus (3010), a completely different type of internal bus address (3320) can be added together in a straight line (linearly).

[0313] However, the sharing of chiplet connection (3020) / bus (3010) lines between the above-mentioned completely different computing devices (200) must have the same address width, such as 64 bits if it is 64 bits and 32 bits if it is 32 bits. Furthermore, when executing the same process between computing devices (200) that are processors, it goes without saying that the same architecture must be used.

[0314] However, if you want to share only the memory in a chiplet connection (3020) structure with a crossbar, such as between CPUs or GPUs, such as Apple's Unified Memory or within the application processor (200) of a smartphone, you do not need to do so, and only need to comply with the width of the internal bus line (3010).

[0315] Moreover, even if the first computing device (201) or the second computing device (202), which is a processor (200), or the memory expander is suddenly removed in a parallel or ring-bus structure without rebooting within a single node / kernel (500), the bus address table (3300) structure presented in FIG. 2A can still operate normally. Of course, as a matter of course, in the serial chiplet connection (3020) structure, normal operation is not possible in that case because the bus (3010) connection is disconnected in the middle. Furthermore, since the method presented in FIGS. 2A can transmit the chip internal bus (3010) signal as it is without conversion when connecting the chiplet connection (3020), there is no difference between a serial connection and a ring-bus connection, so it can be said that the ring-bus form is suitable for serial connection.

[0316] In addition, since the internal bus address (3320) between two / multiple memory controllers (220) address areas of SMP (Shared Memory Processing / Symmetric Multi Processing) is first assigned a virtual bus address (3310) before having the internal address (3320), the NUMA problem (an action that can only concentrate work on one side of the processor) can be alleviated to some extent.

[0317]

[0318] Specifically, as mentioned in the above FIGS. 1 and 2, when the first computing device (201), which is the processor (200) in FIG. 2A, first goes through the bootstrap process of FIGS. 3 by the kernel unit (500), a bus address table (3301), which is approximately 21.5 KB in size, is set to arbitrary values ​​in a specific area of ​​the memory controller (220) of the first computing device (201) and the interconnect root complex (420), which is the only bus master (3100) inside the first computing device (201). However, as shown in FIG. 3C, since the virtual memory paging is done every 4 KB, which is the example, the operating system kernel unit (500) must divide the bus address table (3301), which is the example, into 4 KB units and record them in the virtual paging table / directories. Additionally, the starting address value of each virtual page entry that accommodates the divided bus table (3300) may be the same as that in the bus page table (3300) or may have other variations.

[0319] In addition, the kernel (500) is connected to the chiplet connection (3020) of the second computing device (202) through an external chiplet connection (3020) / bus (3010) line connected to the first computing device (201) through the chiplet connection (3020) in the following bootstrapping process, and the internal bus (3010) lines between the two computing devices (200) are connected as one bus line (3010) through the chiplet connection (3020).

[0320] The internal bus address table (3301) of the first computing device (201) is completely set by the kernel unit (500), whereas the bus address table (3302) of the second computing device (202) is still empty, even in the L3 cache (243) / LLC (250).

[0321] However, the first computing device (201) and kernel unit (500) send any byte code or kernel unit (500) thread to the second computing device (202) L3 cache (243) through the chiplet connection (3020) in order to bootstrap the second computing device (202) in the next process.

[0322] Since the above second computing device (202) is in the initial linear unprotected mode, even if the L3 cache (243) / LLC (250) address portion is not set, it can be accessed directly without bus address (3310) conversion through a chiplet connection (3020) such as UCIe.

[0323] Also, as in the initial linear unprotected bootstrap process, the kernel threads inside the second computing device (202) circulate through the multi-core (210) and L3 cache (243) of the second computing device (202), and gradually fill up the bus address mapping table (3302) of the interconnect root complex (420), which is the only bus master (3100) inside the second computing device (202), and the bus address arbitrary area of ​​the memory controller (220), which is an example of approximately 21.5 KB for the second computing device (202).

[0324] However, the address of the bus table (3302) of the second computing device (202) is compared by the kernel unit (500) with various techniques such as hashing or the values ​​of each entry (3330) of the bus table (3301) of the first computing device (201) to prevent the 64-bit / 48-bit virtual bus address (3310) area from overlapping. Normally, even if only the hash function is used, there is almost no case of overlapping of two virtual bus addresses (3310), but there should still be no collision between the two initially generated address tables (3300), the first bus table (3301) and the second bus table (3302).

[0325]

[0326] The above two computing devices, the first computing device (201) bus address table (3301) and the second computing device (202) bus address table (3302), start with 0x00 and end with 0x400BFFFFF in the above example on the physical bus address (3320) inside the processor (200), but the virtual bus address (3310) mapped thereto is paged in units of 4 MB in the above example, and is randomly scattered on the virtual 64-bit / 48-bit bus address (3310) from 0x00 to 0xFFFFFFFFFFFFFFFF without any overlap.

[0327] The meaning of this is that, although hardware verification and security are significant, when the bus (3010) signal is directly transmitted / shared through the chiplet connection (3020) between two computing devices (200), the root complex (420), which is two bus masters (3100), can share and simultaneously control the slaves (3200) between the two computing devices through a single bus line, like a conventional AXI crossbar, by crossing them over.

[0328] In addition, the two bus masters (3100) do not only have twice the bandwidth and controllability of the conventional crossbar, but also, unlike the conventional crossbar, since the two physical bus address (3320) areas are paged in units of 4 MB, there is no need for a linear connection between two or more memory controllers (220). Furthermore, even if the Peripherals or L3 cache (243) bus address (3320) is suddenly located in the physical bus address (3320) area of ​​another slave (3200) in between, since the virtual bus address (3310) is non-linearly set in units of paging, it is completely safe to use it as an address that mixes the meaning of a shared memory address as a software program, in addition to the hardware virtual bus address (3310).

[0329] Moreover, such a nonlinear virtual bus address (3310) does not go through a conversion such as a TLB (260) or a paging table of the kernel unit (500), and as described above, when the kernel unit (500) allocates a virtual paging address, it allocates a physical memory page address by considering the virtual bus address (3310) value, so that it appears transparent to the conventional architecture without a performance degradation unless Protection SWAP (5000) is frequently applied. Of course, if Protection SWAP (5000) is frequently applied to the bus address (3310), the TLB (260) must be recalculated through interaction with the kernel unit (500) because all of its values ​​become invalid.

[0330] In addition, since the virtual bus address (3310) / physical memory address space is non-linearly segmented into bus page precision units, even if a chiplet connection (3020) is added / removed in the middle, if only the memory information there is copied / transferred to another virtual bus address (3310) space by the kernel (500), there is no problem in removing it directly from the software or virtual memory address. This advantage is that when a processor (200) / memory expander is added through a chiplet connection (3020), the kernel (500), which is the operating system, does not require any special processing, such as immediately re-creating the memory paging table.

[0331]

[0332] Referring to Fig. 2a, two things can be observed when an update occurs in the L3 cache (243) of the first computing device (201). As in conventional programming, assuming that there are three variables (170) for simplicity, the cores (210) operate in the first computing device (201), and ultimately, in the L3 cache (243), which is the LLC (250), two variables (170) whose contents inside the L3 cache (243) are updated by the cores (210) can be observed.

[0333] X (171), a pseudo code variable inside the first computing device (201), has a specific memory / bus address value (3310) in the shared virtual memory, and it points to a specific address value (3310) of its own memory controller (220).

[0334] However, the cache (240) is only used as a temporary reference concept to quickly compute its contents, and cannot be guaranteed to have consistency with the contents of the actual memory controller (220). If the contents of the memory do not match, either the contents of the cache (240) or the memory are considered meaningless garbage values.

[0335] In order to prevent such a phenomenon of coherence being broken, inside the conventional cache (240) / TLB (260), a Valid bit that checks whether the virtual address value of the memory is valid or whether the contents of the memory have been loaded (stored) into the cache is usually arranged in an array form together for each cache line / block of about 1 bit, and a Dirty bit that checks whether the value has been updated by the LLC (250) / L2 (242) / core (210) higher than the L3 (243) in the above example is also usually together of about 1 bit. However, here, the concepts such as Cache-Clean / Cache-Flush inside the cache (240) do not significantly affect the contents of the drawing, so they are omitted.

[0336] The above TLB (260) / cache (240) uses these two items to decide whether to fetch data from the memory (220) (Load / Read) or load data (Store / Write). Therefore, when the X value (171) of L3 (243) of the example of Fig. 2a is updated / written by the upper L2 cache (242) or the core (210), the dirty bit of the cache (240) lines / blocks is set to 1.

[0337] In Fig. 2a, it can be seen that two variables (170) are made with the dirty bit set to 1, and the TLB (260) / cache (240) has a write / store process that updates these two variables (170) in the memory through several processes in the conventional case to ensure coherence.

[0338] However, in this patent and FIG. 2a, since the TLB (260) / cache (240) is a slave (3200) rather than a master (3100) unlike in the past, such writing is done by delegating the memory write / store to the sole bus master (3100) of the first computing device (201) rather than the TLB (260) / cache (240) itself.

[0339] In Figure 2a, two types of write operations are derived, similar to the conventional cache write / store operations. One is the simplest write-through method, and the other is the write-back method, which requires a slightly more complex algorithm for the TLB (260) / cache (240).

[0340] The above Write-Through method is literally the best way to achieve consistency, as it writes to the memory with coherence as soon as the dirty bit in the cache changes from 0 to 1, but unlike the speed of the core (210) / cache (240) that moves at the speed of the normal bus (3010), the memory / memory controller (220) cannot keep up with the speed of expensive memory such as GDDR or HBM in the graphic device. The difference is usually 100 times, and at most 1000 times or more, which is the same as the difference in speed between the memory / memory controller (220) and the SSD / disk.

[0341] Therefore, in order to maintain consistency between the cache (240) and the memory in the L3 cache (243), if a dirty bit is generated and write-through is performed immediately, the core (210) / cache (240) cannot perform any work until the memory / memory controller (220) is completely written, and similarly, the bus (3010) is reduced to the speed of the bandwidth of the memory controller (220). For the above reasons, the write-through concept does not exist well in the conventional processor (200), and the write-back concept of updating the memory at regular intervals within the cache (240) / TLB (260) was introduced. However, in the example of FIG. 2a, two write concepts coexist.

[0342]

[0343] Specifically, in FIG. 2a, the variable X (171) inside the L3 cache (243) of the first computing device (201) points to an address value (3310) of the memory / memory controller (220) inside the computing device (201), and the variable Z (173) points to an address value (3310) of the memory controller (220) of the second computing device (202) through the chiplet connection (3020) that connects the two computing devices (200) across the bus (3010).

[0344] In the above example, the write operation to memory for both variables can be viewed as a conventional Write-Back, and this operation is performed by the cache (240) / TLB (260) or kernel (500) at an appropriate time within a range that does not damage the bandwidth of the cache (240) / bus (3010).

[0345] In addition, the part where the conventional Write-Through is used in the present invention is not when accessing each other's memory controllers (220) beyond the chiplet connection (3020), but when writing between the caches (240) between the two computing devices (200) through the chiplet connection (3020).

[0346]

[0347] FIG. 2b is a diagram showing the relationship between the bus page table (3300) and the TLB (260) / cache (240) that has the task of converting a conventional virtual paging address into a physical paging address.

[0348]

[0349] Referring to FIG. 2b, the concept of cache to cache write-through, which was not described in detail in FIG. 2a, and the operation between the bus table (3300) and the internal TLB (260) of the cache (240) are specifically shown.

[0350] The drawing of Fig. 2b shows that the Valid bit and Dirty bit, which were introduced to address the consistency issue between the cache and the memory / memory controller (220) described in Fig. 2a, in the L3 cache (243) of the LLC (250) of the second computing device (202), are located together with the internal cache lines / blocks.

[0351] Also, although it does not represent the entire cache blocks, for interaction with the upper cache, L2 (242), you can see that the remaining 36 bits, excluding 12 bits for 4KB paging offset from the 48 bits, which is an example of the internal virtual memory page address concept, are reduced to 18 bits in the L2 cache (242).

[0352] The reason why the virtual memory paging TAG (address identification) bits can be greatly reduced from 36 bits to 18 bits is that the L3 cache (243), which is LLC (250), has a shared cache (240) concept between multi-cores (210), so that the multi-cores (210) refer to completely different virtual memory spaces, and in order to distinguish them, all 48-bit virtual addresses must be distinguished. However, in the above example, since the L2 cache (242) and the L1 cache (241) are all separate for each core (210), there is no problem in distinguishing the context within the virtual paging address even with 18 bits.

[0353]

[0354] Specifically, the conventional X86 / 64 virtual paging address has a 48-bit value, and the upper 16 bits have meaning as control / protection bits between the operating system (500) and the processor (200), so it can ultimately be viewed as having a 64-bit virtual address.

[0355] Also, when the L3 cache (243) internally has a 48-bit virtual paging address, since such address values ​​are designed to be expandable / compatible with the conventional 32-bit virtual paging address, if you look briefly at its internal structure, having a paging unit of 4KB means that you only need to distinguish which line within the actual paper page of 4KB it points to, so 2^12 bits that can index 4KB in bit units are used as the paging offset in the virtual address value.

[0356] Since not all 4KB pages are loaded into the cache (240) in the actual L3 cache (243), the cache index / line value and the block offset value within it are each divided into 6 bits, and are used to point to the cache lines and cache blocks within the L3 cache (243) and to fetch or load only that portion.

[0357] In the 48-bit virtual memory paging address value, which is an example of the above, 36 bits excluding the 12-bit offset must be distinguished as the shared cache (240) within L3 (243), so all 36 bits must be used as a cache block tag (TAG) in the cache line / index as a tag value.

[0358] Additionally, each cache line / index typically has 1 bit of Valid and 1 bit of Dirty. This means that if we assume the L3 cache (243) is 4 MB as in the example above, there are 8192 64 cache blocks in the form of cache lines / indexes within the L3 (243).

[0359] This reason is related to the size of the cache block data per cache line / index, and when each cache line / index has 512 bits of cache block data in x86 / 64, since the 64-bit architecture in the example above can read and write in units of 64 bits (8 bytes), dividing 512 bits by 64 bits (block units) means 64 blocks, which means that each cache line / index has 0 to 63 cache blocks, as shown in Fig. 2b. This again means that 8192 cache lines / indexes are required in the example above of 4 MB of L3.

[0360] Also, the fact that the L3 cache (243) is 4MB actually means that it has a larger capacity inside the cache (240), excluding the 36 bits of the preceding virtual address TAG bits and the 2 bits of the Valid / Dirty bits.

[0361]

[0362] Referring to FIG. 2b, the cache blocks and cache lines / index elements within the conventional L3 cache (243) do not change significantly because they are architecturally compatible regardless of whether the bus page table (3300) exists or not, and when the conventional TLB (260) updates one cache block (one 8-byte) within the L3 cache (243) by the upper core (210) / L2 (242), the dirty bit where the corresponding cache line / cache block is located changes to 1, and it can be seen that the only part in FIG. 2b where the dirty bit is 1 is the part that requires consistency with the memory controller (220) in the same format as the conventional Write-Through or Write-Back.

[0363] In the above example, the part where the dirty bit is set to 1 points to the virtual paging address area, which is a part of the TLB (260) of FIG. 2b, and the address value is converted into a physical paging address by the TLB (260) and is returned to the virtual bus address (3310) of the bus page table (3300) of the above example.

[0364] The physical paging address value in the above TLB (260) is allocated by considering the bus address value (3310) when the kernel unit (500) allocates virtual memory paging, so the values ​​between the two match. Strictly speaking, since the precision between the two tables is different by 4KB and 4MB, it has an area as large as the bus paging precision, and the physical paging address value is between the virtual bus address (3310) areas, and can be seen as matching / existing (Hit).

[0365] In addition, if the physical memory paging address value exists (HIT) in the virtual bus address (3310) value area of ​​the bus table (3300), it is converted into an internal bus address (3320) value that matches the value in the bus page table (3300), and a bus write (Bus Write) is performed on an area of ​​the memory / memory controller (220) inside the processor (200) pointed to by the internal physical bus address value (3320), or a write-back is performed to the memory controller (220).

[0366] Moreover, a bus write request coming through a shared chiplet connection (3020) from another computing device (200) does not have a matching value within the precision area in the bus page (3300) as in FIG. 2b, but is queried by the bus master (3100) to the TLB (260) / L3 (243) internally within the chip (200) as expressed in FIG. 5a, so if there is a part of the TLB (260) / L3 (243) that matches the above bus write request, the area is changed from a physical memory paging address to a virtual memory paging address through the TLB (260), and then the cache block of the TAG bit that is referencing the virtual address value within the LLC (250), L3 (243), is replaced.

[0367] This replacement (Replace) can be viewed as a Cache to Cache through the bus (3010) / chiplet connection (3020) between the two computing devices (200) proposed in the present invention, or a Write-Through to the cache (240) address. Unlike the Write-Back to the memory controller (220), this can be performed without delay at high speed through the bus (3010) / chiplet connection (3020). Moreover, the act of replacing the L3 cache (243) of the LLC (250) can be viewed as a Cache-Flush (in contrast, Write-Back is a Cache-Clean act of writing a dirty cache (240) to the memory (220)) in which the values ​​are continuously overwritten and updated by the core (210) within the cache (240) before being Write-Back from the conventional core (210). However, while this Cache-Flush is performed only internally from the conventional core (210) to L3 (243), the above-mentioned Cache (240) Replace / Flush is performed in the opposite manner through Write-Through to the cache (240) of another computing device outside the chip (200) via the chiplet connection (3020), so it cannot be said to be completely the same as the Cache-Clean or Cache-Flush in the conventional concept.

[0368]

[0369] FIG. 2c is a drawing showing an example of a case in which the bus address table (3300) shown in FIG. 2a and FIG. 2b is not present.

[0370]

[0371] Referring to FIG. 2c, when looking at the interaction between the root complex (420), which is the two bus masters (3100) inside the two computing devices (200), and the bus slaves (3200) underneath it, it can be seen that, similar to the conventional NUMA / SMP problem, regardless of whether or not the bus page table (3300) exists, there still exists a problem due to memory write-back congestion between the two computing devices (200) due to the speed of the memory controller (220).

[0372]

[0373] To solve the above problem, referring to FIG. 2c, even in a conventional SMP / NUMA structure without a bus table (3300), the conventional NUMA problem can be alleviated through cache (240) to cache (240) write between the two.

[0374] This is because, even without a bus page table (3300), in an SMP symmetric structure, an address space (3320) is newly created between two computing devices (200) as L3 (243) / core (210) is demoted to a slave (3200) by the bus master (3100).

[0375] The biggest difference between the master (3100) and the slave (3200) in the conventional bus structure (3010) is that the bus master (3100) can exist as 0, but since it is usually the entity that dominates (reads / writes), the address value (3320) is not assigned to the entity that is being dominated (reads / writes), and when L3 (243) / core (210) is the master (3100), there is no address value (3320).

[0376] However, as can be seen in the above drawing 2c, since the core (210) / cache (240) is demoted to a slave (3200) of the bus (3010), an address value (3320) can also be assigned to the cache (240) / core (210), and thanks to this, cache-to-cache write-through between the two has become possible without a bus page table (3300).

[0377] The above method is an example, and as previously emphasized, even if there is a small bus (3010) cycle delay, it is not simply a matter of demoting the core (210) / cache (240) to a slave (3200), but rather the presence of a bus page table (3300) is beneficial, as will be shown in the following examples.

[0378] Of course, in the example shown in Fig. 2c above, the NUMA problem caused by the cache coherence problem has been alleviated, but the speed difference between the memory controller (220) and the L3 (243) / core (210) is still enormous, so when cache coherence is required, regardless of whether there is a bus page table (3300), the conventional Write-Through / Cache-Flush method is used between the two as a cache (240) to cache (240), and when that work is finished, the method of Write-Back / Cache-Clean to the memory (220) is chosen, just like in the conventional cache (240) / core (210).

[0379] However, it would be ideal if the present invention could cleanly solve the race condition problem between multiple caches (240) / buses (3010) / chiplet connections (3020), or the write-through or write-back selection problem, but the race condition problem is a problem that cannot be solved cheaply in hardware even between multi-cores (210) within a single processor (200) L3 (243) / LLC (250) shared cache (240). However, in the kernel (500), which is the operating system, the remaining 16 bits above the conventional 48 bits of the virtual address have already been presented as elements such as spin locks for resource preemption, together with various control / protection bits, to easily solve such problems in software.

[0380] In addition, with this cache (240) to cache (240) access alone, the problem of having to use only expensive graphics cards in areas requiring acceleration such as AI and the problem of having to use exclusive libraries such as CUDA are alleviated.

[0381] As the name suggests, the conventional CUDA attempted to solve the NUMA problem by simply viewing the graphics processors (200) as MMP (as opposed to the SMP mentioned above), and at some point, it maintained consistency by performing the Write-Back between the graphics card cache (240) / memory (220) at once as SMP.

[0382] However, unlike the example of Fig. 2c, if there is a bus table (3300), it is possible to easily move back and forth between the cache (240) and the memory (220) between computing devices (200) with different architectures, such as graphics cards, as a shared virtual bus address (3310) without a library such as Intel's OneAPI, so it is recommended to have Fig. 2c as an example.

[0383]

[0384] Referring back to FIG. 2c, it can be seen that even without the bus page table (3300), the core (210) / cache (240) is demoted to a slave (3200), so that cache (240) to cache (240) access is possible, but the conventional SMP symmetric structure between the two processors (200) is still required.

[0385] On the right side of Figure 2c, you can see the date bars representing the memory mapping, or physical bus address (3320) area. As explained above, the physical memory page addresses used by the software must be linear, and no other slave (3200) address area (3320), such as other peripheral devices or HALs, should be included in between, and the HAL bus address area (3320), such as such peripheral devices, must be outside of it.

[0386]

[0387] In addition, as can be seen with reference to the above-described FIG. 2c, thanks to this cache (240) to cache (240) access structure, there is no need to use an expensive and extremely large L4 cache (244) or HBM, etc., rather than an L3 cache (243) / LLC (250), as in the conventional IBM Power9 architecture, to address the speed degradation problem of the conventional memory / memory controller (220), and access to the memory (220) is similar to conventional storage devices in that local areas are read / written much more frequently than other areas.

[0388] Therefore, most of the problems can be alleviated by using the Write-Through method of the cache (240) to cache (240) method, and using an L4 cache (244) / HBM / GDDR, etc. that is smaller than the L3 cache (243) with the LRU Write cache presented in FIG. 2c only when writing back to the memory (220). However, FIG. 2c is only for an example without the bus page table (3300), and if there is no actual bus table (3300), it is difficult to connect more than two computing devices with a chiplet connection (3020) because Symmetric; SMP still exists. Therefore, it is much more advantageous to use the bus page table (3300) as recommended in this patent.

[0389]

[0390] In conclusion, referring to FIG. 2c, when the X variable (171) as an example in the internal L3 (243) of the first computing device (201) is updated (dirt) by the core (210) of the first computing device (201) or the upper L2 cache (242), the bus master (3100) of the first computing device (201) checks the dirt status of the L3 cache (243) area and then propagates such bus write through the chiplet connection (3020) even if it is not the address value (3320) of its own memory controller (220) or internal slave (3200). The bus write propagated through the chiplet connection (3020) reaches the bus master (3100) of the second computing device (202) through the chiplet connection (3020) of the second computing device (202), and the bus master (3100) likewise knows that the address value is not its own address value, but since the TLB (260) / L3 (243) of the second computing device (202) may be referencing it, it queries the TLB (260), and since the address value matches (HIT) the TLB preview address value or the TAG value for distinguishing the virtual address inside the L3 (243), it can be seen that one part of the L3 (243) cache block inside the second computing device (202) has been Cache-Flushed and replaced (Replaced) through Write-Through by the bus master (3100).

[0391] In addition, variable X (171), which exchanges contents between the first computing device (201) and the second computing device (202) L3 (243) at the chiplet connection (3020) / bus (3010) speed through Write-Through, causes the value of X (171) to be Write-Back / Cache-Cleaned to the memory through variable Z (173), which points to an address value (3320) of the memory / memory controller (220) inside the first computing device (201), similar to the conventional method, when the operation between the two is finished.

[0392] As described above, this Write-Back behavior reduces the speed of the bus (3010) to the speed of the memory controller (220), so as suggested above, before the memory controller (220) writes to the memory, a small L4 cache (244) with an LRU for write can be used with SRAM / HBM / GDDR to mitigate the overall bus (3010) speed reduction.

[0393]

[0394] FIG. 3 is an exemplary diagram showing the operation of the chip internal bus (3010), chiplet connection (3020), and TLB (260) inside the cache (240) through the open interconnect link (400) presented in FIGS. 1 and 2.

[0395]

[0396] Referring to FIG. 3, as shown in FIG. 2c, when there is no bus page table (3300), it is shown how the memory physical page viewed by conventional software is mapped to the bus internal physical address (3320), and similarly, when there is a bus page table (3300), it is shown how it is mapped to the virtual bus address (3310) area proposed by the present invention.

[0397] However, the linearity and nonlinearity in the mapping between memory pages, virtual pages, and physical pages, or virtual buses and internal buses, as shown in FIG. 3, are complementary. Unlike what is shown in FIG. 3, if a virtual memory page is linear in a conventional memory address, the physical memory page on the opposite side appears nonlinear, and conversely, if the memory physical page is linear, the virtual memory page appears nonlinear.

[0398] Also, this is the same in the bus page address concept. If the virtual bus address (3310) is linear, the internal bus address (3320) appears nonlinear, and conversely, if the virtual bus address (3310) is viewed nonlinearly, the internal bus address (3320) appears linear.

[0399] Moreover, as explained in FIGS. 1 and 2, when there is no bus page table (3300), the slave elements in the internal physical bus address can have a size of at least 4 KB to 64 KB, whereas when there is a bus page table (3300), the precision of the bus page, which is 4 MB in the example, must be the minimum size due to fragmentation within the bus page.

[0400] The part to be noted in the above bus page table (3300) section is that when a virtual bus page address (3310) is introduced, a physical memory page can have an address area or possibility such as 2×48 or 2×64, just like a conventional memory virtual page address. This is because the physical address limit of the conventional physical memory page address concept has been transferred to the internal physical bus address (3320) thanks to the bus paging table (3300).

[0401] In addition, it can be seen that the actual limit of the address of the internal physical bus (3310) is that, in the case of the above-mentioned example of a 16 GB memory controller (220), 2 power 34 actual bus address lines are physically required. These internal physical bus (3310) address lines must be wired with at least 34 physical address lines regardless of whether a bus page table (3300) exists or not when the above-mentioned 16 GB memory controller (220) is used as a bus slave (3200). However, the bus (3010) data lines are always the same as the above-mentioned example of 64 bits.

[0402] Of course, in the case of FIGS. 1 and 2, in addition to the 16 GB memory controller (220), there are three more remaining bus slaves (3200) as 4 MB bus address elements, so the actual bus address line is exactly 2 35, requiring one more line.

[0403]

[0404] Referring to FIG. 3, it can be seen that the bus table (3300) is organically combined with the TLB (260) inside the cache (240) and the external chiplet connection (3020) and operates. As mentioned in FIGS. 1 and 2, although not shown abstractly in the drawing, the bus page table (3300) physically located inside the bus master (3100) causes the bus master (3100) to traverse the bus page table (3300) as a bus table controller (3120) when a bus read / write (3210) signal or a bus stream (3230) signal, which will be described in more detail later in FIG. 5, enters the bus master (3100) through the internal bus (3010) / chiplet connection (3020).

[0405] At this time, the input bus signal is a virtual bus address (3310), and is sequentially compared with the values ​​of the virtual bus address (3310) of the bus page table (3300), and is sequentially traversed just as it normally traverses within the cache (240) / TLB (260). As will be described in detail in FIG. 3b below, although the virtual bus address value of the bus page table (3300) is conceptually described as having the entire 64-bit virtual address value as an example, due to the various physical limitations mentioned above, the lower 22-bit value that can internally distinguish 4 MB of the bus page precision (Granularity) may be discarded as an offset, and only the upper 42-bit / 26-bit value obtained by subtracting 22 bits from the remaining 64-bit / 48-bit virtual address (3310) value may be used.

[0406] If there is an entry (3330) that matches the upper bit of the virtual address value (3310) in the bus page table (3300) of the above example, it exists (HIT) inside the bus page table (3300), which means that the subject of the bus read / write exists internally (Locally) among the internal slaves (3200) of the chip (200) of the above example.

[0407] If there is a hit in the bus page table (3300), the internal bus master (3100) of the chip (200) operates the internal bus (3010) to respond to the bus signal coming from the internal TLB (260) / cache (240) status checker (3160) or another external computing device (200) via the chiplet connection (3020), converts the virtual bus address value (3310) of the bus page table (3300) into an internal bus address value (3320) corresponding to the internal bus slave (3200), and then performs the bus read / write operation on the bus slave (3200) address value by delegating it to the bus master (3100).

[0408] In addition, although the bus page table (3300) in the above-described Figure 3 is traversed in its entirety, if there is no value matching the virtual bus address value (3310), the traversal is terminated. In this case, it means that the bus request coming through the chiplet connection (3020) within the chip (200) or the area that the TLB (260) within itself requests to update / read does not physically exist in the internal bus slave (3200) of the actual chip (200). This means that the bus address value (3310) of the other chip is referenced, so even if the traversal within the bus page table (3300) is finished, the TLB (260) / cache (240) is queried while traversing the bus page table (3300) to see if the bus address value (3310) is referenced internally within the chip (200). If the TLB (260) / cache (240) references the virtual bus address value (3310), a cache flush or write-through occurs immediately, which is the same as the cache to cache write-through described above. In addition, it can be seen that the virtual bus paging address value (3310) here acts as a global / virtual bus address (3310) between all chips (200) via the chiplet connection (3020).

[0409] Conversely, when an update is requested to the bus master (3100) in the TLB (260) / cache (260) inside the chip (200), even if the virtual address value (3310) being referenced does not exist inside the bus page table (3300), it may be referenced as a global / virtual bus address (3310) outside the chip, so while traversing the bus page table (3300), a bus signal for reading and writing the virtual bus address (3310) is also transmitted to the outside through the chiplet connection (3020).

[0410]

[0411] FIG. 3a is a diagram schematically illustrating the operation of the bus page table (3300) in FIG. 3 with the TLB (260) / cache (240) and bus address conversion (4000) within the chip (200).

[0412]

[0413] Referring to FIG. 3a, although the internals of the bus master (3100) are depicted in more detail in FIG. 5a, when looking at the flow from the perspective of the bus table (3300) and the TLB (260) / cache (240), the flow represented by this diagram is for each bus (3010) clock within the chip (200), and the bus master (3100) within the chip (200) typically has a bus (3010) clock that is synchronized to the core (210) / cache (240) speed, either generated within the bus master (3100) or brought in from the outside.

[0414] At each bus clock, the busmaster (3100) checks for an incoming busstream (3230) signal requesting a bus read / write, as can be seen in more detail in Figure 5a. If a bus stream input (3231) comes in from a chiplet connection (3020) or an interconnect link (400) inside the bus master (3100), the bus table (3300) inside the bus master (3100) is traversed according to the bus request stream (3230), as shown in part 4100 of FIG. 3a. As mentioned in FIG. 3, there are cases where a match (Hit) is made inside the bus table (3300) and reading / writing of the local bus (3010) inside the chip (200) is required. However, as shown in FIG. 2, there may also be cases where the bus stream (3230) request references the cache (240) / bus slave (3200) element outside the chip (200) as a virtual / global bus address (3310), so the bus slave (3200) element inside the chip (200) A query is also made to the TLB (260) / cache (240) by inducing a bus read / write. Of course, as will be described later in Fig. 5a, the procedure for checking the TLB (260) / cache (240) inside the chip (200) can be separately provided as a bus stream input (3231) like the action from the chiplet connection (3020) or interconnect link (400) to reduce the internal bus (3010) cycles, but this is not particularly recommended.

[0415] When step 4100 of FIG. 3a is performed, as mentioned above, it branches into step 4200 of traversing the bus table (3300) inside the bus master (3100) and step traversing the TLB (260) / cache (240) inside the chip (200). If there is a matching value (Hit) in the virtual bus address (3310) inside the bus page table (3300) in step 4200, the bus master (3100) converts the matching virtual bus address (3310) inside the bus page table (3300) into an internal physical bus address (3320) value corresponding to the corresponding virtual bus address (3310) in step 4300 of FIG. 3a, and then, in step 4400, the bus master (3100) reads / writes the internal bus slave (3200) as a bus controller, and the process ends. In that case, the bus master (3100) repeats the above schematic flow for each bus clock, thereby organically operating with the bus master (3100), the bus page table (3300), and its own TLB (260) / cache (240) within the chip.

[0416] However, if there is no table entry (3330) that matches (Hit) even after traversing the bus table (3300) in the above 4200 section, it is asynchronously checked whether the TLB (260) / cache (240) that is branched in the same way as in the above 4100 section is referencing the virtual / global bus address (3310) that matches the bus request stream. Of course, this action does not necessarily have to be asynchronous, but for the overall performance of the bus (3010), it is recommended to allocate asynchronously as in the bus allocator (3130) section of FIG. 5a.

[0417] In addition, the speed difference between the above bus page table (3300) traversal and the TLB (260) / cache (240) internal traversal is mostly determined by how many entries each table has to traverse, as mentioned in FIGS. 2A and 2B. If the number of TLB (260) lookup item entries in the cache is less than a few hundred due to physical limitations, the TLB (260) / cache (240) HIT response may be faster than the 4200 steps of the bus table (3300) traversal. Since this part may significantly vary depending on the physical circumstances inside each processor (200) or the table precision, there may be various modifications by those skilled in the art.

[0418] However, since most of the branches in step 4100 of the above-mentioned FIG. 3a are referenced / exist in one of the two tables, it is highly likely that one of the two branches will take a route, and if there is a part in the TLB (260) / cache (240) that matches (Hits) the virtual / global bus address (3310) of the bus request stream (3230) as an internal physical memory page address, as mentioned in FIGS. 2 and 3, a read / write operation is performed on the cache line / cache block that references the virtual bus address (3310) as a physical memory paging address within the LLC (250) and cache (240) of step 4500 of FIG. 3a. Of course, most of them are likely to be writes, which are the same as the Cache to Cache Write-Through through the chiplet connection (3020) mentioned in the above-mentioned FIGS. 2 and 3, and the Cache-Flush by the bus master (3100).

[0419] Of course, in the case of branching at step 4100 of FIG. 3A, there may be cases where it does not exist (Hit) in the bus page table (3300) and is not referenced in the TLB (260) / cache (240). In such cases, since the chiplet connection (3020) directly copies / propagates the internal signals of the bus (3010) between other chips (200), such a bus request is not performed within the chip (200) of the example of FIG. 3A, but is performed between other chips (200).

[0420]

[0421] Referring to FIG. 3a, if there is no bus request through the bus stream input (3231) at each bus clock, in such a case, the bus master (3100) inside the chip (200) must perform the action delegated to the TLB (260) / cache (240) that had the authority of the conventional master (3100), and if there is no bus request from the interconnect link (400) or chiplet connection (3020), it performs the action of preempting the internal bus (3010).

[0422] Specifically, in FIG. 3A, when there is no bus request, step 4600 is performed, and as mentioned above, the logic (3160) for checking the TLB (260) / cache (240) inside the chip (200) is also shown in FIG. 5A, but it is recommended that the internal table controller (3120) of the bus master (3100) have it. Similar to the prior art, this part can also be received in the form of a bus stream input (3231), such as an interconnect root complex (420) or a chiplet connection (3020). Of course, it is not recommended to receive it as a bus stream input (3231) in order to reduce the number of bus cycles, as it can be exploited.

[0423] In the step 4600 of the above-described FIG. 3a, if the TLB (260) / cache (240) status checker (3160) determines that the TLB (260) / cache (240) inside the chip (200) is in a state where the dirty bit is 1 and requires an update, or in an invalid state requiring a bus read, since the physical memory paging address referenced by the TLB (260) / cache (240) also functions as a virtual / global bus address (3310), as in the step 4100, in the step 4700, the bus request is copied / propagated in the form of a bus stream (3230) through the chiplet connection (3020) to the outside of the chip (200), and thereafter, the step 4200 of FIG. 3a, which is a process of traversing the bus table (3300), and the step traversing its own TLB (260) / cache (240), are performed.

[0424] In this case, both tables (more precisely, they are propagated to other chips via chiplet connection) are traversed synchronously / asynchronously as in the above 4100 step, and since it is the same as traversing the TLB (260) / cache (240) according to its own request, it can traverse the TLB (260) / cache (240) only once and then update / read directly to the above Dirt / Invalid area.

[0425] Of course, as described in the above 4500 step, most of them are write operations, and in the 4600 flow of Fig. 3a, when the TLB (260) / cache (240) exists (Hit) and writes in step 4500, it is the same as writing through itself / self-flushing the cache, so the result is the same whether or not such an operation is performed inside the TLB (260) / cache (240).

[0426] This is because, when performing a cache-to-cache Write-Through / Cache-Flush, the cache (240) of one chip (200) must be self-referenced, so that sophisticated processing may be required internally in the TLB (260) / cache (240) as in the past. Of course, even if such processing is not performed internally, the result is the same, and only a few more bus cycles are required. Similarly, the present invention omits such internal TLB (260) / cache (240) actions by describing the actions of the bus table (3300) and the bus master (3100).

[0427]

[0428] Referring to the process after step 4700 of FIG. 3a, in addition to step 4500, which describes the Write-Through / Cache-Flush, which is the TLB (260) / cache (240) self-reference, in the remaining branch, step 4600, the flow of step 4300 can be seen in which the stream (3230) for which the TLB (260) / cache (240) status checker (3160) made a bus request exists (Hit) in the bus page table (3300).

[0429] If the flow proceeds from step 4600 of FIG. 3A to step 4300, it is the same as the virtual bus address (3310) that the TLB (260) / cache (240) wants to update (Write) / read exists as a bus slave (3200) by the bus table (3300) inside the chip (200), and such a flow is similar to an act of Write-Back / Cache-Cleaning from the TLB (260) / cache (240) to a conventional memory controller (220), or reading the internal bus slave (3200) of its own chip (200) as an internal physical bus address (3320).

[0430] Of course, it is also possible that both branches split at step 4700 may not exist (Hit), which also means that the TLB (260) / cache (240) inside the chip is referencing a bus slave (3200) of another chip outside the chip (200) as a virtual / global address (3310).

[0431]

[0432] FIG. 3b is a diagram illustrating the procedure for setting the initialization process and bus address paging (3300) when the chip (200) mentioned in FIGS. 1 and 2 is internally powered on, and the method for sharing the safe bus address table (3300) with the kernel (500) after the nonlinear protection mode / Protection SWAP (5000).

[0433]

[0434] Referring to FIG. 3b, in the initial non-protected mode, as described in FIGS. 1 and 2 of the present invention, although the virtual bus address (3310) has the same linear value as the internal physical bus address value (3320), if the remaining SET bits (3340) except for the virtual bus address (3310) and the internal physical bus address (3320) in each entry (3330) of the bus page table (3300) are not set to 1, even if the bus slaves (3200) inside the chip (200) know the virtual bus address value (3310) and request read / write to the bus master (3100), the bus master (3100) can never access the address value (3320) and cannot even use it.

[0435] What this means is that, in the examples of FIGS. 1 and 2, the GPIO (314) inside the chip, the Peripherals such as GPS, I2C, the memory controller (220), the modem (320), and even the buttons of the smartphone are not usable at all unless the kernel unit (500) initializes and sets the entry (3330) in the bus address table (3300) to SET (3340) Enable.

[0436] This is intentionally disabled for hardware reliability and verifiability, as it is similar to not being able to turn on a smartphone with a button physically wired (230) to the smartphone application processor (200). Instead, it is suggested that the power / volume button GPIO (314) be transferred to a USB-C power chip, or a display driver (310) combined with Apple's Force Touch.

[0437] Specifically, as step 5100 of FIG. 3b, the touchscreen / force touch display driver (310) connected to the easily replaceable interconnect link (400) unrelated to the application processor (200), or the operation of connecting USB-C, or pressing the GPIO (314) button transferred to the power chip, sends a Power On / Wake On packet to the root complex (420) of the processor (200) as an interconnect device (300) / endpoint (430), and the application processor (200) of the smartphone that received it operates only then.

[0438] In addition, the Wake On / Power On packet via Force Touch is valid even when the smartphone does not have a physical button or even a connector such as Type-C, and the display driver (310) and Force Touch, etc. are connected to the application processor (200) through a verifiably open interconnect root complex (420).

[0439] However, the processor that received the Power On / Wake On packet cannot read the firmware or kernel (500) from the SSD or flash of the interconnect endpoint (430) by itself because the cache (240) of the core (210) and all lower bus slave (3200) elements do not have the SET bit (3340) set in the bus table (3300).

[0440]

[0441] In a conventional processor (200), the initial booting from SSD or flash or the reading of the kernel (500) was possible because the processor (200) always started from a hardware-provided starting address. Even if KASLR (Kernel Address Space Layout Randomization), which randomizes even the kernel address area in paging units in the kernel section (500) to be booted in the future, is used, it is disabled because the unreliable processor (200) already starts by hooking from a hardware-fixed starting address.

[0442] For the above reasons, the promised start address or the initialization of the processor (200) must not be done by hardware, and must be relaxed together with the interaction of the kernel (500). Therefore, as mentioned in FIGS. 1 and 2, rather than simply sending a Wake On / Power On packet through the interconnect root complex (420) opened to the processor (200) in the beginning, as step 5200 of FIG. 3b, a verifiable and lightweight kernel (500) of several hundred KB must be loaded together with the force touch display driver or the USB-C connector power management unit to which the GPIO has been transferred, and sent to the interconnect root complex (420).

[0443] In addition, a small kernel of several hundred KB at the firmware level is sent together with N (3700) random numbers generated from the display driver or power management interconnect endpoint (430), and is used as a random bus address (3310) that can be set by writing through the bus table (3300) of the internal bus master (3100) of the processor (200), as will be described in detail in FIG. 3c based on the N (3700) random numbers.

[0444] The initial kernel part (500) that operates only within the above-described cache (240) is intended to cope with conventional firmware, so there is no need to map all addresses within the processor (200) bus (3010) to operate it, and the bus master (3100) within the processor (200) is set in the bus page table (3300) so that only 4 MB (1 entry), which is the size of the example L3 cache (243) / LLC (250), can be used initially.

[0445] In addition, as in step 5300 of FIG. 3b, before the initial Protection SWAP (5000) / linear unprotected mode, the virtual bus address (3310) area of ​​the cache (240) linearly set to be the same as the internal physical bus address value (3320) is transparently visible and executed for requests from the core (210) / L3 cache (243) as if there were no bus page table (3300). Therefore, in step 5300 of FIG. 3b, only the part of the bus page table (3300) that is SET (3340) enabled can be circulated. In addition, after completing the initial bootstrap process of Fig. 3b and when the nonlinear protection mode / Protection SWAP (5000) is applied, even if the above areas are set to SET (3340) Enable, in order to access the cache (240) area or other bus areas, the virtual bus address (3310) values ​​of the arbitrarily set table entry (3330) must be known.

[0446] Likewise, even before the nonlinear protection mode / Protection SWAP, the bus virtual address (3310) value is set linearly to the internal bus address (3320), but since the SET bit (3340) is set to 0 by the bus table (3300) in the area other than the cache (240), the lightweight kernel part (500) existing only in the cache (240) cannot access and operate other slave (3200) elements within the bus (3010). The same applies to the hardware core (210) / L3 cache (243) itself.

[0447] As step 5400 of FIG. 3b, multiple cores (210) are cycled for obfuscation, and as step 5500 of FIG. 3b, only a core (210) that succeeds in Write-Through on a random virtual page first according to a race condition among multi-cores (210) can access it only when the bus master (3100) sets the internal bus table (3300).

[0448] However, as explained above, initially, the bus page (3300) is initially mapped inside the interconnect root complex (420) / endpoint (430) which is the bus master (3100), and in the case of the above example, only one bus page (3300) which is the cache (240) area is mapped. In addition, since the lightweight kernel part (500) at the firmware level is smaller than the L3 cache (243) / LLC (250) even though the memory (220) space has not been allocated yet, the TLB (260) / cache (240) miss does not occur even without the memory, and as the 5300 step of the above FIG. 3b, the core (210) and the bus master (3100) are repeatedly circulated only within the cache (240) address area.

[0449] In addition, since only the bus master (3100), which is delegated to read / write by the conventional core (210) / L3 (243), knows the virtual bus address of the L3 cache (243) / LLC (250), it is necessary to know the bus virtual address value (3310) to access it.

[0450] In step 5400 of FIG. 3b, depending on the race condition between the multi-cores (210), one of the virtual memory page creation successes first, which is step 5500 of FIG. 3b, randomly writes through random virtual bus addresses (3310) to be recorded in the bus page table (3300) using the N (3700) random numbers as address values, which is step 5600 of FIG. 3b, and succeeds in Protection SWAP (5000) / reset.

[0451] Specifically, in step 5600 of FIG. 3b, the address where the arbitrary virtual bus table (3300) is located in the form of a memory virtual page entry, the N random numbers (3700) are used as bus address values ​​(3310) to cause a bus write. In this way, it can be said that the Protection SWAP (5000) / bus page table (3300) reset, which fills all of the bus page tables (3300) physically in the bus master (3100) or virtually in the kernel unit (500) memory page entries with arbitrary values, in step 5700 of FIG. 3b, has been completely performed. After going through the above process, all of the bus page addresses (3310) appear nonlinear and are mapped / set to unpredictable values. In addition, it can also be said that the protection mode has finally been entered.

[0452] Step 5800 of FIG. 3b is the last step of the schematic flow, and since the bus table (3300) has already been physically non-linearly reset / Protection SWAP (5000) in step 5700, all upper bits except the lower 22 bits that accommodate the bus precision of 4 MB in the TLB (260) / cache (240) are made invalid, and after going through the process of correctly calculating the memory physical paging address again by the kernel unit (500), the kernel (500) or application (900) can operate normally even within the protection mode.

[0453]

[0454] The reason for having this obfuscation or randomization process by race condition is to make it extremely difficult to verify the inside of the processor core (210) or other hardware intervention elements.

[0455] In the above Protection SWAP (5000) / Bus Table (3300) reset process, the entire bus page table (3300) can be set to an arbitrary value. However, if not, and some areas are not set to SET (3340) Enable, other remaining memory areas or areas such as Pheriperals are still not accessible or usable because they are not set within the bus master (3100). In addition, in order to share the virtual bus address (3310) values ​​filled with these random numbers with the kernel (500), which is the operating system, after booting in the future, while creating a virtual memory address table / directory in the kernel (500) area, N (3700) random random number value entries are recorded by dividing the bus table area (3330) into 4KB units, which is an example of a virtual memory paging size, and the actual mapped random number values ​​may or may not have a Protection SWAP (5000) process so that the hardware (210 / 240) / software (900), not the kernel (500), cannot predict them.

[0456] Moreover, in the case where the Protection SWAP (5000) / bus table reset is partially re-established, all upper address values ​​except for the example 22 bits of bus precision in the TLB (260) / cache (240) referencing the area become invalid and must be recalculated as step 5800 of the above-mentioned Fig. 3b. In addition, as a matter of course, in the above-described Fig. 3b, when the bus page table (3300) is divided into N (3700) and the bus virtual address value (3310) is located, after the Protection SWAP (5000) / Bus table (3300) is reset (Write-Through), the N (3700) random random values ​​given by the interconnect endpoint (430) at step 5200 of the above-described Fig. 3b also change each time accordingly, since the bus table (3300) has changed nonlinearly.

[0457] The above resettable bus page table (3300) area is initially partially set and the table (3300) gradually grows larger. Although the virtual bus address (3310) of the entry (3330) can be rewritten several times through Protection SWAP (5000) for integrity, the memory pages referenced by the TLB (260) must be recalculated as mentioned, which causes a slight performance degradation.

[0458]

[0459] In conclusion, the size of the above bus page table (3300) is different from the complex memory paging table typically used in the kernel (500), and there is only one per chiplet connection (3020) / bus (3010) of the processor (200), and its capacity is also very small, at approximately 21.5KB in the above example. For example, in the case of separately operating multiple chiplet connections (3020) / buses (3010) through an interconnect link (400) such as CXL in one computing node and one operating system kernel unit (500), each processor (200), i.e., each separate chiplet connection (3020) / bus (3010), has a chiplet connection (3020) / bus (3010) address space that is mutually isolated by the interconnect link (400) such as CXL. This means that each chiplet connection (3020) / bus (3010) has a separate bus address mapping table (3300) per operating system (500).

[0460] However, in the case where the open chiplet connection (3020) such as the UCIe is continuously connected / expanded, it is as if the bus (3010) is shared internally within the chip by the bus table (3300), so the bus table (3300) still has a separate 21.5KB table (3300) of the example internally per chip (200), but overall, the table (3300) shares the same virtual bus address (3310) space as one bus table (3300) between the operating system kernel part (500) and the shared chiplet connection (3020) / bus (3010).

[0461] The reason why the above bus address table (3300) is recorded in an area of ​​an arbitrary memory (220) and shared with the kernel unit (500), which is the operating system, is to initialize and allocate / manage peripherals such as GPIO (314) or SPI, modem chip (320), memory (220), etc. inside the processor (200) chip only by the kernel unit (500). This bus address paging table (3300) must not be arbitrarily set even by the open interconnect root complex (420), which is the bus master (3100), and in the Protection SWAP (5000) / nonlinear protection mode, it can be set only by the operating system kernel unit (500), which knows the location of the table (3300) as N (3700) arbitrary memory virtual page entries.

[0462] In addition, by going through a gradual bus table (3300) bootstrap / reset process through the interaction of the kernel (500) and the interconnect root complex (420) / endpoint (430) which is an obfuscated bus master (3100), rather than the hardware (200) itself, even the internal area of ​​the chip (200) of the untrusted hardware becomes a stepping stone that provides the kernel (500) with the possibility of control and verification.

[0463] The bus page mapping table (3300) inside the above bus master (3100) and the bus page table (3300) that is written through by the bus master (3100) trigger / latch as an address determined by N random numbers that cannot be specified by hardware are set to a virtual bus page address (3310) of about 21.5 KB in the example in a non-linear protection mode and made accessible as the kernel (500) gradually bootstraps the internal bus (3010) of the processor (200).

[0464]

[0465] More specifically, in the initial linear unprotected mode, the various control / operations generated by repeatedly circulating the core (210) and the cache (240) are sent to the bus master (3100), which is the interconnect root complex (420), by the kernel (500) of the cache (240), along with CXL.io (460) or an interconnect enumeration request, and the interconnect link unit (400) inside the chip sends an initial enumeration command to all interconnect devices (300) / endpoints (430) connected to the bus (3010) delegated from the L3 cache (243) as the CXL or interconnect root complex (420).

[0466] However, in order for the core (210) / L3 cache (243) to send an interconnect link (400) packet such as CXL to the bus master (3100) which is the interconnect root complex (420), a certain area (3330) of the initial bus address table (3300) must be allocated / set (SET) with a bus address in the form of a bus slave (3200) such as the conventional CXL / interconnect root complex (420) to trigger the interconnect link (400) Enumeration packet, etc.

[0467] However, as will be described later in FIGS. 5A and 5B, if the interconnect (420 / 430) is viewed as Stream In (3231) / Stream Out (3232) rather than address-based read / write, it is not necessary to assign a bus address as in the conventional format. By setting any area within the L3 cache (243) / LLC (250) as a bus intercept (6000) trigger area with the above alternative, it is possible to induce the creation of a packet in the CXL / interconnect root complex (420). The reason for using this bus stream (3200) method is that, as will be described later in FIGS. 4A and 5B, it is also suitable for remotely triggering a bus intercept (6000) between the interconnect link (400), the chiplet connection (3020), and the internal buses (3010).

[0468] In addition, since the initial unprotected mode can only use the L3 cache (243) in a very limited manner, just like the lightweight kernel part (500), it is difficult to exchange a large amount of bulk data, unlike in the protected mode where a memory area of ​​several hundred MB or more (220) is set as a mapping coherence area of ​​an interconnect link (400) such as CXL. However, in the initial linear unprotected mode, only very basic commands of an interconnect link (400) such as CXL need to be exchanged.

[0469] After the enumeration between the kernel (500) and the interconnect endpoint (430) devices (300) within the cache (240) is completed, the kernel (500) sends a command to read a kernel that can be booted as an operating system from an SSD or flash device through the interconnect root complex (420).

[0470] However, the above kernel and boot elements are loaded through the bus address (3310) that is non-linearly set in the memory controller (220) area by the interconnect root complex (420) and the bus master (3100) like the conventional DMA, and as will be described in detail later in FIGS. 5, the bus master (3100) of the endpoint (430) already operates its internal bus table (3300) as a lookup table address like DMA or IOMMU (DTLB) due to deadlock between the two bus masters (3100), so when loading the kernel image into the memory (220), it is written appropriately to the correct location.

[0471] The bus master (3100), which is the interconnect root complex (420), loads the kernel image of SSD or flash into the memory (220) in small units of bus paging, like the conventional DMA / IOMMU (DTLB), and then the lightweight kernel part at the firmware level can continue the remaining bootstrap with only the kernel (500) loaded into the memory (220) while inheriting the memory virtual paging table including the bus table (3300) of the lightweight kernel part (500) at the firmware level of the previous FIG. 3b through a technique such as the conventional KEXEC.

[0472]

[0473] FIG. 3c is a diagram showing various calculations derived from a virtual bus address (3310) in the bus paging table (3300) of FIG. 3, FIG. 3a, and FIG. 3b, and Protection SWAP (5000), which is a table sharing / protection technique with an operating system (500) using the bus page table (3300) of FIG. 3b.

[0474]

[0475] In the bootstrap process of the above-described FIG. 3b, the software kernel (500) cannot be aware of the bus master (3100), and just as the conventional kernel (500) recognizes a hardware interrupt or DMA, when the core (210) / L3 cache (243) delegates reading / writing of a specific bus address (3310) area to the master (3100), the bus master (3100) internally updates the address value through a hardware latch / trigger within the bus master (3100) based on the N (3700) random number values ​​set in step 5200 of FIG. 3b, and the internal table (3300) thereof is also updated (Write-Through) at the same time.

[0476] However, the bus master (3100) is the first entity to receive the bus table (3300) values ​​recorded in N (3700) parts through the interconnect root complex (420), so it naturally knows the values. However, the applications (900) / processes circulating in the core (210) / L3 cache (243) in the nonlinear protection mode do not know the nonlinear bus page address (3310), and even they are randomized through protection swap, making it difficult to operate normally.

[0477] To avoid such parts, in step 5800 of the above-described Fig. 3b, when the injected kernel (500) allocates a memory virtual page to the application (900), the memory physical page address must always be allocated by considering the bus virtual address (3310), and when the Protection SWAP (5000) / Bus Page Table (3300) is reset in the middle, the physical paging addresses are invalidated, except for the valid 22-bit virtual bus address precision in the above-described example, and then recalculated / validated.

[0478]

[0479] Referring to FIG. 3c, when the Protection SWAP (5000) / bus page table (3300) is reset, it can be seen why the lower 22 bits that accommodate a bus precision of 4 MB are still valid and why the upper bits excluding the lower 22 bits are invalid.

[0480] The bus page virtual address (3310) is derived from the virtual page address, which is a conventional memory page address, and in order for the bus page concept to avoid performance degradation, as mentioned in the various drawings above, the conventional memory page concept must be appropriately applied.

[0481] Referring to the 48-bit part of the conventional x86 / 64 virtual address concept, which is an example in Fig. 3c, we can see that the conventional memory page address is also divided into several parts, and when it has a typical memory page precision of 4KB, we can see that a total of 12 bits are required to distinguish page lines / blocks within the page. Therefore, if we subtract the lower 12 bits, which are the memory page offset, from the 48-bit virtual address concept, the upper 36 bits remain, which can be viewed as the 36-bit virtual page number. As can be seen in Fig. 3c, this 36-bit virtual page number is changed into a 36-bit physical page number by the TLB (260), and the 12-bit lower offset is converted into a memory virtual page value or a physical page value, but the lower offset value is always the same. Therefore, in the 48-bit memory virtual page address concept, the lower 12 bits always have the same value even after address conversion, and only the upper 36 bits are used as a TAG value for processing within the TLB (260) or cache (240).

[0482] In addition, when the kernel (500), which is the operating system, allocates / creates a conventional memory page, the 12 bits, which are the lower offsets in the table / directory, are discarded, and only the 36 bits, which are the upper page numbers, are kept 1:1 within the table. This concept is the same in the bus page table, and as continuously mentioned from the above Fig. 1, in the 64-bit bus virtual address system, which is an example, the value that the actual bus page table (3300) must have is not all 64 bits, but in reality, it is 42 bits, which is 64 bits minus 22 bits, which is the bus page precision, and in the above 48-bit virtual address system, only the upper 26 bits, which is 48 bits minus 22 bits, can be kept as the bus page number. The upper 26 bits of the above bus page address value are values ​​that hit / correspond to the bus virtual page address (3310) inside the actual bus master (3100). When the bus master (3100) traverses the bus page table (3300), the upper values ​​are compared one by one as table entries (3330).

[0483] Also, as can be seen from this bus page address system, if the Protection SWAP (5000) / Bus Table (3300) is reset as a protection mode in Fig. 3b, only the upper 26 bits of the virtual bus page number in the above example are invalidated, so even in the memory page address concept inside the conventional TLB (260) / cache (240), not only the 12-bit offset in the above example, but also the 9-bit value of the upper PTE is still smaller than the 22-bit bus page precision / offset when combined with the 12-bit value, so it is still valid.

[0484] However, in the conventional memory page address concept, the PDE 9 bits, PDPE 9 bits, and L4 9 bits, which are higher bits than the PTE, are all invalidated. Accordingly, even if the bus precision is not the example 4 MB in FIG. 3c but a completely different value, when resetting the Protection SWAP (5000) / Bus Table (3300), it can be seen through FIG. 3c how far it is valid and from where it must be completely recalculated (invalidated) in the conventional memory page concept.

[0485]

[0486] Referring to FIG. 3c, when the conventional memory virtual page address value is converted into a memory physical page address value in the TLB (260), it can be seen that the memory physical page address value is first matched as a virtual bus address value (3310) in the bus page table (3300).

[0487] In addition, it can be seen that the virtual bus page address value (3310) matched in the bus page table (3300) is the same as the lower 22-bit offset in the case of the above bus page precision example of 4MB, and only the upper 26 bits in the 48-bit address system and the upper 42 bits in the 64-bit address system are converted into the address, and the actual bus slave (3200) read / write is performed as the internal physical bus address value (3320).

[0488] Moreover, as mentioned in FIGS. 2 and 3, the conventional physical memory page address has a 2x34 physical address mapping limitation of 16 gigabytes as an example and a linear mapping limitation that caused SMP / NUMA problems, and it can be seen that the internal physical bus address (3320) of the bus page table (3300) must be mapped 1:1 to the conventional bus slave (3320).

[0489] Also, referring to FIG. 3c, when the bus page table (3300) has a physical address mapping of 16 gigabytes as an example, and when the bus table (3300) has a bus page granularity of 4 MB as an example, it can be known by a calculation formula (3500) how many lines / entries (3330) the bus table (3300) should have in total.

[0490] For example, if the above-described example bus slaves (3200) require a total of 16 gigabytes of bus addresses (3320), it can be seen that a total of 4096 bus page table (3300) entries (3330) / line are required, which is 16 GB divided by 4 MB. Of course, in FIGS. 1 and 2, it can be guessed that a total of 4099 bus page table entries (3330) are required because three 4 MB slaves are added to the 16 gigabyte memory controller (220).

[0491] In addition, in the case of the above example of 4096 bus page table (3300) lines / entries (3330), the size virtually required as a physical / kernel part (500) page table entry can also be known, and the calculation formula is (3600), as shown in Fig. 3c, 4096 entries times (virtual bus address size - bus page precision size + bus SET / other bit size). If we calculate it, if we have a full 64-bit virtual address system, it requires approximately 21.5KB since it is 4096 times (64-22+1), and if it corresponds to the 48-bit virtual address value used in the conventional x86 / 64, it requires approximately 13.5KB since it is 4096 times (48-22+1), physically or as a memory page entry in the kernel section (500), as can be seen from the calculation formula 3600 in Fig. 3c.

[0492] Therefore, in the case of the conventional x86 / 64 virtual address system, when using the bus page table (3300), the 16 GB example above only requires a very small SRAM of about 13.5 KB and a processing unit for the bus master (3100), and when it is shared / stored by dividing it into 4 KB, which is the conventional memory virtual page entry, by linking it with the kernel unit (500), it can be seen from Fig. 3c that since 13.5 KB is divided into 4 KB, only 4 (3700), which is about 3.375 entries in total, are required.

[0493] This is the initial bus page table (3300) write / read virtual bus address value (3310) when the protection SWAP (5000) / bus table (3300) is reset as described in the above-described Fig. 3b, and the start address values ​​of the four memory virtual pages operate as N (3700) random numbers, and the bus master (3100) only needs to have a total of four 48-bit or abbreviated 4 of 26-bit hardware registers so that the N (3700) random random numbers can be triggered / latched internally in hardware and written through. In this way, the bus master (3100) and the kernel unit (500) can safely share and hide the bus page table as a memory virtual page entry. In addition, it can be clearly seen through the 3700 portion of FIG. 3c that the N (3700) random numbers that the interconnect endpoint (430) in FIG. 3b must send, of course, in the case of the above example, a total of 4 must be sent.

[0494]

[0495] FIG. 4 is a drawing showing a part of a hardware interrupt replacement such as a conventional APIC, which is one of the advantages obtained when having a bus page table (3300) as proposed by the present invention.

[0496]

[0497] Referring to FIG. 4, APIC is a type of hardware interrupt, and is designed to quickly process requests generated from devices (300) inside or outside the computing device (200) in FIG. 4.

[0498] The conventional APIC is directly connected to the L1 cache (241) or the core (210), is linked to the WatchDog, and when a hardware interrupt signal comes, the work being done in the L1 cache (241) and the core (210) is temporarily suspended, and the work requested by the APIC is processed within the WatchDog time, or is passed on to the software interrupt handler of the kernel section (500), and then the work is returned to its own work.

[0499] The reason why APIC is used by connecting it to the core (210) and L1 cache (241) instead of software interrupts, exception handling, or conventional slow hardware interrupts is that the performance of the processor (200) is high, and the throughput is comparable to the bus (3010) processing speed that is beyond our imagination if there are no TLB (260) or other interference factors, but those skilled in the art know that most of the hardware speeds that must be guaranteed at the APIC level are DMAs between devices (300), and such design elements of APIC and DMA have not only lowered hardware integrity and reliability, but have also acted as factors that lower the performance of the CPU.

[0500] DMA, which was introduced for the performance of the CPU (200), cannot operate on its own, so the APIC and the L1 cache (241) / core (210) connected to it must stop working for a while and perform things like DMA work, and such interrupts are one of the few factors that determine the difference between conventional hardware and software.

[0501] However, as can be seen in Fig. 4, interrupts, which are one of the many factors that determine hardware and software, can be completely replaced through structures such as the bus page table (3300), and the method for doing so is introduced.

[0502]

[0503] In Fig. 4, number 6000 represents the part where the bus intercept replaces the conventional APIC. In order to perform software interrupts or tasks of the kernel (500) as quickly as the conventional hardware APIC, the L3 cache (243) resides therein, and as the area (6200) is updated, the assembly / bytecode (6100) area residing in the L3 (243) / LLC (250) enters the L2 (242), and ultimately passes through the L1 (241) to reach the core (210).

[0504] This allows the work done by conventional interrupts to be replaced by software, and if the CPU design is normal, the processing of what was residing (6100) in the L3 cache (243) block area must be guaranteed to be processed within a certain amount of time, in the same way as the conventional CPU (200) did, such as round robin. In addition, the speed of such a CPU device (200) is beyond our imagination, so it can mostly satisfy the level required by hardware interrupts. However, if even that is slow, an exception may occur internally, but this is avoided as much as possible because it can be exploited in hardware.

[0505]

[0506] Referring to FIG. 4, there are various ways (6100) to allow only a specific part to reside in L3 (243) / cache (240), and in FIG. 4, if conventional assembly (6100) programming, etc. is used in L3 (243) / LLC (250), and then the trigger area (6200) that causes a bus intercept (6000) in L3 (243) is updated (Dirt) or made invalid (Invalid), and then the cache (240) / kernel unit (500) leaves the updated / invalid area (6200) as is, it may remain in the CPU (200) forever while continuing to reside (6100) in L3 (243).

[0507] However, this is just an example, and there may be various applications, and to prevent these areas (6000) from being exploited by hardware / software (900), a technique such as OneTime Vector (6510) that uses the vector (6500) used when entering the interrupt handler (6500) of the kernel (500) in the conventional APIC only once or twice and then replaces it with another vector can be utilized. Also, as with the conventional OTP (OneTime Pass), something like a token can be loaded into it, but this is at the discretion of the kernel (500). Furthermore, multiple triggers (6200) / resident areas (6100) that cause bus intercepts (6000) of the L3 cache (243) / cache (240) may exist and then disappear.

[0508] When jumping to the location pointed to by the OneTime Vector (6510) in the above memory (220) area, it goes to the same location as where the interrupt handler (6500) of the kernel (500) enters after the hardware interrupt such as the APIC or the software interrupt finishes its role. Therefore, it can be seen that the purpose of the hardware interrupt such as the APIC above has been replaced. However, unlike those who are not skilled in the art, it would be clearly understood by those skilled in the art that the concept of bus intercept (6000) does not mean intercepting the bus (3010), but rather replaces the action of the conventional interrupt through the bus (3010).

[0509] Moreover, the resident areas (6100) including the trigger (6200) area causing the bus intercept (6000) of FIG. 4, the operating system (500) handler entry (6510) area, and other obfuscation / protection, etc., may not all reside in the L3 (243) / cache (240) in units of memory pages, but may reside (6100) / reference only as part of the cache block / line. In addition, other bus intercept (6000) related local intercept (6010), or global intercept inbound (6020), global intercept outbound (6030) areas that propagate / receive bus intercept (6000) to the entire chiplet connection (3020) are, as shown in FIG. 4, only specific trigger (6200) trigger areas reside (6100) / reference L3 (243) / cache (240), and the remaining code area / byte code area is only loaded into the memory (220) area, etc., and when a bus intercept (6000) is triggered (6200), the remaining areas may be loaded into the cache (240) due to a cache miss, etc.

[0510]

[0511] FIG. 4a is a diagram illustrating a method for mitigating a bus intercept (6000) that replaces the above hardware interrupt element so that it cannot be exploited by untrusted hardware or conventional malicious software (900).

[0512]

[0513] Referring to Fig. 4a, although the hardware interrupt that replaces the conventional APIC has been transferred to software, the protection for this can be exploited not only by malicious software (900) but also by untrusted hardware.

[0514] Therefore, as suggested above, when entering OneTime Vector (6510) through bus intercept (6000), instead of going directly to the interrupt handler (6500) of the conventional operating system (500), OneTime Vector (6510) goes to one Vertex (6650) in a graph (6600) that is widely used in conventional data structures.

[0515] In the case where the operating system (500) dynamically or arbitrarily creates a graph (6600) having N vertices (6650), if the operating system (500) itself enters it with a normal routine / token, it will go to the interrupt handler (6500) of the real kernel (500) hidden in the middle (6650) after traversing several times without affecting the performance at all, but if not, it will cause an exception or an access to an incorrect (non-existent) bus address (3310). As mentioned in another drawing above, in the case where confidential computing is required, this processing can cause the root complex (420), which is the only bus master (3100), to reset or turn off the processor (200), or in the general case, it can cause it to return to the graph (6600) above.

[0516] For example, the part that statistically randomly guesses for performance in the conventional TLB (260) / L3 cache (243) may be exploited as a part that determines which memory area to hook / modify, but such an action will statistically pass through the graph (6600) Vertex (6650) several times, but will statistically fail shortly after on the graph (6600).

[0517]

[0518] FIG. 4b is an example diagram of a method for enabling interconnection (6700) between the above-described bus intercept (6000) and the areas that cause the bus intercept (6200) through cache (240) to cache (240) access between processors (200) in a bus (3010) / chiplet connection (3020) shared structure, such as a bus intercept (6000) that replaces an interrupt.

[0519]

[0520] Referring to Fig. 4b, the above hook / modulation is statistically not limited to avoiding a few times in the graph (6600) of Fig. 4a, and may be modulated only in the entire memory (220) area, or within the cache (240). To prevent such exploitation, the above bus intercept (6000) areas are connected (6700) complementarily or otherwise between the cores (210), and between the first computing device (201) and the second computing device (202), which are exemplary processors, through a mutual reference method such as through a virtual bus address (3310).

[0521] For example, when an update / modification occurs in one core (210) or processor (201), the same bus (3010) read / write is propagated to the other processor (202), so even if the modification is successful once or twice, it causes an exception / failure in another part or a reset of the processors (201 / 202) within two or three bus (3010) cycles.

[0522] The reason why these methods are possible is that, as shown in FIGS. 2 and 3, the core (210) / cache (240) is merely a bus slave (3200) in this patent, and is assigned a single virtual bus address (3310) as a chiplet connection (3020) / bus (3010), shares it, and operates on the virtual bus address (3310).

[0523]

[0524] Figure 5 is a drawing showing a bus direct access (Direct Bus Access) part that replaces conventional DMA / IOMMU (DTLB), etc., in the bus page table (3300) structure.

[0525]

[0526] Referring to FIG. 5, this is a diagram illustrating a method of maintaining mapping coherence between interconnect links (400) such as CXL, even when the bus (3010) is not shared between two computing devices, a first computing device (201) and a second computing device (202), by using a bus page table (3300). This effect of FIG. 6 is an advantage obtained when having a bus page table (3300).

[0527] In the past, in order to maintain mapping coherence from a second computing device (202) acting as a host to a first computing device (201) acting as a device (300), there was an element that delegated the core (210) inside the second computing device, such as a conventional DMA (Direct Memory Access), or the cores (210) were directly moved so that the CPU core (210) could directly read and write data between the first computing device (201) and the second computing device (202) by a method such as polling.

[0528] However, the parts that the conventional kernel (500) shown in Fig. 5 provides to the user application (900) as infrastructure (600) or system API can also be provided in the form of a Unix socket (620), and eventually, when the role is properly performed or performed in any form, they have one large buffer allocated to a virtual memory page.

[0529] For example, if it is a Unix socket (620) of an audio framework (650) in which the kernel (500) processes / synthesizes audio signals of the user application (500), each user application (900) has small audio sockets (620) / buffers, and they are ultimately calculated and combined into one large buffer for each target audio channel according to the sampling rate / resolution.

[0530] The above example is only for the audio framework (650), and in the case of the graphics framework (640), there is a socket (620) / buffer of the Vulkan / OpenGL graphics framework (640) to perform / save the work for rendering / acceleration tasks, etc. Also, the network is the same, and even all actions such as opening, reading, and writing files must pass through the socket (620) / buffer of this infrastructure (600) / framework.

[0531] Normally, the kernel part (500) data structure has a ring buffer structure as shown in Fig. 5 to perform this processing quickly and have a large capacity, but it may also have various data structures such as a queue / stack.

[0532]

[0533] In addition, the act of reading or writing a ring buffer such as the example of FIG. 5 cannot be delegated to DMA, and at least once, a core (210) such as a CPU must directly write or read in a manner such as polling by a user application (900).

[0534] Moreover, in order to achieve the behavior of using the conventional device or device (300) such as the graphic framework (640) rendering socket (620) / buffer, or network socket (620) buffer, it must reach a small buffer existing inside the device (300), which is the first computing device (201), and thus, it is necessary to send data there and perform mapping coherence, and for that, another last / device (300) buffer existing inside the kernel unit (500), or kernel driver unit (510) is required.

[0535] The conventional mapping coherence has the same meaning and operation between the user application part (900) and the framework (600) socket (620) / buffer of the kernel part (500), or between specific memory areas and specific memory areas, but the mapping coherence meant in this patent means the mapping coherence between the kernel part (500) of the destination / kernel driver (510) device (300) buffers inside the kernel part, each of which has one for each host and device (300) between the second computing device (202) and the first computing device (201).

[0536] This kernel part (500), or kernel driver (510) buffer area must be written to by the kernel part (500) infrastructure part (600) by directly moving the core (210) in the user application part (900) such as polling, and then by directly moving it once again to write to the kernel driver (510) last / device (300) buffer area at the destination of the second computing device (202).

[0537] However, the act of reading and writing buffering for the first computing device (201), which is the device (300), is performed in a timely manner without wasting CPU (200) resources, such as polling, when the buffer is filled or emptied by the device (300), which is the first computing device (201), by combining with a conventional hardware interrupt, unlike the socket (620) / buffers of the framework unit (600).

[0538] In addition, the conventional DMA takes the method of delegating the work that the CPU (200) must do in a timely manner by interrupt when the buffer between the first computing device (201) and the second computing device (202) is updated, and reading and writing to the buffer on the other side, and when the work is performed or completed, it reports back to the core (210) / CPU (202) through a hardware interrupt such as the APIC.

[0539]

[0540] Referring to Fig. 5, one can see a portion that uses a bus page table (3300) and a bus intercept (6000) to replace the above DMA or interrupt.

[0541] In the DMA section above, the method and role of ensuring mapping coherence between the second computing device (202) and the first computing device (201) are similar, and the act of writing to the buffer inside the infrastructure part (600) / framework of the kernel part (500) by the application part (900) / CPU (202) directly through a polling method, etc., cannot be replaced by an element such as DMA, and it is similar in that it must be written or read directly at least once in a format such as polling to the inside of the kernel part (500) driver (510), which is the end part of the second computing device (202).

[0542] However, in the subsequent operation, in order to trigger or receive an interconnect link (400) packet such as CXL / PCIe from the CPU (202), like a conventional DMA, a bus intercept (6000) that replaces the interrupt presented in the above-described Figures 4 must be triggered (6200). Since the CPU (202) must perform at least one write or read operation from the user application section (900) to the framework / infrastructure section (600), or from the framework / infrastructure section (600) to the kernel section (502) or the internal driver (510), the CPU (202) / kernel section (502) triggers (6200) before and after the buffer write / read.

[0543] After that, as with DMA, a bus intercept (6000) is triggered (6200), so the CPU (202) / core (210) continues to perform its work, and the sole master (3100) of the bus (3010) takes over the act of reading and writing the bus (3010) like a conventional DMA.

[0544] However, as an example, unlike DMA, the entity that triggers (6200) the bus intercept (6000) need not be the second computing device (202), which is the host; conversely, the first computing device (201), which is a device (300) that shares some of the same virtual bus address (3310) / global bus address (3310), may do so.

[0545] In addition, unlike the example of FIG. 5, the subject requesting read / write of the bus (3010) through the bus intercept (6000) can also be reciprocal without having to be determined to one side, unlike the conventional DMA. However, it is preferable that the subject that triggers (6200) the bus intercept (6000) and requests read / write of the bus (3010) be the computing device (200) itself, like the conventional DMA.

[0546]

[0547] Referring to Figure 5, in addition to showing the replacement of the conventional DMA with a bus intercept (6000), the flow of replacing an element such as the conventional IOMMU (DTLB) is shown by an arrow (7000 series) in the interconnect link (400).

[0548] The recognition / initialization and mapping coherence flow is represented in the order of arrows (7000 series) through the interconnect link (400) protocol, such as CXL, which is an example of FIG. 5 between the first computing device (201) and the second computing device (202). However, the interconnect link (400) protocol of FIG. 5 is not limited to CXL as described in this patent, but in the case of the example CXL, there are currently three types of device protocols.

[0549] While replacing the conventional DMA, maintaining mapping coherence between two computing devices is not an easy problem, like the NUMA / SMP problem between two computing devices (201 / 202).

[0550] For example, the professor team that developed CXL Type 2.0 and the previously mentioned DirectCXL also mentioned these as the reasons for the introduction of DirectCXL and CXL Type 3.0. Contrary to the development purpose of CXL proposed by the consortium including Intel and several large vendors, they explained that the cache coherence problem maintained through CXL.cache(480) in CXL Type 1.0 and Type 2.0 is not as easy as the problems between NUMA / SMP or caches(240) in a wide memory area. For the reasons above, CXL3.0 does not have the cache CXL.cache(480) protocol, and is designed to faithfully play only the role of a memory expander.

[0551]

[0552] For the reasons described above, in the case of CXL type 1.0 and CXL type 2.0 that use a cache coherence maintenance protocol, as described as an example above, an element that plays a role similar to a conventional IOMMU or TLB (260) that converts a virtual page address to a physical page address in the buffer area of ​​the device driver (510) located in the second computing device (202) exists for CPU DMA, but in order to support a protocol such as CXL.cache (480), an element such as DTLB must also exist in the device (300) and the first computing device (201) as examples.

[0553]

[0554] Specifically, the conventional PCIe was developed with the previous PCI in mind for upward compatibility. However, in the PCI era, personal computing devices (200) were very old-fashioned, the concept of virtual paging addresses was just emerging, and the capacity of the buffer memory required by the device (300) was very large, reaching several megabytes. Therefore, referring to FIG. 5, a small area of ​​the memory, called BAR (Base Address Register), was used as a tunnel, or WINDOW, that maintains mapping coherence between the second computing device (202) and the first computing device (201).

[0555] As the name suggests, this BAR area contains information such as memory address values ​​and sizes to be used as a memory tunnel / WINDOW between two computing devices (200) at the same time as a register area related to reading / writing the setting values ​​of the device (300), and has been shared through the device setting, initialization, and enumeration process such as conventional PCIe or CXL.io (460) and allocated to the memory area of ​​each computing device.

[0556] However, as the concept of virtual paging address and the speed of the two computing devices (201 / 202) rapidly developed, whether it was a host or a device (300), the area that existed in the above physical paging address had to be implemented in the concept of a virtual address, and the area used as the device (300) driver (510) buffer also became larger to keep up with the speed of the computing device (200).

[0557] As a result, in order for BAR to be allocated with the virtual address concept, the introduction of the IOMMU concept, such as the TLB (260) that translates the virtual page address, as in the core / L3, is required, and this not only reduces the performance of the computing device (200), but also causes a security issue that requires TLB (260) processing between the two computing devices when the virtual address translation table between the two computing devices (201 / 202) is missing.

[0558] For reference, TLB (260) is an abbreviation for Translate Lookup Block. Since it is impossible to have all the information about the address in the specific virtual paging table / directory area fixed in the kernel section (500) for address translation, if the value of the address to be translated is unknown (Invalid), it means that it looks up only that value in the kernel section and has it in the form of a limited table for preview (Lookup).

[0559] For example, even if the TLB of a conventional IOMMU is fully valid through kernel queries, the lookup area is usually small, 12 or slightly more, so even if it has 12 4KB blocks, it is very small to cover all the buffers allocated to the virtual address of the current kernel driver (510). Currently, there is no way to alleviate this problem, as it requires discarding virtual addresses or a new form of caching (which does not actually exist).

[0560] For these reasons and in order to preserve the PCI compatibility of the time when the conventional virtual address concept of PCIe was born, BAR 0 allocated within the kernel (500) must operate even when DMA or device settings are not yet complete, so to be exact, since the IOMMU area in the kernel cannot be allocated to the outbound area exceeding 4 GB even in the 64-bit system which is different from the 32-bit address system, it is allocated within a linearly limited size in paging units even within the internal (500) virtual address, and for other mapping coherence, a method was commonly used in which BAR1 to BAR5 point to the virtual address area allocated within the infrastructure (600) / framework or kernel driver (510).

[0561] When the actual DMA and DTLB / IOMMU operate between the two computing devices (201 / 202), they process large-capacity ring-like buffers through the above areas, and when traversing the ring / buffer back and forth, if the address value is invalid, the value is updated first and then the ring / buffer is traversed again. As explained above, the Lookup Table can have only a very small number of Lookup Tables, so in order for the buffer or ring to continue to traverse and move forward, one or more of them must continue to be sacrificed, just like inside the cache (240) / TLB (260).

[0562]

[0563] However, in FIG. 5, when replacing the conventional DMA with a bus intercept (6000) without using a method such as the conventional DMA or IOMMU / DTLB, the core (210) / cache (240) of the conventional CPU (202) must directly write or read the buffer at least once or twice before delegating the act of triggering (6200) it to the DMA, so as to directly trigger the bus intercept (6000) before and after the buffer read / write, as in the above, the virtual (global) bus address table (3301) area of ​​the first computing device (201), which is the device (300), which operates as a lookup table inside the IOMMU / DTLB, is updated in advance as step 7400 of FIG. 19 through an interconnect protocol such as CXL.cache (480), and then the bus Triggers interception (6000) (6200).

[0564] However, unlike the second computing device (202) which is the host, the first computing device (201) which is the device (3000) is an embedded device, so in FIG. 5, the L2 cache (242) is expressed as a shared cache and LLC (250) which is the last level cache, and the area of ​​the memory (220) is also not more than a few MB like STM32.

[0565] Therefore, even if the granularity of the bus table (3301) of the first computing device (201) is 4KB, which is the same as the memory virtual paging size of the second computing device (202), which is the host, there is no problem in traversing the bus table (3301) or having the physical capacity, and if the first computing device (201) has the granularity of 4KB, which is the example, the virtual address lookup entry replacement that DTLB / IOMMU was doing is sent in advance and replaced before the second computing device (202), which is the opposite side, causes (6200) a bus intercept (6000) through the CXL.cache (480) in step 7400 of FIG. 5.

[0566] Of course, it is not necessary to match the 4KB granularity, and even if it has a size of 8KB or larger, the problem is that the area of ​​the first computing device bus table (3301) that operates as the lookup table before and after the bus intercept (6000) or the area of ​​the first computing device bus table (3301) that operates as the lookup table before and after the bus intercept (6000) can be updated more frequently through CXL.cache (480) as in step 7400 of FIG. 5, which is an example.

[0567]

[0568] Referring to FIG. 5, the bus page table (3301) of the first computing device (201) that operates as such a conventional Lookup Table can be done by the kernel part (501) of the firmware / operating system of the first computing device (201), which is the device (300), but since this patent recommends a firmware-less method that enables hardware reliability and verification and maintenance of the embedded device like software, the bus page table (3301) of the first computing device (201), which is such an interconnect device, is written as a bus (3010) read / write protocol such as CXL.mem (470), as step 7200 of FIG. 5, after the second computing device (202), which is an interconnect host, is recognized / enumerated / negotiated through CXL.io (460), as step 7100 of FIG. 5.

[0569] CXL.mem (470) of FIG. 5 was designed for a memory expander, abandoning conventional cache coherence, but in the present invention, it is used as a protocol for reading / writing a virtual / global bus address (3310). In addition, the bus page table (3001) of the first computing device (201), which is the interconnect device (300), is created together when a specific area bus table entry (3330) of the first computing device (201) is written through, as described in other drawings above, and operates as a DTLB of the first computing device (201), which is the device (300) of the example.

[0570] However, each bus table (3300) area (3330) is rewritten, and although it may be possible to use the same method as CXL.mem (470) as in the embodiment of FIG. 5, it may be said that it is desirable in terms of performance to have CXL.cache (480) access the address value (3310) of the bus table (3301) Entry (3330) at step 7400 of FIG. 5 and quickly replace it.

[0571] Of course, the above two cases are examples of FIG. 5, and each bus table (3300) area (3330) is rewritten, unlike FIG. 5, and CXL.mem (470) or CXL.cache (480) in step 7400 of FIG. 5 does not access the address value (3310) of the bus table (3301) Entry (3330), but a conventional slow method such as CXL.io (460), which is an example of the interconnect link (400), may be used.

[0572] However, in order to prevent each area (33330) that operates as a split address translation lookup entry area from being accumulated, this is not recommended in the present invention.

[0573] In addition, in order for the second computing device (202) to trigger a remote bus intercept (6000) in the first computing device (201), an area having mapping coherence in the two bus page table areas and a bus intercept (6000) trigger area (6200) between the two computing devices (201 / 202) must be shared / corresponded as part of the virtual (global) bus address dimension.

[0574] Moreover, the act of causing the above bus intercept (6000) trigger (6200) can be caused (6200) through a shared virtual bus address (3310) using CXL.mem (470) as an example step 7300 of FIG. 5, just as the above bus intercept (6000) is complementary between two computing devices, or it can be caused (6200) by updating the virtual bus address (3310) with CXL.cache (480) as a step 7400 of FIG. 5, which is ultimately the same act.

[0575]

[0576] In conclusion, we sought a method to replace the conventional DMA and IOMMU / DTLB by appropriately mixing the interconnect link (400), bus table (3300), and bus intercept (6000) between two computing devices (201 / 202) through the 7000-series methods of the above-described FIG. 5.

[0577] Also, as can be seen in FIG. 5, the advantage obtained by replacing the conventional DMA / IOMMU (DTLB) is that the second computing device (202), which is the root complex (420) of the interconnect, can directly and transparently access all internal elements of the first computing device (201), which is the endpoint (430) of the interconnect, without going through the firmware (501) elements such as the core (210) / LLC cache (250), and read and write the bus (3010) of the first computing device (201), which is an example.

[0578] This means that the HAL (Hardware Abstract Layer), i.e., the hardware abstraction area, can be directly and transparently accessed from the second computing device (202) through the bus table (3301) and interconnect link (400) of the first computing device (201) without lowering the performance of the operating system kernel (501) of the first computing device (201), which is the firmware, in the low-performance core (210) / cache (240) of the conventional embedded device (300).

[0579] The conventional HAL is not limited to GPIO (314) for various peripheral devices such as SPI or I2C, but can be any element corresponding to a bus slave (3200), and even if the HAL is limited to only peripheral devices, it still has a role as hardware that can be transparently defined in software (900), and in a firmware-less form in the two computing devices that the present patent seeks to implement, the second computing device (202) can directly access the slave (3200) area such as the HAL. In addition, instead of the firmware-less form, a method may be adopted in which the second computing device (202) transparently accesses the same bus area (3200) as the core (210) / cache (240) of the first computing device (201) that drives the firmware (501) of the device (300).

[0580] Moreover, direct access to a physical area such as GPIO (314) in such HAL can be written to or read from the HAL area as a virtual bus address value (3310) without the need to have a buffer in the form of a virtual buffer allocated to the driver (510) of the kernel section (501) shown in FIG. 5 or a buffer limited to a linearly allocated area within the kernel section (501), even when the performance of the first computing device (201) such as the embedded system (300) is very low.

[0581] Such applications are advantageous in places where real-time performance is required, such as in the drive driver (310) of a display device combined with an interconnect link unit (400), or in audio embedded devices (300) such as audio consoles, and where low price competitiveness, such as printer devices, is required. For example, since the first computing device (201) described above already has more than enough performance even with the same level of STM32 used in the conventional embedded device (300), it can also be used by using an open chiplet bus (3020) such as UCIe to connect the interconnect endpoint (430) such as a standardized CXL that acts as the internal chip (201) and the bus master (3100).

[0582]

[0583] FIG. 5a is a drawing that more clearly shows Direct Bus Access, which replaces DMA / IOMMU (DTLB) shown in FIG. 5 above, from the internal perspective of the bus master controller (3110) and the bus page table controller (3120).

[0584]

[0585] Referring to FIG. 5a, one can see the bus page table (3300) and the act of controlling it as a bus table controller (3120), and the bus master controller (3110) that organically links with an interconnect link (400) such as CXL in FIG. 5 and a chiplet connection (3020) such as UCIe.

[0586] In FIG. 5a, an open switch bus / fabric matrix (3010) exists between a bus master (3100) and slaves (3200) within each of the first computing device (201) and the second computing device (202). The bus slaves (3200) connected through the internal bus (3010) reach the root complex (420) / endpoint (430), which is the sole master (3100) of the bus, and within the bus master (3100), the bus master controller (3110) is further divided into a bus stream unit (3220) that links the interconnect link (400), chiplet connection (3020), etc., and a bus read / write unit (3210). Within the bus master controller (3110), the bus table (3300) exists in a physical form such as SRAM.

[0587] In addition, there is a bus table controller (3120) that manages a bus table (3300), a bus synchronous / asynchronous allocator (3130) that re-allocates bus requests to buses (3010) / chiplet connections (3020) etc. in conjunction with the bus table (3300), and a bus data mover (3140) that is composed of a plurality of bus channels (3150) such as a conventional DMA that receives allocations from such allocator (3130).

[0588] The above bus data mover (3140) is connected to a chiplet connection (3020) such as UCIe, Steam Out (3232), or Stream Out (3232) of an interconnect link (400) such as CXL, which is again controlled by bus channels (3150), and bus slaves (3200) are connected as a bus R / W (3210).

[0589] Moreover, since the TLB (260) / cache (240), which was the conventional master (3100), becomes a slave (3200) in the bus table controller (3120), a TLB (260) / cache (240) checker (3160) that takes over that role exists, and must function as an element (3160) that checks the TLB (260) / cache (240) change through bus read / write as a slave (3200) area located inside the L3 cache (243) / LLC (250) address or separately.

[0590] However, the Valid Check inside the cache / TLB (260) is omitted in the example drawing because, as in the conventional cache (240), the TLB (260) sacrifices a specific part to read a specific virtual paging table / directory area of ​​the kernel (500) by referencing that area by making it into a form similar to the conventional Invalid.

[0591] However, as shown in the arrow in Fig. 5a, the TLB (260) / cache (240) verification element (3160) may also be brought into the bus table controller (3120) and the synchronous / asynchronous allocator (3130) in the form of Stream In (3231), such as the Stream In (3231) from the chiplet connection (3020) such as UCIe in Fig. 5a, or the interconnect link (400) such as CXL, but this is not recommended because it may be exploited by untrusted hardware, etc.

[0592]

[0593] Referring to FIG. 5a, the bus page table (3300) physically existing in the bus master controller (3110) / bus table controller (3120) queries / traversals the bus tables (3302 / 3301) as in the conventional cache (240) when a Stream In (3231) signal comes in from the chiplet connection (3020) or the interconnect link (400) by the bus allocator (3130), and as in the conventional cache (240), if the address (3310) requesting bus read / write in the Stream In (3231) signal and the lower bits (22 bits) as much as the granularity, and the matching upper bits (26 bits) are between the above-mentioned granularity areas, it is considered to exist (Hit) in the virtual / global bus address (3310). After converting the internal physical bus address value (3320) in the bus table (3301 / 3302), bus read / write (3210) or bus stream out (3232) is performed by the bus data mover (3140) channels (3150) allocated to the actual bus allocator (3130).

[0594] Here, the difference between the bus Read / Write (3210) and Stream IN (3231) / Stream OUT (3232) for the bus (3010) is that all bus slave (3200) elements can be read / written and are assigned a bus address value (3320), but a chiplet connection (3020) such as UCIe and an interconnect link (400) such as CXL can be read / written with a bus address value (3320) as before, but unlike before, the internal bus (3010) and the external chiplet connection (3020) / interconnect link (400) are processed as Stream In (3231) / Stream Out (3232) so that they can organically interact with each other.

[0595] The above distinction and processing are made only as a distinction between bus Stream (3230) and bus Read / Write (3210) in the example of Fig. 5a, as there are various common bus read / write and bus data stream standards such as AXI / AHB or UCIe.

[0596]

[0597] In addition, as shown in FIG. 5a, the bus data mover (3140), or bus channels (3150), which are elements assigned by the bus allocator, may have only a single channel (3150), unlike FIG. 5a, and the asynchronous concept of the channel (3150) and the bus allocator (3130) introduced for performance may be configured as a single channel or synchronously, not as shown in the drawing.

[0598] However, since race conditions are not such a big problem as previously explained, we recommend N channels (3150) and asynchronous bus allocation (3130) as shown in Fig. 5a above so that many tasks can be processed at once even if the bus (3010) stops in terms of performance.

[0599] In addition, these N channels (3150) are connected in an N-to-N manner to bus slaves (3200) and interconnect link (400) Stream Out (3232) such as CXL, or chiplet connection (3020) Stream Out (3232) such as UCIe in a form similar to a crossbar of the conventional AXI, so that the N channels (3150), which are actual physical masters of the bus (3010) in the form of a conventional crossbar, can be accessed simultaneously to bus slaves (3200), chiplet connection (3020) Stream Out (3232), and interconnect link (400) Stream Out (3232).

[0600] Through the methods of Fig. 5a, even if the bus is stopped for a moment when a stream input (3231) comes in, the bus can quickly and asynchronously cross over to finish processing, and the bus master (3100) can receive the stream input (3231) again, or the status of the TLB (260) / cache (240) can be continuously and periodically checked through the TLB / cache status checker (3160).

[0601] The application of the bus intercept (6000) or bus table (3300) in the remaining interconnect link (400) is the same as the 7000 series steps of FIG. 5.

[0602]

[0603] FIG. 6 is an example diagram of a part that can be applied to a local switch (410) or a network such as the Internet through application with a conventional NTB when a bus paging table (3300) is applied to an interconnect link (400) such as CXL, as in the above FIG. 5.

[0604]

[0605] Referring to FIG. 6, the organic operation within the bus master (3100) and root complex (420) / endpoint (430) described in the above FIGS. 5 is illustrated in the drawing, but FIG. 6 shows a method of applying / replacing the conventional NTB structure and switching (410) with a bus table (3300) by using the bus master (3100) and bus table controller (3120) designed from FIG. 1, the bus intercept (6000) of FIG. 4, and the operation method (7000) as an IOMMU / DTLB through organic combination with the interconnect link (400) of FIG. 5, as the role of the NTB within the conventional interconnect switch (410).

[0606] In Fig. 6, there are also NT conversion elements, which are conventional NTB elements, and these NT conversion elements are a part that explains how the NTB inside the interconnect switch (410) can communicate (Transaction) in a non-transparency manner, and this address conversion (Translate) can be replaced with the bus page table (3300), etc.

[0607]

[0608] More specifically, in order for the NTBs (411, 412, 413) inside the conventional interconnect switch (410) to communicate 1:1, the NTs must communicate (transact) with each other's addresses inside the switch, so the B / D / F (Bus / Device / Function) value that recognizes the NTB as an endpoint (430) as a root complex (420) and the B / D / F value that enables the NTs to operate with each other must correspond 1:1.

[0609] Through these values ​​such as B / D / F, the operating system (502) of the second computing device (202), which is the host, manages the devices (300), and conversely, it is also necessary to convert 1:1 to enable communication (transaction) between devices within the switch (410).

[0610] The above translation action is performed as a request ID when a request is made by inducing an interconnect link (400) packet such as PCIe or CXL within each host's NTB (411 / 412 / 413).

[0611] However, the above-mentioned Request ID address translation is only normally operated between the root complex (420) and the device, the endpoint (430), at a simple 1:1 link level, and for communication (Transaction) between the root complex (420) and the root complex (420), which is the purpose of the conventional NTB switch (410), not only masquerade negotiation (Masquerade Handshaking) / masquerade communication (Masquerade Transaction) that makes the two appear as endpoint (430) devices, but also an address translation lookup table, such as the conventional IOMMU (DTLB), is required for a shared area for mapping coherence, which is allocated to a virtual page address between the two computing devices, which are the root complexes (420).

[0612]

[0613] The above Lookup Table function was performed within each of the conventional NTBs, but as suggested in FIGS. 5 and 6, it can be replaced through the bus paging table (3300) of the interconnect switch (410).

[0614] Such a method can be similarly operated through interconnect link (400) protocols such as CXL of the root complex (420), which are hosts connected to the interconnect switch (410), as in FIGS. 5 and 5a. For example, the root complex (420), which is a host, recognizes, initializes, and enumerates NTBs (411, 412, 413) as its endpoint (430) devices through the conventional PCIe or the example of CXL.io (460), and reads / writes such information through the NTB Register of FIG. 6, which contains information on configuration elements such as BARs of such NTB devices.

[0615] The process thereafter is, as in FIGS. 5 and 5a, a conventional memory such as CXL.mem (470) as an example between the root complex (420) and the endpoint (430) NTB at step 7200, or a virtual / global bus address (3310) presented in the present patent can be read / written to the area where the bus page table (3300) is located, and as in FIGS. 5 and 5a, as step 7400, the bus table (3300) of the switch (410) through CXL.cache (480) is replaced in advance as in FIGS. 5 and 5a, by replacing each entry (3330) that operates as a conventional DMA or Lookup Table before causing (6200) a bus intercept (6000) between the endpoint (430) device (300) NTB and the root complex (420). The root complex (420) is applied so that the root complex (420) can have mapping coherence beyond NTB and NTB.

[0616]

[0617] However, the table that mediates communication (Transaction) as B / D / F or Lookup Table between NTBs in FIG. 6 is the bus page table controller (3120) of FIG. 6, or the bus table (3300) of the only master (3100) inside the switch (410), and both operate simultaneously as B / D / F (3320) or Lookup Table (3310).

[0618] Each entry (3330) of the internal bus page table (3300) of this switch (410) is usually 12 or 24, etc., for each NTB, like the conventional DTLB Lookup Table, but in FIG. 6, the bus page table (3300) inside the bus table controller (3120) is merged / divided by the number of NTBs times 12 or 24, and when this area is updated in each NTB through a protocol such as the example CXL.cache (480), the physical bus page table (3300) inside the bus table controller (3120) is also updated through bus Read / Write as the actual merged (only) table (3300).

[0619] Therefore, in the above example, the number of entries (3330) of the internal bus page table (3300) of the switch (410) is the number of NTBs that can be connected to the switch (410) and the number of entries that operate as the Lookup Table of each NTB, multiplied by 12 in the above example, and one more is required for the mCPU (210) that operates as the original bus address (3310) within the switch (410).

[0620] Accordingly, the number of bus page table (3300) entries (3330) that must be used as the NTB B / D / F and NT Lookup Table inside the interconnect switch (410) is not the 4 MB granularity of the other examples mentioned above, but the 4 KB granularity optimized for DTLB. In the case of 12 lookup tables in a large-capacity switch (410) with 48 NTBs, it is only about a little over 500, and even in the case of 24, it is only a little over 1000. This number of table entries (3330) can show lower cycle traversal and better performance than in the computing devices (200) mentioned above, which are the hosts.

[0621]

[0622] In addition, unlike the previous drawings, the bus table controller (3120) inside the interconnect switch (410) must traverse at least twice for 1:1 communication (Transaction), and when the 16-bit B / D / F value of the first NT is hit as the internal physical bus address (3120) inside the bus page controller (3120), the 64-bit or 48-bit NTB Lookup Table mapped to it is converted into a virtual bus address value (3310), and then the traversal is completed once, and the virtual bus address value (3310) 64-bit / 48-bit is found from the next table (3300) entry (3330) with the virtual bus address value (3310), and the 16-bit B / D / F mapped (allocated) to it is After having the internal physical bus address value (3320), only the B / D / F address value is changed to the request ID and sent to the NTB (411, 412, 413) or endpoint (430).

[0623] In the case of the NTB inside the switch (410) that received the above communication (Transaction) packet, in order to make it appear as modified to the root complex (420) of the computing device (200) connected to it in the packet, it changes it back to a 1:1 mapped / set 16-bit B / D / F value, or hides the 64-bit Look Up Table virtual address value of the host device, or converts it into a 1:1 mapped address value (3310) and sends it to the root complex (420) that is the host in order to apply it as an address value on the Internet, etc.

[0624] Even within the conventional interconnect NTB switch (410), the conversion (Translate) and operation method are similar, but in FIG. 6, the conventional DMA, etc., CXL, which is an example of the interconnect link (400) protocol, and the 7000 series method of FIG. 5, which utilizes the bus page table (3300) as an IOMMU / DTLB Lookup Table, are applied, and the only difference is that the table (330) within the switch (410) is traversed twice in total.

[0625] However, to be exact, after traversing about 1 / 2 of the way in the first round, we only need to traverse the remaining 1 / 2 again, so it converges to traversing once.

[0626]

[0627] Referring to FIG. 6, the NTB of the interconnect switch (410) and the mCPU (210) may have a bus table (3300) that sets or maps the internal B / D / F values ​​(3320) for the NT to NT switching / routing and the Lookup values ​​(3310) that operate between the root complexes (420) and each host via the NT, in the cache (240) area inside the mCPU (210). Although such a bus table (3300) may also be in the memory controller (220) area, since it does not require the many entries (3330) as suggested in the present patent and FIG. 6, it is recommended that the cache (240) inside the mCPU (210) virtually have it as a paging entry, as in other drawings.

[0628] Therefore, when the NT B / D / F values ​​(3320) or BAR Lookup values ​​(3310) are changed by the host (420) or the mCPU (210), they are simultaneously reflected (Write-Through) in the internal bus table (3300) of the switch (410) and the cache (240) of the mCPU (210) by bus read / write, and are simultaneously reflected (Write-Through) in the 64-bit BAR Lookup values ​​(3310) and 16-bit B / D / F (3320) of the physical bus page table (3300) that operates as the Lookup Table within the table controller (3120) of the interconnect switch (410).

[0629] In conclusion, switching / routing utilizing the bus table (3300) in the interconnect switch (410) will initially be used only within a local network such as an enterprise, but if it does not pass through a few nodes, it can operate sufficiently as an Internet that can be applied worldwide.

[0630]

[0631] However, due to space limitations, the above-described Fig. 6 only shows applications within a local switch (410). As for the application method on the Internet, since the bus virtual address value (3310) operating as the NTB Lookup Table in Fig. 6 is made of a 64-bit address system, that part can be used in conjunction with the conventional IPv4 (32 bits) or IPv6 (128 bits), or as another alternative, the virtual bus address value (3310) can be identified as an IP by corresponding the address value through a hash.

[0632] In addition, if various techniques are applied appropriately, 1:1 communication is possible without routing address translation that was attempted to be introduced in IPv6, and since all problems occurring in the current network are problems that arise from packet-based nature, it is possible to return to the state of safety, high performance, and isochronousness of the previous circuit method without the above IP allocation by linking with hash-based DNS.

[0633] Moreover, such circuit nets omit processes such as address translation like in the conventional IPv6, and the processing bandwidth and performance of network devices are dramatically improved, and unlike IPv6, elements such as address allocation are used once and discarded at a specific transaction timing that is deadlocked 1:1 through the various techniques mentioned above, with the host root complex (420) and NTB (411, 412, 413) using an entirely arbitrary 1:1 Correspond value.

[0634] However, the 16 bits inside the link such as B / D / F, or the SET bit (3340), may still be left to be set by the mCPU (210) as a management element through the switch (410) QoS or the mCPU (210) of the telecommunications company. Of course, if the switch (410) of the telecommunications company is not trusted, the NTB may be alleviated by the consumer connecting the endpoint (430) to the switch (410) through a chiplet connection (3020) such as UCIe, as shown in FIG. 6. In that case, the B / D / F value, which operates as the internal physical bus address (3320) of the endpoint (430) of the switch (410), and the Lookup Table Entry value, which operates as the virtual / global bus address (3310), are conventional IPs, which the consumer can continue to change at will.

[0635]

[0636] In order to more clearly express the technical idea of ​​the present invention, the attached drawings briefly express or omit components that are not related to or have little to do with the technical idea of ​​the present invention.

[0637]

[0638] Although the configuration and features of the present invention have been described above based on embodiments according to the present invention, the present invention is not limited thereto, and it is obvious to those skilled in the art that various changes or modifications can be made within the spirit and scope of the present invention, and therefore, it is made clear that such changes or modifications fall within the scope of the appended patent claims.

Claims

1. In an interconnect computing device system, Contains at least one computing device (200), At least one computing device (200) above, It is divided into at least a first computing device (201); and a second computing device (202); Connected to a single bus via chiplet connection (3020); and Connected to a separate bus via an interconnect link (400); characterized by at least one of: The above chiplet connection (3020) is Extended bus space through bus table address translation; and An extended bus space that does not go through bus table address translation; consisting of at least one of: The first computing device (201) and the second computing device (202) are configured to operate as a single extended bus space. Interconnect computing device system.

2. In paragraph 1, At least one computing device (200) above, When connected to a separate bus through the above interconnect link (400), A root complex (420) connected to an interconnect link (400) as a master; and At least one of the endpoints (430) connected to the interconnect link (400) as a slave; With the characteristic, The above interconnect link (400) is Connection between the above root complex (420) and the above endpoint (430); and It is composed of at least one of the above root complex (420) and the above root complex (420); The first computing device (201) and the second computing device (202) are configured to operate as a single extended memory space; Interconnect computing device system.

3. In paragraph 1, At least one computing device is, A bus master (3100) connected to the internal bus (3010) as a master; and Further comprising a bus slave (3200) connected to the internal bus (3010) as a slave; It further includes an internal bus (3010) that mediates between the above bus master (3100) and the above bus slave (3200), The above internal bus (3010) is The above bus master (3100) and the above bus slave (3200), FullyConnected / FabricMatrix Connected; and Share Connected / Switchbus Connected; characterized by at least one of the following; Interconnect computing device system.

4. In paragraph 3, The above bus slave (3200) is, It is divided into at least one of the core part (210); register / HAL (230); memory controller (220); and IP block; characterized by responding to the bus read / write control of the bus master (3100), Interconnect computing device system.

5. In paragraph 3, The above bus master (3100) is, Root complex (420) in the above interconnect link (400); and At least one of the endpoints (430) in the above interconnect link (400) is determined, The above bus master (3100) is, The internal bus (3010); the chiplet connection (3020); and The interconnect link (400) is characterized by mediating between Interconnect computing device system.

6. In paragraph 5, The above bus master (3100) is, Includes table controller (3120), The above table controller (3120) For control and access of the bus slave (3200) within the bus master (3100), a bus page table (3120) referred to in performing the bus address conversion method (4000); A bus assigner (3130) that assigns commands to a data mover (3140); A data mover (3140) that assigns the assigned command of the above bus allocator (3130) to one of the bus read / write (3210) and bus stream (3220) to at least one channel (3150); A bus read / write unit (3210) that reads / writes to a bus slave (3200) using a bus address value in response to a command from the above data mover (3140); and Further comprising a bus stream (3230) that performs the command of the above data mover (3140) without a bus address value; Interconnect computing device system.

7. In paragraph 6, The above bus page table (3300) is Consists of at least one table entry (3330), At least one table entry (3330) above, An internal physical bus address section (3320) that is an address that physically controls and accesses the internal bus (3010); and Includes an arbitrary virtual bus address section (3310) set (5000) and allocated by the operating system section (500); The physical address of each entry (3330) has a size / distance with a degree of precision, A bus slave is allocated at least one physical bus address entry area. Corresponds to an arbitrary physical bus address (3320) by a virtual bus address (3310), The above physical bus address (3320); and virtual bus address (3310) address conversion (4000) is performed; the recording; and the setting method (5000); are performed. Interconnect computing device system.

8. In paragraph 7, The above bus page table (3300) is It is characterized in that the physical address precision size / distance of each entry (3330) can be set differently, Further including a precision field that can record the precision of each entry (3330). Interconnect computing device system.

9. In paragraph 7, The above bus page table (3300) is As a part of the above bus page table controller (3120), it includes at least one bus page entry (3330); The above bus page entries (3330) are stored and correspond to the internal physical bus address section (3320) and the virtual bus address (3310) in the form of a register such as SRAM, To convert the virtual bus address (3310) to the physical bus address (3320) (4000), Performing Load and Store operations; Interconnect computing device system.

10. In paragraph 9, The above bus page table (3300) is An LRU cache that stores the most recently used address values ​​and data for faster comparison between the above bus page entries (3330) and the requested virtual bus address (3310); Bus Address Preview (Lookup Table); Binary binary order for fast traversal / search of the bus page table (3300); and A Snoop Filter field for reducing traffic between chiplet connections (3020); including at least one more of the following: Interconnect computing device system.

11. In paragraph 6, The above data mover (3140) is further comprising at least one data move channel (3150); At least one of the above data move channels (3150) Control commands and data to the bus stream connection (3230) and the bus read / write unit (3210). characterized by being allocated synchronously; and / or asynchronously; Interconnect computing device system.

12. In paragraph 7, The above bus read / write unit (3210) is Through the corresponding internal physical bus address value (3320) according to the allocation of the above data move channel (3150), As a bus master (3100), access the internal bus (3010) Characterized in that it performs bus read / write control on the above bus slave (3200). Interconnect computing device system.

13. In paragraph 6, The above bus stream (3230) is From the above chiplet connection (3020) to the above bus master (3100), A bus stream input section (3231) receiving input; and Includes a bus stream output section (3232) that outputs; Characterized by inputting / outputting data without a bus address value, Interconnect computing device system.

14. In paragraph 4, The above core part (210) is It consists of at least one single core, Cashier (240); Address translation preview section TLB (260); and Cache status checker (3160); including at least one more; Interconnect computing device system.

15. In paragraph 14, The above core part (210) is Demoted from Bus Master (3100), It is characterized by being assigned a bus address value and operating as a bus slave (3200). Interconnect computing device system.

16. In paragraph 14, The above cache unit (240) is It consists of at least one cache block line, The above cache block line is, TAG field; and Including more data fields; In case of LLC (Last Level Cache), Includes additional Valid / Dirty fields, As a bus slave (3200), a bus address value is assigned, Responds to the above cache line / block value according to the bus read, Characterized in that the cache line / block value is input / updated according to bus writing. Interconnect computing device system.

17. In paragraph 14, The above address translation preview section / TLB (260) and cache status checker (3160) are As a bus slave (3200), it is assigned commands from the bus read / write unit (3210). Data and Valid / Dirty status fields in cache / TLB, Bus Read / Write (3210) response; and Busstream (3230) response; characterized by at least one of: Interconnect computing device system.

18. In paragraph 6, The above register / HAL (230) is As a hardware abstraction layer, It consists of at least one hardware register block, Respond to the above register block value according to the internal bus read, Characterized in that the register block value is input / updated according to the internal bus write. Interconnect computing device system.

19. In paragraph 6, The above memory controller (220) As a bus slave (3200), it is assigned commands from the bus read / write unit (3210). characterized by performing memory read / write and responding to bus read / write, Interconnect computing device system.

20. In paragraph 7, At least one bus page table entry (3330) A virtual bus page address field (3310) indicating a global bus address; and It further includes an internal bus page address field (3320) indicating the physical bus address; The above global bus address entry (3310) is The bus address value corresponding to the internal bus page address field (3320) indicating the above physical bus address, Characterized by setting it arbitrarily and loading it into a volatile storage device (Store). Interconnect computing device system.

21. In paragraph 20, The above virtual bus page address (3310) is Upper translation address bits; and Contains the sub-bus page address offset / precision bits; It corresponds non-linearly and randomly to the internal physical bus address (3320). The above upper translation address bits correspond to the above bus page table entry (3330); and / or If the above entry (3330) exists between precision / offset; Characterized in that the upper conversion address is converted into the upper value of the internal physical bus address (3320). Interconnect computing device system.

22. In paragraph 20, The above internal physical bus page address (3320) is, It corresponds linearly and statically to at least one internal physical bus address line, Characterized by starting from address 0x00 and reaching the size of the internal bus address, Interconnect computing device system.

23. In paragraph 20, The above virtual bus address field (3310) is Corresponds to the physical memory page address value, The above physical memory page address value is, Characterized in that it is converted into a virtual memory page address value by TLB (260). Interconnect computing device system.

24. In paragraph 20, The above internal physical bus page address (3320) is, As a unique physical bus address shared only by the internal bus (3010) of the computing device that is not shared throughout the above chiplet connection (3020), It is characterized by the addressability of the square of the number of internal physical bus lines of 2, The addressability of the above internal bus address is: Corresponds to the internal bus slave address (3320). The above internal bus slave address (3320) is characterized by having the same space / size as the internal physical bus address line. Interconnect computing device system.

25. In paragraph 20, The above bus page table entry (3330) is Include the SET / Enable field (3340), The above SET / Enable field (3340) is Contains at least one of the following: flag bits; and precision / spare bits; Corresponding to the above flag bit value, It is characterized by determining whether to perform conversion between the virtual bus address of the bus table entry (3330) and the internal physical bus address. Interconnect computing device system.

26. In paragraph 20, The above bus page table entry (3330) is By the bus page table line / entry calculation formula (3500), The number of entries / lines is equal to the bus table precision divided by the required internal physical bus capacity. By the table size / capacity calculation formula (3600), Multiply the above number of lines / entries by (the number of bus bits with the offset removed + the extra bits), Characterized in that the size / capacity of the above bus page table is Interconnect computing device system.

27. In paragraph 20, The virtual bus page address (3310) is When Protection SWAP / Bus page table reset (5000), The upper address of the virtual bus page, excluding the offset / granularity bits, becomes invalid. The parent address of the above invalid virtual bus page is: The upper address of the above physical memory page; and the upper address of the above virtual memory page; are transferred to, Further configured to recalculate the invalid upper address in the above TLB / cache unit (240), Interconnect computing device system.

28. In paragraph 20, The above virtual bus page address (3310) is As a global bus address shared by the internal bus of the entire chiplet connection (3020), Addressability is characterized by the power of 2 (full address value - precision address value), The addressability of the above global bus address is: It is transferred to the physical memory page address, The above physical memory page address is characterized by having the same space / size as the above global bus addressability. Interconnect computing device system.

29. In paragraph 26, The size / capacity of the entire bus page table above is: If it is a 4KB memory page directory, More than the number divided by 4KB; and If it is a 64KB memory page directory, More than the number divided by 64KB; at least one of them has a random value of N (3700), It is characterized in that the above N random values ​​are shared as promised addresses in the operating system bus address table, The upper bits, excluding the offset / precision, When LATCH / Trigger occurs during bus page table address conversion, The values ​​of the bus page table entry (3330) in the above bus page table hardware register, Characterized by Write Through, Interconnect computing device system.

30. In paragraph 29, The above bus page table precision / offset is 22 bits, If the above virtual memory address system is the Intel 48-bit virtual address system, With a 9-bit address field value, The L4 address field; PDPE address field; PDE address field; and PTE upper 1 bit address field correspond to the 28-bit virtual bus page address value. The PTE lower 8-bit address field and the lower 12-bit memory page offset field correspond to a virtual bus page 20-bit offset / precision, The above bus page table precision / offset is 22 bits, If the above virtual memory address system is an ARM48-bit virtual memory address system, The PGD address field; PUD address field; PMD address field; and PTE upper 1-bit address field correspond to the 28-bit virtual bus page address value. The PTE lower 8-bit address field; and the lower 12-bit memory page offset field; are characterized in that they correspond to a virtual bus page 20-bit offset / precision. Interconnect computing device system.

31. In paragraph 14, The above cache unit (240) is As a bus slave (3200), a bus address is assigned, Read / write the bus to the destination bus slave address of the cache unit (240), Between at least two core parts (210), As a consistency method of the first cache unit (241) and the second cache unit (242), One block area of ​​the first cache unit points to its own address value, and one block area of ​​the second cache unit points to the address value of the block area of ​​the first cache unit; and A block area of ​​the second cache unit points to its own address value, and a block area of ​​the first cache unit points to the address value of the block area of ​​the first cache unit; at least one of which is Self Referenced. Characterized in that the above Cache to Cache Write Through, Interconnect computing device system.

32. In paragraph 31, The above cache unit (240) is Through the above chiplet connection (3020), Broadcasting global virtual bus address (3310) bus read / write; and, Crossover to another busmaster's busslave; and Characterized in that the bus page table (330) between the two computing devices is crossed to perform bus reading / writing. Interconnect computing device system.

33. In paragraph 14, The above cache unit (240) is As a bus slave (3200), a bus address is assigned, Read / write the bus to the destination bus slave address of the memory controller (220), As a consistency method of the cache unit (240) and the memory controller (220), One block area of ​​the first cache unit (241) points to the address value of the memory controller (220), and one block area of ​​the second cache unit (242) also points to the address value of the same memory controller (220). The internal bus (3010) is halted (HALT) / cycle consumed until the above memory controller (220) and the cache unit (240) being written to are completed. Characterized in that the above cache is written back to memory. Interconnect computing device system.

34. In paragraph 33, The above memory controller (220) Including an LRU cache (244), The above LRU cache (244) is Contains a flag bit field or a recently used address field; When reading / writing a bus with a memory controller (220) bus slave address as the destination, It is characterized by mediating between the memory and the memory controller (220) by buffering recently used address values ​​and data. Interconnect computing device system.

35. In paragraph 16, The above cash block line is, Includes a block TAG field; and a block data field; The above cache block line is, The INDEX value is determined according to the order, As a bus slave (3200), it is assigned commands from the bus read / write unit (3210). Characterized by responding to bus read / write with the stored value of the cache line / block, Interconnect computing device system.

36. In paragraph 1, At least one computing device is, Further comprising a computing device operating system (500), The above computing device operating system (500) is It is divided into at least a first operating system (501); and a second operating system (502); Through the global virtual bus address (3310) corresponding to the above physical memory bus address (3320), Characterized in that the first operating system (501) and the second operating system (502) share at least one conflicting / matching bus address value. Interconnect computing device system.

37. In paragraph 36, The above second operating system (502) is When the driver buffer is updated, by the cache resident / residence area (6100), Bus interception of the second operating system (502) is triggered, Through the interconnect link (400) protocol, Remotely set up the above bus table and notify / doorbell, The first operating system (501) remotely triggers a bus intercept; and The second operating system (502) remotely triggers a bus intercept; performs at least one of the following; Interconnect computing device system.

38. In paragraph 37, If the above interconnect link (400) protocol is CXL (Compute eXpress Link), Step (7100) of setting up and handshake the bus page via CXL.io (460); or Perform the step (7200) of setting up the bus page via CXL.mem(480); Further steps (7300) are performed to read / write bus pages and remotely intercept buses via CXL.mem (480); Perform further steps (7400) to modify some entries (3330) of the bus page or Notify / DoorBell via CXL.cache (470); Performing at least one of the steps (7500) of intercepting a remote bus by reading / writing the modified entry / existing entry (3330) through CXL.mem (480); Interconnect computing device system.

39. In paragraph 37, The above cache resident / residence area (6100) is, As the above bytecode / assembly (6100), Residing in at least one cache, Points to a graph (6600) consisting of at least one Vertex (6650); At least one of the above Vertex (6650) By the condition of the trigger / trigger (6200), After entering the upper cache (240), enter the core. Points to / performs a conventional OS interrupt entry handler (6510); and Characterized by pointing to / performing the following Vertex (6650) address value; Interconnect computing device system.

40. In paragraph 39, The above Vertex (6650) and Graph (6600) are Protected by the interconnected area, While residing in the first computing device cache unit (241) and the second computing device cache unit (242), The above virtual bus address field (3310) shares / points to the same address value, If at least one cache unit (240) is modified, The target core / computing device cache resident area is also written / updated by bus intercept and bus intercept propagation. Characterized by causing failure / exception handling during modification, Interconnect computing device system.

41. In paragraph 2, At least one computing device (200) above, It is mediated between the root complex (420) and the root complex (420) through the endpoint NTB (Non Transparency Block), The above NTB (411) is A bus slave (3200) located inside the interconnect switch (410); and At least one of a bus slave (3200) located inside the computing device (200) and via a bus stream connection (3230); The above endpoint NTB (411) is inside the switch, The internal physical address (3320) of the above bus slave (3200) corresponds to the address of the above endpoint (430) and mediates, The bus table controller (3120) of the above switch (410) serves as a bus master (3100). The virtual bus address (3310) corresponding to the above bus slave internal physical address (3320) and the target bus slave internal physical address (3320) corresponding to the virtual bus address value (3310) that matches / conflicts with the above, Characterized in that remote bus interception and routing are performed from a first computing device to a second computing device within the switch. Interconnect computing device system.

42. In paragraph 41, The above virtual bus page address (3310) is The above Host BAR (Base Address Register) has an arbitrary value and corresponds to the internal physical address (3320). The above bus slave internal physical address (3320) is It is characterized by B / D / F (Bus / Device / Function) corresponding to values, Interconnect computing device system.

43. In paragraph 42, The above host virtual bus page address value (3310) is Host's BAR2 is the upper bit and BAR1 is the lower bit; and BAR1 is determined by at least one of the upper bits and BAR2 is determined by at least one of the lower bits; The above synthesized BAR value is hashed through 64 bits, 32-bit long IPv4; 128-bit long IPv6; and In addition, it is characterized by corresponding to at least one of the IP address values ​​of bit length; Interconnect computing device system.

44. In paragraph 42, When the above NTBB / D / F values ​​(3320) and / or BAR Lookup values ​​(3310) are changed by the root complex (420) or the switch mCPU (210), By bus read / write, it is simultaneously reflected (Write-Through) in the switch (410) internal bus table (3300) and the mCPU (210) cache (240). The BAR LookUp values ​​(3310) of the physical bus page table that operates as a Lookup Table inside the table controller (3120), which is an interconnect switch (410), are simultaneously reflected (Write-Through) to B / D / F (3320). Interconnect computing device system.

45. In a bus page table conversion method (4000) in an interconnect computing device system, If a bus request remains / is entered, A step (4100) of querying whether the TLB (260) / cache (240) inside the chip (200) references the above address through the bus allocator (3130) while simultaneously traversing the bus table (3300); A step (4200) of traversing the virtual bus address section (3310) of the bus table (3300) and comparing it with the requested bus address value; How to convert bus page table.

46. ​​In paragraph 45, If the bus request remains / is not entered, Step (4600) of checking the address conversion preview section (260) / cache status checker (3160); including; How to convert bus page table.

47. In paragraph 46, If invalid / update is required after step 4600 of checking the above address conversion preview section (260) / cache status checker (3160), Step (4700) of copying / propagating a bus request to the outside of the chip (200) through a chiplet connection (3020); and Further comprising a step (4200) of traversing the virtual bus address section (3310) of the bus table (3300) and comparing it with the requested bus address value; How to convert bus page table.

48. In paragraph 47, If a snoop filter table exists in the above cache (240) / TLB (260) or the Snoop MESI protocol is used, Before step 4700, which copies / propagates the bus request to the outside of the chip (200) through the above chiplet connection (3020), Query the snoop filter table, If the above Snoop Filter field exists as a bus page table entry, A step of determining whether to generate the above traffic; further comprising: How to convert bus page table.

49. In paragraph 45 or paragraph 47, After step 4200 of traversing the virtual bus address section (3310) of the above bus table (3300) and comparing it with the requested bus address value, The corresponding address value exists / matches in the above table (3300), If the SET / Enable field does not exist; and If the above entry SET / Enable field exists and is set; at least one of Step (4300) of converting the bus table (3300) into an internal physical bus address (3320) corresponding to the virtual bus address (3310); A step (4400) in which the bus master (3100) reads / writes the internal bus slave (3200) based on the internal physical bus address (3320) using the bus controller; How to convert bus page table.

50. In paragraph 45 or paragraph 47, After the step (4100) of querying whether the TLB (260) / cache (240) inside the chip (200) references the address through the bus allocator (3130) while traversing the above bus table (3300), if the address exists in the preview / cache section, A step (4500) of performing a requested bus read / write (Cache-Flush / Write-Through) to a specific cache block / line within LLC (250) or cache (240) as a bus master (3100); How to convert bus page table.

51. In the method for setting up Protection SWAP / Bus Page Table in an interconnect computing device system (5000), If it is not a bootstrap process, A step (5600) of causing a bus write by using the address where the above arbitrary virtual bus table (3300) is located in the form of a memory virtual page directory / entry and the above arbitrary N (3700) random numbers as bus address values ​​(3310); A step (5700) of physically filling all of the above bus page tables (3300) in the bus master (3100) or virtually filling the kernel unit (500) memory page directory / entry with random values; and A step (5800) that makes all upper address bits, except for the lower address bits that accommodate the bus precision in the TLB (260) / cache (240), invalid and causes the kernel unit (500) to correctly calculate the memory physical paging address again; How to set up protection SWAP / bus page table.

52. In paragraph 51, If it is a bootstrap process, Before step 5600, which triggers bus writing by using the above random N (3700) random numbers as bus address values ​​(3310), Step (5100) of generating a Power / Wake On packet by a post chip connected to an interconnect, a Type-C connection board, or the transferred power button; A step (5200) of receiving a Wake On / Power On packet received by CXL or interconnect, a lightweight kernel part (500) at the firmware level, and N generated random numbers (3700); A step (5300) in which a lightweight kernel (500) circulates a bus master (3100) through a cache (240), a core (210), and a bus page (3300); A step (5400) in which a lightweight kernel part (500) causes a race condition and obfuscation in a multi-core (210) to create a virtual memory page table; and A step (5500) in which the core (210) that succeeds in creating the memory page table / directory first according to the race condition hides the address value of the random number N (3700) in the virtual memory page; How to set up protection SWAP / bus page table.

53. In paragraph 51 or 52, The above N random numbers (3700) are The value of converting the virtual memory page byte size to bit size divided by (full virtual memory page address - bus page offset / precision + remaining bits) has at least the number of values. It is characterized by filling in the table entry virtual address values ​​based on the start / end addresses of the above N pages. How to set up protection SWAP / bus page table.

54. In a bus interception method (6000) in an interconnect computing device system, A step (6100) of updating (Dirt) or making invalid (Invalid) the trigger area (6200) that causes a bus intercept (6000) in L3 (243), and then allowing only a specific part to reside in L3 (243) / cache (240); If the cache (240) / kernel section (500) does not change the updated / invalid area (6200), It further includes a step of permanently residing (6100) in L3 (243); When the cache (240) / kernel part (500) is changed to the updated / invalid area (6200), Further comprising a step (6500) of entering the interrupt handler (6500) of the conventional operating system (500); How to intercept a bus.

55. In a bus interception method (6600) having N vertices (6650) in an interconnect computing device system, When the cache (240) / kernel part (500) is updated / changed to the invalid area (6200), Including a step (6650) of traversing the Vertex (6650) of the above graph (6600); If there is an operating system entry handler, It further includes a step (6510) of entering the OS handler; If there is no operating system entry handler, Further comprising a step (6650) of re-entering another Vertex on the above graph; Bus interception method with N vertices.

56. In a mutually confirmatory link bus interception method (6700) in an interconnect computing device system, Between the cores (210) which are the bus intercept (6000) area and the Vertex (6650) area of ​​the graph (6600), and between the first computing device (201) and the second computing device (202) which are processors, It is characterized by being linked in a way that it mutually references virtual bus addresses (3310) that are complementary or otherwise coincident / conflicting, etc. If at least one area is modified, Further comprising a step of causing a Fault / Reset; Mutually Assurance Link Bus Intercept Method.

57. In paragraph 55 or 56, The above Graph (6600) is, Contains at least one Vertex (6650), The above Vertex (6650) is characterized in that it points to at least one other Vertex (6650) on the graph, At least one of the above Vertex (6650) Obfuscated, One-time token check area execution step; Interconnected area; actual interrupt entry function (6500) execution stage; and Further including the area execution step pointing to another Vertex (6650); How to intercept a bus.

58. In paragraph 54, The above bus interception method (6000) is If it is local inbound (6010), By reading / writing the internal bus (3010) above, a bus intercept (6000) is triggered (6200) only within the local bus (3010). The above updated / invalid area (6200) is changed, and the remaining code area / byte code area is loaded into the cache (240) due to a cache miss, etc. How to intercept a bus.

59. In paragraph 54, The above bus interception method (6000) is In case of global inbound (6020), A remote bus intercept of the target bus page table is received / triggered (6200) through the above chiplet connection (3020). The above updated / invalid area (6200) is changed, and the remaining code area / byte code area is loaded into the cache (240) due to a cache miss, etc. How to intercept a bus.

60. In paragraph 54, The above bus interception method (6000) is In case of global outbound (6030), A bus intercept (6000) is triggered (6200) within the local bus (3010) by reading / writing the internal bus (3010). The above updated / invalid area (6200) is changed, and due to a cache miss, etc., the remaining code area / byte code area is loaded into the cache (240). It is characterized by propagating / inducing (6200) a remote bus intercept of the target bus page table through the above chiplet connection (3020). How to intercept a bus.

61. In the remote bus table interception (7000) in the interconnect computing device system, From the second computing device (202) to the first computing device (201), As an act of presetting each entry (3330) that operates as a conventional DMA or Lookup Table, Step 7100 of setting / negotiating via CXL.io (460); and A step (7200) of setting the area where the own bus page table (3300) is located by writing through reading / writing the virtual / global bus address (3310) to the CXL.mem (470) between the root complex (420) and the endpoint (430) NTB; including; How to intercept remote bus tables.

62. In paragraph 61, As an act in which the second computing device (202) and the first computing device (201) are mutually consistent, Further comprising a step (7300) of inducing a remote bus intercept through a shared virtual bus address (3310) using CXL.mem (470); How to intercept remote bus tables.

63. In paragraph 61, As an act of replacing each entry (3330) that operates as a conventional DMA or Lookup Table in advance, Before / after the step (7300) that causes a bus intercept (6000) between the NTB, which is the endpoint (430) device (300), and the root complex (420), Further comprising a step (7400) of causing an update of the virtual bus address (3310) of the switch (410) bus table (3300) through CXL.cache (480); How to intercept remote bus tables.

64. In paragraph 61, As an act of mutual consistency between the first computing device (201) and the second computing device (202), as an act of consistency, Further including the step (7500) that causes the actual remote bus read / write to CXL.mem(480); How to intercept remote bus tables.

65. In a remote bus interception method using NTB (Non Transparency Block) in an interconnect computing device system, The first step of traversing the switch bus page table (3300) to find a virtual bus address (3310), which is a BAR composite value, corresponding to the internal bus physical address (3320) of the NTB B / D / F; A second round of traversing the remainder of the table to find a target virtual bus address (3310) that matches / collides with the virtual bus address value (3310), which is the BAR composite value obtained in the first round; A step of converting an address to NTB B / D / F, which is an internal physical bus address (3320) corresponding to the above-mentioned matching / conflicting virtual bus address (3310); How to intercept remote buses via NTB.

66. In Articles 45 to 65, One way of protesting, Contains at least one command configured to be executed, Computer program.

67. In paragraph 66, The above computer program, Configured to be stored and executed by hardware, A computer-readable, non-transitory recording medium.

68. In paragraph 2, The above endpoint (430) is, Force Touch; Touch Screen; USB-C board; and At least one of the GPIO buttons transferred to the above touchscreen / board; Wake On / Power On packets; and A lightweight kernel unit for generating the above N random values ​​(3700); characterized by sending it to the root complex (420). Interconnect computing device system.

69. In paragraph 6, The above chiplet connection (3020) is By means of an optical transceiver, between chips; and between chiplets; configured to be detachable; Interconnect computing device system.

70. In paragraph 1, The above chiplet connection (400) is AXI (Advanced eXtensible Interface); Quick Path Interconnect (QPI); Ultra Path Interconnect (UPI); and Further configured to use at least one of the following means: UCIe (Universal Chiplet Interconnect Express); Interconnect computing device system.

71. In paragraph 1, The above interconnect link (400) is USB-C (Universal Serial Bus Type-C); Thunderbolt; MiniSAS-HD; Peripheral Component Interconnect Express (PCIe); and Further configured to use at least one of the WBE (Wireless Bus Extension); Interconnect computing device system.

Images