System on chip, electronic equipment and model loading method

Abstract

The invention discloses a system on chip, electronic equipment and a model loading method. The system-on-chip adopts an integrated packaging structure and at least comprises a first controller and a second processor which are jointly integrated in the same package, and the first controller is in communication connection with the second processor; the first controller is used for loading model data stored in the first memory to the second memory in response to a model loading instruction sent by the second processor based on access control of the first controller to the first memory outside the package; and the second processor is used for acquiring the task data and the model data from the second memory and performing model calculation by utilizing the task data and the model data. According to the system on chip, the first controller and the second processor are jointly integrated in the same package by adopting an integrated package structure, and communication connection is realized. The physical distance and the data transmission path between the first controller and the second processor are shortened from the hardware architecture level, and the physical delay of data transmission among different devices is reduced.

Description

Technical Field

[0001] This application relates to the field of chip technology, and in particular to a system-on-a-chip, electronic device, and model loading method. Background Technology

[0002] In traditional hardware architectures, functional modules such as controllers, processors, and memory are deployed in a distributed manner. When AI model-related data is transmitted between these different modules, it relies on the central processing unit as an intermediary. This transmission method not only results in long transmission paths due to the physical distance between modules, causing significant physical delays, but also severely restricts the real-time performance of AI tasks involving large-scale model data. Summary of the Invention

[0003] In view of the above problems, this application provides a system-on-a-chip, an electronic device, and a model loading method.

[0004] To solve the above-mentioned technical problems, this application proposes the following solution:

[0005] In a first aspect, this application provides a system-on-a-chip (SoC) that adopts an integrated packaging structure and includes at least a first controller and a second processor integrated in the same package, wherein the first controller and the second processor are communicatively connected.

[0006] The first controller is used to load model data stored in the first memory into the second memory based on its own access control to the first memory outside the package, in response to the model loading instruction sent by the second processor.

[0007] The second processor is used to retrieve task data and model data from the second memory, and to perform model calculations using the task data and model data.

[0008] In some embodiments, the system-on-a-chip further includes a third switch, which is communicatively connected to the first controller, the second processor, and the central processing unit. The third switch is used to realize data interaction between the first controller and the second processor, so that the data does not need to be relayed through the central processing unit.

[0009] In some embodiments, the system-on-a-chip also integrates a power control module, which is used to turn off the main frequency and power supply of the second processor and maintain the power supply of the second memory when no model calculation request from the second processor is detected within a continuous preset time period.

[0010] In some embodiments, the first controller is further configured to determine the preloaded target model based on the model invocation logic of the target application when the main frequency and power of the second processor are off and the second memory is powered on, and to read the initial fragment of the inference startup phase in the target model from the first memory and transfer the initial fragment to the second memory for storage.

[0011] In some embodiments, the first controller is further configured to: count the historical startup frequency of each model, filter out models whose historical startup frequency meets the preset calling conditions, and use the models as target models to be preloaded;

[0012] For each target model to be preloaded, extract the input layer data fragments that the target model depends on during the inference startup phase, and determine the input layer data fragments as the initial fragments for preloading;

[0013] When the remaining space in the second memory is insufficient to store the initial fragments to be preloaded, delete the preloaded data corresponding to the models that have not been invoked for a preset time period.

[0014] Secondly, this application provides an electronic device, comprising:

[0015] System-on-a-chip, central processing unit, first memory and second memory;

[0016] The system-on-a-chip includes at least a first controller and a second processor, and the system-on-a-chip is used for at least:

[0017] The first controller loads the model data from the first memory into the second memory;

[0018] The second processor performs neural network inference calculations based on the model data and task data in the second memory to obtain the calculation results;

[0019] The calculation results are transmitted to the central processing unit.

[0020] In some embodiments, the second memory is partitioned into shared storage space for storing flag bit states. These flag bit states are used to restrict conflicting operations on the target address by another party through state locking when the second processor or the first controller performs an operation on the target address.

[0021] In some embodiments, the flag state includes: a first read tag corresponding to the first controller, a second read tag corresponding to the second processor, and a write tag for managing global write operations;

[0022] Specifically, when the first read tag is activated, the first controller is restricted from writing to the target address; when the second read tag is activated, the second processor is restricted from writing to the target address; and when the write tag is activated, the activation states of the first and second read tags are simultaneously restricted.

[0023] Thirdly, this application provides a model loading method, the method including:

[0024] Obtain a model loading request from the target application. The computation request is used to trigger the second processor of the system-on-chip to send a model loading instruction to the first controller of the system-on-chip.

[0025] After receiving the model loading instruction, the first controller sends the storage location of each model in the first memory to the second processor.

[0026] The second processor sends instructions to the first controller based on the storage location and computation request, triggering the first controller to read the model data corresponding to the computation request from the first memory and store the model data in the second memory.

[0027] In some embodiments, when the first controller writes model data to the target address of the second memory based on the model loading instruction, it triggers a global write flag lock and prevents the second processor from performing a read operation on the target address.

[0028] When the second processor reads model data from the target address of the second memory, if the global write flag of the target address is not locked, the model data reading is performed.

[0029] By employing the above-described technical solution, the technical solution provided in this application has at least the following advantages:

[0030] The system-on-a-chip (SoC) of this application integrates a first controller and a second processor within a single package and establishes communication connectivity. This shortens the physical distance and data transmission path between the first controller and the second processor at the hardware architecture level, reducing physical latency in data transmission between different devices. Simultaneously, the first controller can directly respond to model loading commands sent by the second processor based on its access control to the external first memory, loading model data stored in the first memory into the second memory. The entire data loading process does not rely on a central processing unit for relay, further reducing intermediate data transmission steps and lowering data loading latency.

[0031] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0032] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0033] Figure 1 This paper shows a schematic diagram of the structure of a system-on-a-chip provided in an embodiment of this application;

[0034] Figure 2 This illustration shows a structural schematic diagram of an electronic device provided in an embodiment of this application;

[0035] Figure 3 This illustration shows a schematic diagram of mutual access between a first controller and a second processor according to an embodiment of this application;

[0036] Figure 4 This illustration shows a schematic diagram of a shared storage space data scheduling method provided in an embodiment of this application;

[0037] Figure 5 A schematic flowchart of a model loading method provided in an embodiment of this application is shown. Detailed Implementation

[0038] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0039] In the embodiments of this application, the terms "first," "second," etc., do not have a logical or temporal dependency, nor do they limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0040] In this application, the term "at least one" means one or more, and the term "multiple" means two or more.

[0041] It should also be understood that the term “if” can be interpreted as “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrase “if determination…” or “if detection [the stated condition or event]” can be interpreted as “when determination…” or “in response to determination…” or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0042] According to at least one embodiment of the present application, the system-on-a-chip may include: a first controller and a second processor integrated in the same package, wherein the first controller and the second processor are communicatively connected.

[0043] The first controller is used to load model data stored in the first memory into the second memory based on its own access control to the first memory outside the package, in response to the model loading instruction sent by the second processor.

[0044] The second processor is used to retrieve task data and model data from the second memory, and to perform model calculations using the task data and model data.

[0045] For ease of understanding, combined with Figure 1 Please provide an explanation.

[0046] A system-on-a-chip (SoC) can be, for example, an integrated circuit module in a single package, such as a 12mm × 12mm SoC. The SoC includes a first controller and a second processor integrated within the same package. The first controller and the second processor communicate with each other via inter-die interconnects or single-die direct memory access interconnects. Inter-die interconnects refer to the use of high-density interconnect structures to create direct communication paths across dies when the first controller and the second processor are integrated on different dies. Single-die direct memory access interconnects refer to the use of integrated direct memory access modules to create direct communication paths when the first controller and the second processor are integrated on the same die.

[0047] The first controller in this application can be, for example, an NVMe controller. As a bridge between external storage and on-chip computing, it integrates storage control logic supporting interface protocols such as Peripheral Component Interconnect Express (PCIe) and Universal Flash Storage (UFS). This logic can adapt to the communication requirements of the first memory, ensuring the compatibility and stability of data interaction. Simultaneously, relying on support for high-speed interface protocols, the controller can directly establish a communication link with the externally packaged first memory, efficiently reading AI model data stored in the first memory.

[0048] The secondary processor is the core of AI computation, typically a Neural Processing Unit (NPU) or other unit with efficient parallel computing capabilities. It simultaneously retrieves task data and model data from the secondary memory, and then utilizes its hardware architecture specifically designed for AI computation (such as dedicated circuits for matrix multiplication and convolution operations) to execute the AI ​​model's computational process based on these two types of data. Taking image tasks as an example, it can complete the inference process from feature extraction and feature classification to result output through a convolutional neural network model.

[0049] When the second processor needs to perform model calculations, it sends a model loading instruction to the first controller. After receiving the instruction, the first controller reads the corresponding model data from the first memory through the external storage interface and loads it into the second memory.

[0050] Overall, the system-on-a-chip (SoC) workflow forms an efficient closed loop. The external first memory permanently stores the AI ​​model. When the second processor sends a model loading command, the first controller loads the model data into a second memory, independent of and located close to the SoC. The second processor then retrieves the model and task data from this second memory and performs AI calculations. This architecture effectively reduces the physical latency caused by data transfer across devices. Simultaneously, leveraging its independent access control over the external first memory, the first controller can transfer the model data stored in the first memory to the second memory upon receiving a model loading command from the second processor. The entire data loading process does not require a central processing unit (CPU) as an intermediary, further reducing intermediate steps in data transmission and effectively minimizing model data loading latency.

[0051] According to PCIe bus rules, all PCIe devices (such as the first controller) must be mounted under the CPU's root complex, and communication must be relayed through the CPU. Address allocation and permissions are uniformly managed by the CPU. Even if the first controller and the second processor are integrated into a 12mm×12mm area using a single package, without a third switch, they still need to forward data through the CPU. This leads to the following problems: First, data transmission latency increases. The model loading latency of traditional CPU relay is about 100 microseconds, while in typical real-time AI inference scenarios on the terminal side, the total time from model loading to inference completion must be controlled within 200 microseconds. At this time, the latency caused by CPU relay accounts for more than 50%, directly restricting the real-time performance of inference tasks. Second, CPU resources are consumed in large quantities. In a system based on a mainstream x86 architecture processor, the simple data transfer process can consume more than 40% of the CPU computing power, causing significant lag when the system runs multiple tasks in parallel (such as simultaneously handling user interface rendering and background process scheduling).

[0052] To address this, in some embodiments, a third PCIe switch with multi-port forwarding capabilities is integrated within the system-on-a-chip (SoC) package. The third switch establishes communication connections with the first controller, the second processor, and the central processing unit (CPU) via PCIe interfaces. Its operation mechanism, coordinated with the initialization configuration and data forwarding process, is as follows: During initialization, the CPU allocates dedicated PCIe address spaces to the first controller and the second processor, and configures this address mapping relationship in the third switch's internal routing table. When the second processor triggers a model loading instruction, the first controller generates a PCIe transaction packet containing the target address (the second processor's PCIe device address) and sends it directly to the third switch via the on-chip PCIe link. The third switch, based on the pre-stored address mapping relationship, forwards the transaction packet directly to the corresponding PCIe port of the second processor, bypassing the CPU root complex entirely. This overcomes the hardware topology limitation that prevents direct communication between devices within the package without additional routing components.

[0053] Taking facial recognition as an example, the transmission path of model data from the first controller to the second memory is shortened to three hops: first controller, third exchange, second processor, and second memory. The data transmission latency can be reduced from about 100 microseconds when the CPU is involved to less than 20 microseconds. Moreover, the CPU can process other system tasks in parallel (such as background process scheduling and user interface rendering), avoiding the situation where the CPU has excessive computing power due to the data transfer work.

[0054] In addition, when the central processing unit needs to send control commands to the first controller or the second processor, such as the initialization configuration of the first controller during system startup or the computing power mode setting of the second processor, the third switch also supports communication between the CPU and the first controller and the second processor, so as to ensure the normal operation of the overall system control and management logic while ensuring AI tasks.

[0055] In some embodiments, the integrated packaging structure of this application also integrates a power consumption control module. The function of the power consumption control module is to optimize energy efficiency during the idle period of the AI ​​task. The specific working mechanism is as follows: The power consumption control module monitors the model calculation request status of the second processor in real time. A preset continuous period of no request is set in advance through system configuration, such as 30 seconds commonly seen in terminal devices. When it is detected that the second processor has not received any AI model calculation request for 30 consecutive seconds (such as not triggering AI functions such as portrait unlocking or voice assistant wake-up), the power consumption control module actively shuts down the main frequency clock signal and power supply circuit of the second processor, so that the second processor enters a deep sleep state to reduce ineffective power consumption. At the same time, the module maintains the power supply of the second memory. This is because the second memory serves as a key data temporary storage carrier when subsequent AI tasks are started, and maintaining its power supply can avoid the initialization delay after the memory is powered on again.

[0056] When the second processor is in a sleep state with its main frequency and power off, and the second memory remains powered on, the first controller in this application initiates a model preloading process to achieve rapid wake-up of AI tasks. The core logic of this process is based on the model calling patterns of the target application. First, the first controller uses a built-in statistics module to record the historical launch frequency of AI models associated with each AI application (such as a facial recognition unlock application or a real-time translation application) on the terminal device. For example, it counts the number of launches of each model in the past 7 days and presets a calling condition (such as a launch frequency of ≥50 times in the past 7 days, or an average launch frequency of ≥5 times per day). Models that meet this condition are selected as target models to be preloaded. These models are usually core models that users use frequently, and preloading their key data can improve the response speed when the user triggers the system.

[0057] For each selected target model to be preloaded, the first controller further analyzes the model's inference startup logic and extracts the input layer data segments that it must rely on during the inference startup phase as the initial segments for preloading. For example, for a convolutional neural network model used for facial recognition unlocking, its input layer data segments include image preprocessing parameters, first-layer convolutional kernel weight data, and dimensional configuration information of the input feature map. These data are the core content that the model needs to call first when starting inference. Preloading only this part of the data can ensure fast startup while minimizing the occupation of the second memory space.

[0058] If, during the preloading process, the first controller detects that the remaining space in the second memory is insufficient to store all the initial fragments to be preloaded (for example, if the total capacity of the second memory is 2GB, the stored fragments occupy 1.8GB, and the new fragments to be preloaded require 0.3GB), then the space management mechanism is triggered. That is, the first controller queries the last call time corresponding to each preloaded data, deletes the preloaded data corresponding to models that have not been called by any AI task for a preset period (e.g., 90 days), and releases the storage space to accommodate the new target model initial fragments. This ensures the preloading needs of high-frequency models while preventing the second memory space from being occupied by invalid data for a long time.

[0059] An electronic device according to one embodiment of this application includes:

[0060] System-on-a-chip, central processing unit, first memory and second memory;

[0061] The system-on-a-chip includes at least a first controller and a second processor, and the system-on-a-chip is used for at least:

[0062] The first controller loads the model data from the first memory into the second memory;

[0063] The second processor performs neural network inference calculations based on the model data and task data in the second memory to obtain the calculation results;

[0064] The calculation results are transmitted to the central processing unit.

[0065] For ease of understanding, combined with Figure 2 Please provide an explanation.

[0066] The electronic device in this embodiment can be applied to terminal devices (such as computers, tablets, mobile phones, etc.) that support real-time AI inference tasks. The hardware architecture of the electronic device in this embodiment includes four main modules: a system-on-a-chip, a central processing unit, a first memory, and a second memory. The specific descriptions of each module are as follows.

[0067] To enable PCIe bus connectivity and data exchange between the central processing unit (CPU) and other hardware modules, and to ensure the coordinated operation of the electronic device's hardware architecture, the CPU in this embodiment employs a processor with PCIe root complex functionality, such as an x86-based Core Ultra processor or an ARM-based Snapdragon 8 Gen4 processor. In the electronic device, the CPU is primarily responsible for operating system scheduling, peripheral management, and application-layer processing of the final calculation results (such as feeding the calculation results back to the user interface).

[0068] The primary memory is a large-capacity storage device independent of the system-on-chip (SoC) package. It can be a solid-state drive (SSD) supporting the PCIe protocol, universal flash storage (UFS), or similar devices. The core function of the primary memory is to store various AI model data long-term, such as the FaceNet model for facial recognition and the lightweight Transformer model for real-time translation. The storage capacity of the primary memory is configured according to the AI ​​functional requirements of the electronic device, typically ranging from 512GB to 4TB. Communication between the primary memory and the SoC is relayed through the Platform Controller Hub (PCH). An example transmission path is: primary memory, platform path hub, SoC interface, primary controller, ensuring stable reading of model data.

[0069] The second memory is a high-speed storage device deployed close to the on-chip system, independent of the system-on-chip package. Specifically, it can be static random access memory, HyperRAM, or low-latency double data rate synchronous dynamic random access memory (Low-Latency DDR, LPDDR). The capacity of the second memory is typically configured from 512MB to 2GB. Although its capacity is smaller than the first memory, this type of memory offers advantages such as low access latency and fast read / write speeds. Its core function is to temporarily store critical data during AI inference, specifically including two categories: one is model data loaded from the first memory, covering core information such as model weight parameters and network layer structure; the other is the raw data of the task to be processed, such as image pixel data for facial recognition and voice signal data for voice interaction. Temporarily storing data in the second memory allows the second processor to retrieve data locally during subsequent calculations, avoiding latency caused by frequent access to the external, low-speed first memory, and also reduces power consumption during data transfer across devices, thus meeting the needs of efficient collaboration between the first controller and the second processor.

[0070] It should be noted that the deployment of the second memory is not limited to forms independent of the system-on-a-chip (SoC) package. In scenarios with extremely high requirements for data transmission latency and where the on-chip package space is suitable, it can also be integrated with the first controller and the second processor within a single SoC package. This on-chip integration design can further shorten the physical transmission path between the second processor and the second memory, compressing data access latency from short-distance off-chip transmission to on-chip direct connection. Even when packaged on-chip, the second memory will still preferentially use high-speed media such as static random access memory (SRAM), and its capacity configuration is usually maintained in the range of 512MB to 2GB. This design avoids excessive on-chip package space occupied by excessive capacity, while meeting the temporary storage needs of model data and raw task data during AI inference, further enhancing the collaborative efficiency among the first controller, the second processor, and the second memory, and minimizing power consumption and latency issues caused by cross-device data interaction.

[0071] The system-on-a-chip (SoC) adopts an integrated packaging structure, internally integrating at least a first controller and a second processor. The first controller integrates storage control logic supporting interface protocols such as PCIe and UFS, enabling it to directly establish a high-speed communication link with the external first memory, forming the core connection component between external storage and on-chip computing. Simultaneously, this controller possesses address decoding capabilities, such as integrating an AHB decoder for address matching and slave device selection, accurately locating the storage address of model data within the first memory. Furthermore, since AI models involve significantly more data than traditional models, the first controller, as the core component directly accessing the first memory and undertaking the task of loading model data, must possess efficient external storage access capabilities to successfully read the massive amounts of AI model data from the first memory and load it into the second memory, thus meeting the loading and usage requirements of AI models. The second processor can be a neural network processor with efficient parallel computing capabilities, integrating dedicated computational circuits such as matrix multipliers and convolution operation units, capable of efficiently executing various computational logics of the neural network model.

[0072] Electronic devices process AI tasks through the following three steps:

[0073] The first step involves the first controller loading model data. When the electronic device triggers an AI task (such as a user activating facial recognition unlock), the second processor of the on-chip system sends a model loading command to the first controller. Upon receiving this command, the first controller first retrieves the correspondence between all AI models and their storage addresses in the first memory using its own storage control logic, and then transmits this correspondence to the second processor. The second processor selects the target model from the correspondence based on the current AI task requirements and then sends the target model's identifier information back to the first controller. Based on the returned model identifier, the first controller establishes communication with the first memory via the PCIe / UFS interface, locates the storage address of the target model, reads the model data at the corresponding address, and transmits the model data to the second memory for temporary storage via the on-chip high-speed interconnect link (such as inter-die interconnect or single-die DMA interconnect). An example of the transmission path is: first memory, PCH, PCIe interface, first controller, AHB Decoder, second memory. In this embodiment, after the first controller reads the model data, it can directly transmit it to the second memory through the on-chip high-speed interconnect link. During the model selection process, the second processor interacts directly with the first controller without the need for the CPU to participate in the intermediate process, which reduces the number of data transfer nodes and the number of times data is transferred. Therefore, the delay caused by multiple data transfers in the traditional architecture can be avoided.

[0074] The second step involves the second processor performing neural network inference calculations. After confirming that the model data has been loaded into the second memory, the second processor simultaneously retrieves two types of data from the second memory: one is the loaded model data, which serves as the basis for the inference calculation rules; the other is the raw data of the task to be processed, such as the pixel data of the user's facial image captured by the camera in a portrait unlocking scenario. Subsequently, the second processor performs neural network inference calculations on the task data based on the model data, such as performing feature extraction and feature classification on the facial image data in sequence, and finally obtains the inference calculation result, such as the judgment result of successful or unsuccessful matching.

[0075] The third step involves transmitting the calculation results to the central processing unit (CPU). After completing the inference calculation, the second processor transmits the inference calculation results to the CPU via the PCIe link between the on-chip system and the CPU. Upon receiving the results, the CPU executes subsequent operations according to the application logic of the electronic device. For example, in a facial recognition unlock scenario, if the result is a successful match, the CPU controls the screen to unlock and switches to the user's desktop; if the result is a failed match, it controls the display of an unlock failure message. In this process, the CPU is only responsible for the application-level processing of the results and does not need to participate in the earlier model loading and inference calculations, effectively reducing CPU resource consumption.

[0076] In some embodiments, a shared storage space is partitioned within the second memory. This space serves as a central hub for data interaction and coordination between the first controller and the second processor, storing flag states. The core purpose is to resolve potential conflicts that may arise when the first controller and the second processor perform operations on the same target address in the second memory. For example, when the first controller writes model data, the second processor may misread incomplete or corrupted data, or both may write simultaneously, leading to data overwriting.

[0077] like Figure 3 As shown, in the overall hardware interaction system, the third switch is bidirectionally connected to both the first controller and the second processor, providing a high-speed data forwarding path for both and supporting low-latency instruction transmission and data interaction between the first controller and the second processor. The shared storage space is located between the first controller and the second processor, interacting bidirectionally with both to achieve hardware-level control over data access conflicts, ensuring the integrity of model data and task data read / write. Simultaneously, the first controller communicates bidirectionally with the first memory used for long-term storage of AI model data via the system bus, enabling efficient reading of various types of AI model data stored in the first memory. The second processor communicates bidirectionally with the second memory used for temporary storage of key AI inference data via the system bus, enabling rapid access to the model data and original task data required during inference. This multi-component collaborative architecture design, from the dimensions of data transmission efficiency, access conflict control, and storage layer adaptation, jointly provides hardware support for AI model loading and real-time inference tasks, ensuring that AI tasks can be executed with low latency and high reliability.

[0078] From the perspective of the shared storage space partitioning logic, its physical address range corresponds to the core data storage area of ​​the second memory (used to store model data and task data), and flag bits are configured according to the target address block. For example, the core data storage area of ​​the second memory is divided into several 4KB target address blocks (a common data block granularity adapted to AI model data). Each target address block corresponds to an independent set of flag bits, including a first read tag, a second read tag, and a write tag. All flag bits are stored in the shared storage space in binary form, with each set of flag bits occupying 3 binary bits, corresponding one-to-one with the three tags, where 1 indicates an active state and 0 indicates an inactive state. This method of implementing independent flag bit configuration with minimal storage overhead can significantly save storage space and achieve fast switching of flag bit states through hardware logic, thus adapting to the high-speed access characteristics of the second memory.

[0079] The first read tag is a flag that specifically controls the access permissions of the first controller to a target address block. Its activation and deactivation correspond to different operational restrictions: When the first controller needs to read data from a target address block, such as reading a temporarily stored initial model fragment, it first activates the first read tag corresponding to that target address block through hardware logic (setting the flag to 1). At this time, the hardware automatically restricts the first controller's write operations to that target address block. That is, the first controller cannot write new data to the target address block it is currently reading, avoiding inconsistencies caused by reading and writing simultaneously. After the first controller completes the read operation, it immediately releases the first read tag (setting the flag to 0), removing the restriction on its own write operations and ensuring that new data can be written to that address block normally afterwards, such as updating preloaded model fragments. For example, when the first controller reads data from the 0x1000-0x1FFF target address block in the second memory, the first read tag is activated. If the first controller then attempts to write new speech model parameters to that address block, the hardware directly intercepts the write request until the read operation is completed and the first read tag is released, after which the write operation can be executed normally.

[0080] The functional logic of the second read tag is the same as that of the first read tag, but the controlled object is the second processor: When the second processor needs to read data from a target address block, such as reading model data and task data, it activates the second read tag corresponding to that address block (flag bit set to 1). At this time, the hardware restricts the second processor from writing to that target address block to prevent the second processor from accidentally writing data during the reading process and destroying the original data required for inference. After the second processor completes the reading, such as loading the data into its own computing unit, the second read tag is automatically released (flag bit set to 0), and the second processor can then perform write operations on that address block, such as writing intermediate results generated during inference. Taking the human face recognition inference scenario as an example, when the second processor reads data from the target address block 0x2000-0x2FFF, the second read tag is activated. If the second processor attempts to write the intermediate calculation results of the convolutional layer to that address block, the hardware will intercept the write request to prevent the original image data from being overwritten and to ensure that subsequent inference calculations are based on the complete original data.

[0081] The write tag is the highest priority flag, used to manage all write operations to the target address block and to synchronize the states of the two read tags. Its core function is to avoid conflicts between write and read operations, as well as conflicts between multiple write operations: When either party (the first controller or the second processor) needs to perform a write operation to a target address block (e.g., the first controller writes model data read from the first memory to the address block, or the second processor writes inference intermediate results to the address block), the write tag corresponding to that address block is activated first (flag bit set to 1). At this time, the activation states of the first and second read tags are synchronously restricted. That is, regardless of whether the two read tags are currently inactive, they cannot be activated, and if a read tag is already activated (e.g., the second processor is reading data from that address block), a write wait mechanism is triggered, and the write operation can only be executed after the read operation is completed and the read tag is released. When the write operation is completed, the write tag is automatically released (flag bit set to 0), and the activation restrictions of the two read tags are also lifted, restoring normal access permissions. For example, when the first controller needs to write data to the target address block 0x3000-0x3FFF, it first activates the write tag of the address block. If the second processor is preparing to read the address block at this time (the second read tag is not activated), the activation of the second read tag is prohibited until the first controller completes the write operation and the write tag is released. If the second read tag is activated at this time (the second processor is reading the old data of the address block), the first controller will enter a waiting state. After the second processor completes the reading and the second read tag is released, the write operation will be performed to ensure that the newly written model data will not interfere with the ongoing read operation, and at the same time avoid the second processor reading incomplete data that has been partially written.

[0082] like Figure 4As shown, the flag status enables coordinated operation between the first controller and the second processor to avoid conflicts through the following process. During the model loading phase, after the second processor sends a model loading command to the first controller, the first controller determines that model data needs to be written to the target address block 0x4000-0x4FFF in the second memory. At this time, the first controller first activates the write tag for this address block; hardware synchronously restricts the activation of the first and second read tags to prevent the second processor from reading this address block at this time. Subsequently, the first controller reads model data from the first memory through the PCIe interface and writes it to the address block 0x4000-0x4FFF. After the write operation is completed, the first controller releases the write tag. During the inference preparation phase, after the second processor detects the release of the write tag, it activates the second read tag for the address block 0x4000-0x4FFF, and simultaneously reads model data from this address block and task data from the address block 0x5000-0x5FFF. At this time, the activation of the second read tag restricts the second processor's write operation to the address block 0x4000-0x4FFF, ensuring that the model data is read completely. During the inference execution phase, after the second processor completes data reading, it releases the second read tag. Subsequently, it performs inference calculations based on the read model data and task data. If intermediate results need to be stored, they are written to the address block 0x6000-0x6FFF (where no core data is stored). At this time, the write tag of this address block is activated, and the read tag activation is restricted to prevent intermediate results from being misread.

[0083] The aforementioned coordinated flag state process achieves precise synchronization between the first controller and the second processor in data read and write operations. Specifically, during the model loading phase, activating the write tag restricts read operations to prevent incomplete data from being read while being written; during the inference preparation phase, activating the read tag restricts write operations to ensure that model data is not tampered with during reading; and during the inference execution phase, tag control of the intermediate result storage area prevents incomplete intermediate data from being misread. This coordinated logic based on flag state switching eliminates read and write conflicts at the hardware level, ensuring the integrity and accuracy of data throughout the entire loading, reading, and processing process. Simultaneously, the hardware-level state switching response is rapid, adapting to the core requirements of real-time AI inference tasks for low latency and high reliability.

[0084] Figure 5 A schematic flowchart of a model loading method according to one embodiment of this application is shown, as follows: Figure 5 As shown, the method includes:

[0085] S501. Obtain a model loading request from the target application. The calculation request is used to trigger the second processor of the system-on-chip to send a model loading instruction to the first controller of the system-on-chip.

[0086] S502. After receiving the model loading instruction, the first controller sends the storage location of each model in the first memory to the second processor.

[0087] S503. The second processor sends an instruction to the first controller based on the storage location and the calculation request, triggering the first controller to read the model data corresponding to the calculation request from the first memory and store the model data in the second memory.

[0088] To make it easier to understand, each of the above steps will be explained separately.

[0089] In step S501, the trigger scenario for obtaining the model loading request from the target application is commonly seen in the AI ​​function invocation process of terminal devices. For example, when a user opens a facial recognition unlock application, starts a real-time voice translation function, or wakes up a smart assistant to perform a semantic understanding task on an AIPC, the target application (such as a facial recognition unlock app or a voice translation program) sends a model loading request to the on-chip system. This request is essentially an instruction carrying a model identifier and a task type. The model identifier is used to specify the AI ​​model to be loaded (such as a facial feature extraction model or a speech recognition model), and the task type is associated with the range of model data required for inference (such as only loading the feature extraction layer parameters of the model, or loading a complete large language model fragment). When the on-chip system receives this request, it triggers the second processor to send a model loading instruction to the first controller.

[0090] Step S502 focuses on the response logic of the first controller to the load command. As the core hub connecting the on-chip system and the first memory, the first controller, upon receiving the model load command from the second processor, first retrieves and organizes the mapping relationship between all AI models and the storage addresses in the first memory through its built-in memory management module. Then, it transmits this mapping relationship to the second processor via the on-chip high-speed interconnect link (such as die-to-die interconnect or single-die DMA interconnect). After the second processor selects the target model according to the current AI task requirements and feeds back the model's identification information to the first controller, the first controller then executes the subsequent model data reading operation based on the feedback.

[0091] In step S503, after receiving the storage location information sent by the first controller, the second processor, in conjunction with the computation request of the target application in step S501, further determines the range of model data to be read. For example, if the storage location information shows that a large language model occupies the address range 0x0001_0000-0x0008_FFFF in the first memory, and the computation request only requires the initial data for the inference startup phase, the second processor selects the first 512MB of input layer weights and network configuration data in this address range as the content to be loaded. Subsequently, the second processor sends a targeted loading instruction to the first controller, specifying the target address of the data to be loaded, the specific reading range of the first memory, and the priority of data transmission (e.g., AI inference tasks have a higher priority than ordinary data read / write). After receiving the instruction, the first controller initiates the reading operation on the first memory, that is, reading model data from the target address range of the first memory through the PCIe interface.

[0092] In some embodiments, when the first controller writes model data to the target address of the second memory based on the model loading instruction, it simultaneously triggers a global write flag lock and performs access conflict control through a preset shared flag in the second memory. That is, when writing model data to the target address of the second memory, the write flag of the corresponding address is set to 1, prohibiting the second processor from performing a read operation on that address at this time, thus preventing the misreading of partially written dirty data. After all the model data has been written to the second memory, the first controller releases the global write flag and the shared flag and sends a loading completion signal to the second processor. At this point, if the second processor needs to read model data from the target address, it can perform the read operation normally because the global write flag has been unlocked, and then perform AI inference calculations in conjunction with the task data. This concludes the entire model loading process.

[0093] In summary, this application solves the high latency problem of AI model loading and inference in traditional architectures by constructing a direct communication architecture of the first controller, the second processor and the third switch within the system-on-a-chip, combined with a hardware-level access conflict control mechanism for shared storage space, and a hierarchical storage design of the first memory (large-capacity long-term storage) and the second memory (high-speed temporary storage). It also eliminates the risk of data read and write conflicts from the hardware level, reduces the power consumption of data transmission across devices, and achieves efficient collaboration between the first controller and the second processor.

[0094] It is understood that, in order to achieve the functions in the above embodiments, the computer device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0095] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] In a typical configuration, the device includes one or more processors (CPUs), memory, and a bus. The device may also include input / output interfaces, network interfaces, etc.

[0097] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0098] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A system-on-a-chip, characterized in that, The system-on-a-chip adopts an integrated packaging structure, including at least a first controller and a second processor integrated in the same package, wherein the first controller and the second processor are communicatively connected. The first controller is used to load model data stored in the first memory into the second memory in response to a model loading instruction sent by the second processor, based on its own access control to the first memory outside the package; The second processor is used to obtain task data and model data from the second memory, and to perform model calculations using the task data and the model data.

2. The system-on-a-chip according to claim 1, characterized in that, The system-on-a-chip also includes a third switch, which is communicatively connected to the first controller, the second processor, and the central processing unit. The third switch is used to realize data interaction between the first controller and the second processor, so that the data does not need to be relayed through the central processing unit.

3. The system-on-a-chip according to any one of claims 1-2, characterized in that, The system-on-a-chip also integrates a power consumption control module, which is used to turn off the main frequency and power supply of the second processor and maintain the power supply of the second memory when no model calculation request of the second processor is detected within a continuous preset time period.

4. The system-on-a-chip according to claim 3, characterized in that, The first controller is further configured to, when the main frequency and power of the second processor are off and the second memory is powered on, determine the preloaded target model based on the model invocation logic of the target application, read the initial fragment of the inference startup phase in the target model from the first memory, and transfer the initial fragment to the second memory for storage.

5. The system-on-a-chip according to claim 4, characterized in that, The first controller is also used for: The historical launch frequency of each model is counted, and models whose historical launch frequency meets the preset calling conditions are selected as target models to be preloaded. For each target model to be preloaded, the input layer data fragments that the target model depends on during the inference startup phase are extracted, and the input layer data fragments are determined as the initial fragments for preloading. When the remaining space in the second memory is insufficient to store the initial fragment to be preloaded, the preloaded data corresponding to the model that has not been invoked for a preset time period is deleted.

6. An electronic device, characterized in that, include: System-on-a-chip, central processing unit, first memory and second memory; The system-on-a-chip includes at least a first controller and a second processor, and the system-on-a-chip is at least used for: The first controller loads the model data from the first memory into the second memory; The second processor performs neural network inference calculations based on the model data and task data in the second memory to obtain the calculation results; The calculation results are transmitted to the central processing unit.

7. The electronic device according to claim 6, characterized in that, The second memory is divided into shared storage space for storing flag bit states. These flag bit states are used to restrict conflicting operations on the target address by the other party through state locking when the second processor or the first controller performs an operation on the target address.

8. The electronic device according to claim 7, characterized in that, The flag state includes: a first read tag corresponding to the first controller, a second read tag corresponding to the second processor, and a write tag used to manage global write operations; Specifically, when the first read tag is activated, the first controller is restricted from writing to the target address; when the second read tag is activated, the second processor is restricted from writing to the target address; and when the write tag is activated, the activation states of the first read tag and the second read tag are simultaneously restricted.

9. A model loading method, characterized in that, The method includes: Obtain a model loading request from the target application, the computation request being used to trigger the second processor of the system-on-a-chip to send a model loading instruction to the first controller of the system-on-a-chip; After receiving the model loading instruction, the first controller sends the storage location of each model in the first memory to the second processor; The second processor sends an instruction to the first controller based on the storage location and the computation request, triggering the first controller to read the model data corresponding to the computation request from the first memory and store the model data in the second memory.

10. The method according to claim 9, characterized in that, The method further includes: When the first controller writes model data to the target address of the second memory based on the model loading instruction, it triggers a global write flag lock and prevents the second processor from performing a read operation on the target address. When the second processor reads model data from the target address of the second memory, if the global write flag of the target address is not locked, the reading of the model data is performed.

Images