Data processing method and device

Abstract

A data processing method, applied to the field of artificial intelligence. The method comprises: acquiring a first identifier corresponding to a first model parameter, wherein the first model parameter is a model parameter for model reasoning to be performed and indicated in a first model reasoning request triggered by a first instance; on the basis of a preset mapping relationship, determining a first virtual address corresponding to the first identifier, wherein the mapping relationship comprises a plurality of identifiers and virtual addresses corresponding to the identifiers, physical spaces corresponding to the virtual addresses store model parameters, different model parameters correspond to different identifiers, and the physical spaces are located in a shared memory; and acquiring the first model parameter from a physical space corresponding to the first virtual address, and on the basis of the first model parameter, performing model reasoning indicated by the first model reasoning request. In the present application, access to a shared memory can be performed and model reasoning can be performed on the basis of an access result without interaction and cooperation between instances.

Description

Data processing method and apparatus thereof

[0001] The present application claims priority from the Chinese patent application No. 202410835603.7 filed on June 25, 2024, and entitled "A data processing method and apparatus thereof", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and apparatus thereof. BACKGROUND

[0003] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, enabling machines to have perception, reasoning and decision-making functions.

[0004] Deep learning is an algorithm that uses a multi-layer artificial neural network as an architecture to learn the representation of data, and is divided into two processes: training and inference. The training process is to search for the parameters of the specified deep learning model with the best prediction effect under the given data set by the computer according to a certain optimization algorithm, and the inference process is to analyze and process new data on the trained model to complete prediction or classification.

[0005] In order to meet the performance requirements of AI model inference applications, many special inference acceleration chips have appeared. On these acceleration chips, in addition to greatly improving the computing power by eliminating redundant functional parts in general-purpose processors and adding special parallel matrix operation units, high-bandwidth memory (HBM) is also equipped to improve memory performance. However, due to manufacturing process limitations, the HBM capacity on each acceleration chip is limited, and the cost is high. In addition, the mainstream inference service framework in the industry monopolizes the HBM of the acceleration chip for each service, and the resource utilization rate of the acceleration hardware is generally not high.

[0006] In the prior art, multiple inference requests are inferred in batches in parallel, which can improve the utilization of hardware resources of the GPU. Meanwhile, the same model parameters used by multiple inference requests only need to be loaded from the HBM once, which improves the throughput performance of inference. However, although this technology supports access to the same shared memory by multiple processes, different inference instances need to be coordinated through inter-process communication. This method is not applicable when the communication between different instances is isolated. SUMMARY

[0007] In a first aspect, the present application provides a data processing method, comprising: obtaining a first identifier corresponding to a first model parameter; the first model parameter being a model parameter of a model indicated by a first model inference request triggered by a first instance for inference; determining a first virtual address corresponding to the first identifier according to a preset mapping relationship; wherein the mapping relationship comprises a plurality of identifiers and a virtual address corresponding to each identifier, each virtual address corresponding to a physical space storing a model parameter, and different model parameters corresponding to different identifiers, and the physical space being located in a shared memory; obtaining the first model parameter from the physical space corresponding to the first virtual address and performing model inference indicated by the first model inference request according to the first model parameter.

[0008] After the model is trained, the parameters are fixed, so the unique identifier is also unchanged. Therefore, in the embodiments of the present application, when the instance performs model inference, the unique identifier of the model parameter of the model to be inferred can be obtained. Different model parameters can determine different identifiers, and the mapping relationship contains the virtual address of the shared memory space where the model parameter corresponding to the identifier is stored, so that the model parameter corresponding to the identifier can be accessed. Furthermore, the instances can access the shared memory without interaction and cooperation with each other, and perform model inference based on the access result.

[0009] In a possible implementation, the first model parameter is a parameter of at least one network layer of a machine learning model.

[0010] The first model parameter can represent the structure, parameter value, etc. of the network layer, and the model parameters corresponding to different network layers (network layers with different structures or parameter values) are different.

[0011] In a possible implementation, for the model parameter (for example, the first model parameter) of the model to be inferred, the corresponding identifier (for example, the first identifier) can be determined in the embodiments of the present application. The first identifier and the first model parameter are uniquely corresponding, that is, based on the first model parameter, the uniquely corresponding first identifier can be determined, and based on the first identifier, only the first model parameter and its corresponding identifier exist.

[0012] In a possible implementation, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the method further includes: allocating the first virtual address to the first identifier, the first virtual address being a virtual address of a physical space of the shared memory, so as to construct the preset mapping relationship, the preset mapping relationship including a correspondence between the first identifier and the first virtual address; and storing the first model parameter to the physical space corresponding to the first virtual address.

[0013] The unique identifier of the model parameter can be used when the model is loaded (that is, the model parameter is stored into the memory) or when the model is inferred.

[0014] Specifically, when the model is loaded, the first virtual address can be allocated to the first identifier, the first virtual address being a virtual address of a physical space of the shared memory, so as to construct the preset mapping relationship, the preset mapping relationship including a correspondence between the first identifier and the first virtual address, and the first model parameter is stored to the physical space corresponding to the first virtual address.

[0015] Optionally, when the mapping relationship does not include the first identifier and the corresponding virtual address, the first instance is given a permission to access the physical space corresponding to the first identifier, or the first instance is given a role of an owner of the physical space corresponding to the first identifier.

[0016] Optionally, when the mapping relationship includes the first identifier and the corresponding virtual address, the first instance is given a permission to access the physical space corresponding to the first identifier, or the first instance is given a role of a user of the physical space corresponding to the first identifier but not an owner.

[0017] That is, the invariance of the model parameter after training can be used to calculate a unique identifier of the model parameter, so as to register a memory block that can be shared by other inference instances.

[0018] In a possible implementation, the method further includes:

[0019] When the inference service corresponding to the first model inference request ends, the permission of the first instance to access the physical space corresponding to the first identifier is released; and

[0020] When the first instance is an owner of the physical space corresponding to the first identifier, and there is another instance that is a user of the physical space corresponding to the first identifier, the owner role of the first instance is transferred to one of the other instances, or the virtual space of the first virtual address is released; or

[0021] When the first instance is the owner of the physical space corresponding to the first identifier, and there are no other instances that are users of the physical space corresponding to the first identifier, the physical space of the first virtual address is released.

[0022] In one possible implementation, the method further includes:

[0023] Based on the first model parameters, the first identifier is determined using a checksum or hash mapping method.

[0024] In one possible implementation, the first instance is a virtual machine or a container.

[0025] The above method enables collaborative physical memory sharing between inference service instances in different isolated environments, such as containers and virtual machines.

[0026] In one possible implementation, when other instances need to perform model inference, they can also access shared memory based on the mechanism described above. Taking the second model inference request as an example, a first identifier corresponding to the first model parameter can be obtained based on the second model inference request; the first model inference request and the second model inference request are triggered by different instances that are isolated from each other; according to the preset mapping relationship, a first virtual address corresponding to the first identifier is determined; the first model parameter is obtained from the physical space corresponding to the first virtual address, and the model inference indicated by the second model inference request is executed according to the first model parameter.

[0027] In one possible implementation, the first identifier is generated after model training and recorded in the model file;

[0028] The step of obtaining the first identifier corresponding to the first model parameters includes:

[0029] Read the first identifier corresponding to the first model parameter from the model file.

[0030] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0031] The acquisition module is used to acquire the first identifier corresponding to the first model parameter; the first model parameter is the model parameter of the model indicating inference in the first model inference request triggered by the first instance;

[0032] A mapping model is used to determine the first virtual address corresponding to the first identifier according to a preset mapping relationship; wherein, the mapping relationship includes multiple identifiers and the virtual address corresponding to each identifier, the physical space corresponding to each virtual address stores model parameters, and different model parameters correspond to different identifiers, and the physical space is located in shared memory;

[0033] The inference module is used to obtain the first model parameters from the physical space corresponding to the first virtual address and execute the model inference indicated by the first model inference request based on the first model parameters.

[0034] In one possible implementation, the first model parameters are parameters of at least one network layer of the machine learning model.

[0035] In one possible implementation, the mapping module is further configured to:

[0036] Before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the first virtual address is allocated to the first identifier. The first virtual address is a virtual address of the physical space of shared memory to construct the preset mapping relationship. The preset mapping relationship includes the correspondence between the first identifier and the first virtual address.

[0037] The device further includes:

[0038] The loading module is used to store the first model parameters into the physical space corresponding to the first virtual address.

[0039] In one possible implementation, the mapping module is further configured to, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of the owner of the physical space corresponding to the first identifier, when the mapping relationship does not contain the first identifier and the corresponding virtual address.

[0040] In one possible implementation, the mapping module is further configured to, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of a user but not owner of the physical space corresponding to the first identifier, when the mapping relationship already includes the first identifier and the corresponding virtual address.

[0041] In one possible implementation, the mapping module is further configured to, upon termination of the inference service corresponding to the first model inference request, release the second instance from access to the physical space corresponding to the second identifier; wherein,

[0042] When the second instance is the owner of the physical space corresponding to the second identifier, and there are other instances that are users of the physical space corresponding to the second identifier, the owner role of the second instance is transferred to one of the other instances, or the virtual space of the first virtual address is released; or,

[0043] When the second instance is the owner of the physical space corresponding to the second identifier, and there are no other instances that are users of the physical space corresponding to the second identifier, the physical space of the first virtual address is released.

[0044] In one possible implementation, the mapping module is further configured to:

[0045] Based on the first model parameters, the first identifier is determined by a checksum or hash mapping device.

[0046] In one possible implementation, the first instance is a virtual machine or a container.

[0047] In one possible implementation, the acquisition module is further configured to acquire a first identifier corresponding to the first model parameter based on the second model inference request; the first model inference request and the second model inference request are triggered by different instances that are isolated from each other;

[0048] The mapping module is further configured to determine the first virtual address corresponding to the first identifier according to the preset mapping relationship;

[0049] The inference module is further configured to obtain first model parameters from the physical space corresponding to the first virtual address and execute model inference as indicated by the second model inference request based on the first model parameters.

[0050] In one possible implementation, the first identifier is generated after model training and recorded in the model file;

[0051] The acquisition module is specifically used for:

[0052] Read the first identifier corresponding to the first model parameter from the model file.

[0053] Thirdly, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect above and any of its optional methods.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its optional methods.

[0055] Fifthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any of its optional methods described above.

[0056] Sixthly, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0057] Figure 1 is a schematic diagram of a structural framework for artificial intelligence.

[0058] Figure 2 is a schematic diagram of the application system framework of the present invention;

[0059] Figure 3 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0060] Figure 4 is a schematic diagram of a framework provided in an embodiment of this application;

[0061] Figure 5 is a schematic diagram of a framework provided in an embodiment of this application;

[0062] Figure 6 is a schematic diagram of a framework provided in an embodiment of this application;

[0063] Figure 7 is a schematic diagram of a data processing device provided in an embodiment of this application;

[0064] Figure 8 is a schematic diagram of a terminal device provided in an embodiment of this application;

[0065] Figure 9 is a schematic diagram of a server structure provided in an embodiment of this application;

[0066] Figure 10 is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0067] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0068] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0069] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0070] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0071] First, the overall workflow of an artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The framework is then elaborated on from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0072] (1) Infrastructure

[0073] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0074] (2) Data

[0075] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0076] (3) Data processing

[0077] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0078] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0079] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0080] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0081] (4) General ability

[0082] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0083] (5) Smart Products and Industry Applications

[0084] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0085] Referring to Figure 2, which is a schematic diagram of an architecture for an embodiment of this application, this application can be included in deep learning inference service platform software and deployed on server hardware. The program code of this application exists within the inference service framework and runtime engine of the platform software, as well as outside existing modules. During runtime, the program code of this application runs on the server's host memory 3029 and CPU. Figure 2 shows the implementation of this application in the server 3018 and platform software, where the dashed boxes indicate the relevant modules. The inference service framework 3019 includes a unique identifier calculation / loading module 3020; within the runtime engine 3021, this application provides a shared memory allocation / release module 3022 for XPU decoupling; outside the existing modules of the platform software, this application provides a shared memory pool management module 3023 for XPU decoupling. Furthermore, the modules will be described in subsequent embodiments.

[0086] Since the embodiments of this application involve the application of neural networks, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0087] (1) Neural Network

[0088] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0089] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0090] (2) Instance: An instance is a software module that can run application programs or system programs. Furthermore, this software module can only be called an instance when it is run by the operating system; otherwise, it is just a piece of code. Instances have various forms; for example, processes and threads can both be understood as types of instances. Instances run on nodes.

[0091] (3) Node: A node is a device that has at least one of the functions of processing data and storing data. Nodes run an operating system, and nodes can be distinguished by their operating systems; that is, different nodes run different operating systems. Alternatively, the hardware and software used to run an operating system can be considered to belong to the same node. In this application, a node can be a complete physical machine, such as a terminal or a network device like a server or server proxy. A node can also be a component within a physical machine, such as a processor, memory, or storage device. When a physical machine includes two nodes running different operating systems, for example, if the physical machine includes a central processing unit (CPU) running operating system 1 and a graphics processing unit (GPU) running operating system 2, then the CPU and GPU can be considered to belong to different nodes. When a node is a component, it is a relatively independent entity in terms of hardware. "Independent entity" can be understood as a circuit or module with a specific function, and this component is packaged or assembled on a physical machine. Data access in this application can be understood as one node accessing data maintained by another node through an instance. Devices within a node refer to components or parts within that node. For example, when a node is a physical machine, the CPU and memory are devices within that node. A physical machine refers to a computer packaged as a product, such as a server, desktop computer, all-in-one PC (AIO), laptop, or smartphone.

[0092] (4) Identifier: Used to distinguish one thing from other things of the same or different kinds. For example, the identifier of a node, the identifier of a network, and the identifier of a network interface card. The identifier can be a name, a number, or a distinctive feature, such as a category identifier. This application does not limit the implementation of various identifiers, as long as they can make a distinction. For example, in some implementations of this application, the identifier of the instance's virtual address space is used as the instance identifier, instead of the more common method of using the instance's name or number as the instance identifier.

[0093] (5) Address Space: Also called storage space, it refers to one or more segments of addresses that can be used by a device or instance. For example, the virtual address space of a device or instance is the segment or more virtual addresses that can be used by that device or instance. The virtual address space of a device or instance is allocated by the operating system running that device or instance. Similarly, the physical address space of a device or instance is the segment or more physical addresses allocated to that device or instance. When a device or instance uses this physical address space, other devices or instances cannot use addresses in this physical address space. The physical address space of an instance is allocated by the operating system running that instance. This allocation may be dynamic; for example, as the instance runs, the physical address space it occupies may increase, but there will be an upper limit. The size and range of a device's physical address space are usually fixed.

[0094] (6) Page table: A data structure used to record the mapping relationship between two types of data. It is usually used to convert virtual addresses to physical addresses. Hierarchical page tables are also a type of page table. Page tables are stored in storage media. The base address of the page table, also known as the page table base address, refers to the starting address of the physical address that stores the page table. The size of the page table determines the range of values ​​of the virtual address corresponding to the page table. The virtual address in the page table is the virtual address of the instance or device using the page table, used to indicate the offset of the physical address of a certain position (or a certain entry) in the page table relative to the page table base address. The data stored at a certain position in the page table is the physical address in the physical address space of the instance or device using the page table. A page table includes multiple entries, each of which can indicate the mapping relationship between a virtual address and a physical address of the instance or device using the page table. It should be understood that the entry mentioned here is a functional description, and the specific implementation of the entry is not limited in this application. For example, the mapping relationship between a virtual address and a physical address may be cross-level, or the mapping relationship between a virtual address and a physical address may be indirect. This application does not limit the specific implementation of the page table; for example, it can be a multi-level page table, a hash table, etc.

[0095] (7) Memory sharing is a mechanism for inter-process communication that allows two or more unrelated processes to access the same physical memory space. This mechanism connects the same physical memory to the address spaces of different processes, enabling all processes to access addresses in the shared memory. When a process writes data to the shared memory, the changes made will immediately affect any other process that can access the same shared memory.

[0096] Deep learning is an algorithm that uses a multi-layered artificial neural network architecture to learn representations of data. It consists of two processes: training and inference. The training process involves the computer using a specific optimization algorithm to search for the parameters of a given deep learning model that yields the best prediction results for a given dataset. The inference process, on the other hand, involves analyzing and processing new data using the trained model to perform predictions or classifications.

[0097] To meet the performance requirements of AI model inference applications, many dedicated inference accelerator chips have emerged. These chips significantly improve computing power by eliminating redundant functionalities in general-purpose processors and adding dedicated parallel matrix operation units, and are also equipped with high-bandwidth memory (HBM) to enhance memory access performance. However, due to manufacturing limitations, the HBM capacity on each accelerator chip is limited and its cost is high. Furthermore, mainstream inference service frameworks typically allocate dedicated HBM from each accelerator chip to each service, resulting in generally low resource utilization of the acceleration hardware.

[0098] In existing technologies, batch parallel inference of multiple inference requests can improve the utilization of GPU hardware resources. Furthermore, since the same model parameters used by multiple inference requests only need to be loaded from HBM once, inference throughput performance is improved. However, while this technology supports multiple processes accessing the same shared memory, different inference instances need to coordinate through inter-process communication. This method is not suitable when communication between different instances is isolated.

[0099] To address the aforementioned problems, embodiments of this application provide a data processing method. The model training method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0100] Referring to Figure 3, which is a flowchart of a data processing method provided in an embodiment of this application, as shown in Figure 3, the data processing method provided in an embodiment of this application may include steps 301 to 303, which will be described in detail below.

[0101] 301. Obtain the first identifier corresponding to the first model parameter, wherein the first model parameter is the model parameter of the model indicating inference in the first model inference request triggered by the first instance;

[0102] In one possible implementation, when model inference is required in a particular instance (e.g., the first instance in the embodiments of this application), a first model inference request can be triggered, wherein the first model inference request can instruct inference for a particular neural network. The neural network may include first model parameters, that is, the first model parameters are a subset of the parameters of the model to be inferred.

[0103] In one possible implementation, the first model parameters are parameters of at least one network layer of the machine learning model. For example, they could be parameters of a network layer in the machine learning model, such as a transformer layer or a convolutional layer.

[0104] The first model parameter can represent the structure and parameter values ​​of the network layer. Different network layers (network layers with different structures or parameter values) have different model parameters.

[0105] In one possible implementation, for the model parameters (e.g., the first model parameter) of the model that needs to be inferred, the corresponding identifier (e.g., the first identifier) ​​can be determined in this embodiment of the application. The first identifier and the first model parameter are uniquely corresponding. That is, the first identifier can be uniquely determined based on the first model parameter, and based on the first identifier, only the first model parameter corresponds to it.

[0106] For example, the first identifier can be determined based on the first model parameters using a checksum or hash mapping method.

[0107] In one possible implementation, the first identifier can be generated after model training and recorded in the model file; then, the first identifier corresponding to the first model parameter can be read from the model file.

[0108] For example, the unique identifiers of the model parameters (e.g., the first identifier) ​​can be calculated and recorded in the model file by the MindSpore training framework when the model file is exported during the model training phase. The MindSpore inference framework can then directly parse and read them from the model file.

[0109] For example, the calculation of unique identifiers (e.g., a first identifier) ​​for model parameters can be performed when the inference service framework process loads the model file. The model file may store the model's network structure and model parameter information. The model loading process includes loading the model's network structure, model parameters, and unique identifiers calculated based on the model parameter data into memory (e.g., host memory). Alternatively, the unique identifiers for model parameters can also be calculated and recorded when the model file is generated after model training.

[0110] 302. Determine the first virtual address corresponding to the first identifier according to the preset mapping relationship; wherein, the mapping relationship includes multiple identifiers and the virtual address corresponding to each identifier, the physical space corresponding to each virtual address stores model parameters, and different model parameters correspond to different identifiers, and the physical space is located in shared memory;

[0111] The unique identifiers for model parameters can be used during model loading (i.e., storing model parameters in memory) or model inference.

[0112] Specifically, during model loading, the first virtual address can be allocated to the first identifier. The first virtual address is a virtual address in the physical space of shared memory to construct the preset mapping relationship. The preset mapping relationship includes the correspondence between the first identifier and the first virtual address, and the first model parameters are stored in the physical space corresponding to the first virtual address.

[0113] Optionally, when the mapping relationship does not include the first identifier and its corresponding virtual address, the first instance is granted permission to access the physical space corresponding to the first identifier, or the first instance is granted the role of the owner of the physical space corresponding to the first identifier.

[0114] Optionally, when the mapping relationship already includes the first identifier and the corresponding virtual address, the first instance is granted permission to access the physical space corresponding to the first identifier, or the first instance is granted the role of a user but not the owner of the physical space corresponding to the first identifier.

[0115] In other words, the invariance of model parameters after training can be used to calculate a unique identifier for the model parameters, thereby registering a memory block that can be shared by other inference instances.

[0116] For example, the inference service framework calls the runtime engine interface to request virtual address space within the current process based on the size of the model parameters. For instance, it can request shared memory blocks from a shared memory pool (for simplicity, shared memory in this application can be memory 3030 in the XPU device) based on unique identifiers for each model parameter. It can independently request shared memory blocks from the shared memory pool for each model parameter using the unique identifier of the model parameter and the address space of the shared memory pool as input, without needing to communicate with other instances. If the uniquely identified shared memory block is already in the shared memory pool of the specified device, the current inference service process is added to the user list of that shared memory block. Otherwise, a shared memory block with the unique identifier is created in the shared memory pool of the specified device, and the current inference service process is set as the owner of that shared memory block. Afterwards, it can call the service provided by the XPU driver 3024 to request the allocation of physical memory for the XPU based on the model parameter size and the specified XPU device, and establish a hardware page table mapping with the physical memory using the allocated virtual address space as input. Subsequently, the model parameter data can be copied to this physical memory and accessed directly based on the virtual address.

[0117] Specifically, during model inference (e.g., model inference indicated by the first model inference request triggered by the first instance), the model parameters of the model can be obtained for the model to be inferred.

[0118] After training, the model's parameters are fixed, and therefore, the unique identifier remains unchanged. Thus, in this embodiment, when an instance performs model inference, it can obtain the unique identifier of the model's parameters for the model to be inferred. Different model parameters can determine different identifiers, and the mapping relationship includes the virtual address of the shared memory space where the model parameter corresponding to that identifier is stored, thereby allowing access to the model parameter corresponding to that identifier. This enables instances to access shared memory and perform model inference based on the access results without needing to interact or collaborate with each other.

[0119] 303. Obtain the first model parameters from the physical space corresponding to the first virtual address and perform model inference based on the first model parameters.

[0120] In one possible implementation, when the inference service corresponding to the first model inference request ends, the permission for the first instance to access the physical space corresponding to the first identifier can be revoked; wherein, when the first instance is the owner of the physical space corresponding to the first identifier and there are other instances that are users of the physical space corresponding to the first identifier, the owner role of the first instance is transferred to one of the other instances, or the virtual space of the first virtual address is released; or, when the first instance is the owner of the physical space corresponding to the first identifier and there are no other instances that are users of the physical space corresponding to the first identifier, the physical space of the first virtual address is released.

[0121] Before the inference service instance is destroyed, if the current instance is the owner of the shared memory block and there are no other users, it is directly destroyed from the shared memory pool, and the XPU physical memory and virtual address space are released. Otherwise, if the current instance is the owner of the shared memory block, the ownership role is transferred to one of the users, and the corresponding virtual address space is released; if the current instance is a user of the shared memory block, only the current instance is removed from the user list of the shared memory block, and the corresponding virtual address space is released.

[0122] In one possible implementation, when other instances need to perform model inference, they can also access shared memory based on the mechanism described above. Taking the second model inference request as an example, a first identifier corresponding to the first model parameter can be obtained based on the second model inference request; the first model inference request and the second model inference request are triggered by different instances that are isolated from each other; according to the preset mapping relationship, a first virtual address corresponding to the first identifier is determined; the first model parameter is obtained from the physical space corresponding to the first virtual address, and the model inference indicated by the second model inference request is executed according to the first model parameter.

[0123] The above method enables collaborative physical memory sharing between inference service instances in different isolated environments, such as containers and virtual machines.

[0124] The following is a schematic diagram of a module architecture of an embodiment of this application, with reference to the accompanying drawings:

[0125] Referring to Figure 4, which is a schematic diagram of a module architecture according to an embodiment of this application, the module includes a module for generating, calculating, and loading unique identifiers, a module for requesting shared memory for collaborative purposes, and a management module for a shared memory pool (e.g., a shared memory pool within the XPU). The module for generating and loading unique identifiers is used to generate unique identifiers for each read-only model parameter in the model, and these identifiers correspond one-to-one with the XPU shared memory blocks to be requested. This module can be accessed by the inference framework to obtain model structure and model parameter data. Optionally, the unique identifier associated with the XPU shared memory block is generated based on the numerical content of each model parameter, and is not limited to algorithms such as hash. Identifiers generated from the same model parameter data content must be deterministic and unique. The module for requesting / releasing shared XPU memory for collaborative purposes is used to collaboratively request / release shareable XPU physical memory for each read-only model parameter in the model. This module can be accessed by the inference framework, independently requesting virtual address space and XPU physical memory after model loading is completed, and independently releasing them when the inference service ends. Optionally, this module requests virtual address space for each model parameter through the virtual address management module based on the model parameter size.

[0126] Optionally, this module independently requests shareable XPU physical memory through the XPU physical memory management module based on the model parameter size and the unique identifier generated for it, and establishes a hardware page table mapping between its physical memory pages and the virtual address space through the XPU page table management module.

[0127] Optionally, the module releases the virtual address space through the virtual address management module based on the virtual starting address allocated by each model parameter, and releases the shared XPU physical memory through the XPU physical memory management module.

[0128] The management module for the XPU decoupling shared memory pool manages all created decoupling shared memory blocks for each physical XPU device. This module can be accessed by newly added modules that allocate / release decoupling shared memory to decide whether to create / destroy new shared memory blocks or update the metadata information of existing shared memory blocks.

[0129] Optionally, each shared memory block records and maintains its own metadata information, including but not limited to a unique identifier, size, owner, user list, physical memory address, etc.

[0130] Optionally, shared memory blocks are identified by a unique identifier, and there are no multiple shared memory blocks with the same identifier.

[0131] Optionally, the metadata information of the shared memory block maintains a list of one owner and multiple users. When an owner releases the shared memory block and there are other users, one of those users is promoted to owner. When an owner releases the shared memory block and there are no other users, the shared memory block is destroyed from the shared memory pool. When a user releases the shared memory block, only that user is removed from the user list.

[0132] Figure 5 illustrates a component structure diagram of an embodiment of this application, using the MindSpore inference framework as an example. In the MindSpore 3033 inference framework, the modules involved in this embodiment include a model loading module 3034 and a computation / loading module 3035 for uniquely identifying model parameters. In the NPU runtime library 3036, the modules involved in this embodiment include a virtual address allocation / release module 3037 and a module 3038 for de-cooperative allocation / release of NPU shared HBM. Additionally, there is an NPU de-cooperative shared memory pool management module 3039. In the hardware of the server 3032, the components involved in this embodiment include host memory 3045 and HBM 3047 on the NPU 3046. The parts indicated by the dashed boxes in Figure 5 are modules newly added to the existing software in this embodiment. Furthermore, the modules can be referred to the description of the embodiment, which will not be repeated here.

[0133] To achieve collaborative NPU HBM sharing between different inference instances, this embodiment follows the specific implementation steps as follows:

[0134] The MindSpore inference framework calculates unique identifiers for each model parameter of the inference model.

[0135] When parsing the model file to be inferred, the MindSpore inference framework calculates a unique identifier for each model parameter tensor based on the data content during model parameter loading. The calculation can be accelerated asynchronously using dedicated hardware instructions (such as SIMD). The unique identifier algorithm includes, but is not limited to, checksums and hashes, but it must be ensured that identical model parameter data can only generate a unique identifier. The MindSpore inference framework calls the NPU runtime library interface to allocate virtual address space for each model parameter of the inference model. Based on the unique identifier, the MindSpore inference framework calls the interface provided by the NPU runtime library to request shared memory blocks from the XPU decoupling shared memory pool for each model parameter. The memory management module of the NPU decoupling shared memory pool calls the NPU driver 3040 interface on the specified NPU to allocate HBM and establish a hardware page table mapping with the virtual address space.

[0136] After the inference service ends, the MindSpore inference framework calls the interface provided by the NPU runtime library to close / destroy the shared memory block of the model parameters, including releasing the NPU's HBM (if the current inference service process is the only user of the shared memory block) and virtual address space.

[0137] The instances can be isolated virtual machines or containers. Referring to Figure 6, which uses the MindSpore inference framework as an example, a component structure diagram of an embodiment of this application is shown. The MindSpore inference framework and the NPU runtime library are deployed in virtual machines or containers, while the NPU collaborative shared memory pool management module is deployed in the host OS.

[0138] Referring to Figure 7, which is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, as shown in Figure 7, a data processing apparatus 700 provided in an embodiment of this application includes:

[0139] The acquisition module 701 is used to acquire the first identifier corresponding to the first model parameter; the first model parameter is the model parameter of the model indicating inference in the first model inference request triggered by the first instance;

[0140] For a detailed description of the acquisition module 701, please refer to the description of the embodiment corresponding to step 301 in the above embodiments. The similarities will not be repeated here.

[0141] Mapping model 702 is used to determine the first virtual address corresponding to the first identifier according to a preset mapping relationship; wherein, the mapping relationship includes multiple identifiers and the virtual address corresponding to each identifier, the physical space corresponding to each virtual address stores model parameters, and different model parameters correspond to different identifiers, and the physical space is located in shared memory;

[0142] For a detailed description of the mapping model 702, please refer to the description of the embodiment corresponding to step 302 in the above embodiments. The similarities will not be repeated here.

[0143] The inference module 703 is used to obtain the first model parameters from the physical space corresponding to the first virtual address and perform model inference as indicated by the first model inference request based on the first model parameters.

[0144] For a detailed description of the reasoning module 703, please refer to the description of the embodiment corresponding to step 303 in the above embodiments. The similarities will not be repeated here.

[0145] In one possible implementation, the first model parameters are parameters of at least one network layer of the machine learning model.

[0146] In one possible implementation, the mapping module 702 is further configured to:

[0147] Before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the first virtual address is allocated to the first identifier. The first virtual address is a virtual address of the physical space of shared memory to construct the preset mapping relationship. The preset mapping relationship includes the correspondence between the first identifier and the first virtual address.

[0148] The device further includes:

[0149] The loading module is used to store the first model parameters into the physical space corresponding to the first virtual address.

[0150] In one possible implementation, the mapping module 702 is further configured to, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of the owner of the physical space corresponding to the first identifier, when the mapping relationship does not contain the first identifier and the corresponding virtual address.

[0151] In one possible implementation, the mapping module 702 is further configured to, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of a user but not owner of the physical space corresponding to the first identifier, when the mapping relationship already includes the first identifier and the corresponding virtual address.

[0152] In one possible implementation, the mapping module 702 is further configured to, upon termination of the inference service corresponding to the first model inference request, release the second instance from access to the physical space corresponding to the second identifier; wherein,

[0153] When the second instance is the owner of the physical space corresponding to the second identifier, and there are other instances that are users of the physical space corresponding to the second identifier, the owner role of the second instance is transferred to one of the other instances, or the virtual space of the first virtual address is released; or,

[0154] When the second instance is the owner of the physical space corresponding to the second identifier, and there are no other instances that are users of the physical space corresponding to the second identifier, the physical space of the first virtual address is released.

[0155] In one possible implementation, the mapping module 702 is further configured to:

[0156] Based on the first model parameters, the first identifier is determined by a checksum or hash mapping device.

[0157] In one possible implementation, the first instance is a virtual machine or a container.

[0158] In one possible implementation, the acquisition module 701 is further configured to acquire a first identifier corresponding to the first model parameter based on the second model inference request; the first model inference request and the second model inference request are triggered by different instances that are isolated from each other;

[0159] The mapping module 702 is further configured to determine the first virtual address corresponding to the first identifier according to the preset mapping relationship;

[0160] The inference module 703 is further configured to obtain first model parameters from the physical space corresponding to the first virtual address and execute model inference indicated by the second model inference request based on the first model parameters.

[0161] In one possible implementation, the first identifier is generated after model training and recorded in the model file;

[0162] The acquisition module 701 is specifically used for:

[0163] Read the first identifier corresponding to the first model parameter from the model file.

[0164] The following describes an execution device provided in an embodiment of this application. Please refer to Figure 8, which is a schematic diagram of the structure of an execution device provided in an embodiment of this application. Specifically, the execution device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the execution device 800 may have one or more processors 803; Figure 8 shows one processor as an example). The processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0165] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0166] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0167] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804. Processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps involved in the model inference process described above.

[0168] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0169] This application embodiment also provides a server device. Please refer to Figure 9, which is a schematic diagram of a server structure provided in this application embodiment. Specifically, the server 900 is implemented by one or more servers. The server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 99 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the CPU 99 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the server 900.

[0170] Server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958; or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0171] In this embodiment, the central processing unit 99 is used to execute the data processing method described in the above embodiment.

[0172] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0173] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0174] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0175] Specifically, please refer to Figure 10, which is a schematic diagram of a chip structure provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0176] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0177] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0178] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0179] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0180] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0181] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0182] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit 1003 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0183] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0184] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0185] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0186] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0187] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0189] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0190] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first identifier corresponding to the first model parameter; the first model parameter is the model parameter of the model indicating inference in the first model inference request triggered by the first instance; According to a preset mapping relationship, the first virtual address corresponding to the first identifier is determined; wherein, the mapping relationship includes multiple identifiers and the virtual address corresponding to each identifier, the physical space corresponding to each virtual address stores model parameters, and different model parameters correspond to different identifiers, and the physical space is located in shared memory; Obtain the first model parameters from the physical space corresponding to the first virtual address and execute the model inference indicated by the first model inference request based on the first model parameters.

2. The method according to claim 1, characterized in that, The first model parameters are the parameters of at least one network layer of the machine learning model.

3. The method according to claim 1 or 2, characterized in that, Before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the method further includes: Assign the first virtual address to the first identifier, where the first virtual address is a virtual address in the physical space of shared memory, to construct the preset mapping relationship, which includes the correspondence between the first identifier and the first virtual address; The first model parameters are stored in the physical space corresponding to the first virtual address.

4. The method according to claim 3, characterized in that, Before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the method further includes: When the mapping relationship does not include the first identifier and its corresponding virtual address, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of the owner of the physical space corresponding to the first identifier.

5. The method according to claim 1 or 2, characterized in that, Before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the method further includes: When the mapping relationship already includes the first identifier and the corresponding virtual address, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of a user but not the owner of the physical space corresponding to the first identifier.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When the inference service corresponding to the first model inference request ends, the permission for the first instance to access the physical space corresponding to the first identifier is revoked; wherein, When the first instance is the owner of the physical space corresponding to the first identifier, and there are other instances that are users of the physical space corresponding to the first identifier, the owner role of the first instance is transferred to one of the other instances, or the virtual space of the first virtual address is released; or, When the first instance is the owner of the physical space corresponding to the first identifier, and there are no other instances that are users of the physical space corresponding to the first identifier, the physical space of the first virtual address is released.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the first model parameters, the first identifier is determined using a checksum or hash mapping method.

8. The method according to any one of claims 1 to 7, characterized in that, The first instance is a virtual machine or a container.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Based on the second model inference request, obtain the first identifier corresponding to the first model parameters; the first model inference request and the second model inference request are triggered by different instances that are isolated from each other; Based on the preset mapping relationship, determine the first virtual address corresponding to the first identifier; Obtain the first model parameters from the physical space corresponding to the first virtual address and execute the model inference indicated by the second model inference request based on the first model parameters.

10. The method according to any one of claims 1 to 9, characterized in that, The first identifier is generated after model training and recorded in the model file; The step of obtaining the first identifier corresponding to the first model parameters includes: Read the first identifier corresponding to the first model parameter from the model file.

11. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first identifier corresponding to the first model parameter; the first model parameter is the model parameter of the model indicating inference in the first model inference request triggered by the first instance; A mapping model is used to determine the first virtual address corresponding to the first identifier according to a preset mapping relationship; wherein, the mapping relationship includes multiple identifiers and the virtual address corresponding to each identifier, the physical space corresponding to each virtual address stores model parameters, and different model parameters correspond to different identifiers, and the physical space is located in shared memory; The inference module is used to obtain the first model parameters from the physical space corresponding to the first virtual address and execute the model inference indicated by the first model inference request based on the first model parameters.

12. The apparatus according to claim 11, characterized in that, The first model parameters are the parameters of at least one network layer of the machine learning model.

13. The apparatus according to claim 11 or 12, characterized in that, The mapping module is also used for: Before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, the first virtual address is allocated to the first identifier. The first virtual address is a virtual address of the physical space of shared memory to construct the preset mapping relationship. The preset mapping relationship includes the correspondence between the first identifier and the first virtual address. The device further includes: The loading module is used to store the first model parameters into the physical space corresponding to the first virtual address.

14. The apparatus according to claim 13, characterized in that, The mapping module is further configured to, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of the owner of the physical space corresponding to the first identifier, when the mapping relationship does not contain the first identifier and the corresponding virtual address.

15. The apparatus according to claim 11 or 12, characterized in that, The mapping module is further configured to, before determining the first virtual address corresponding to the first identifier according to the preset mapping relationship, grant the first instance permission to access the physical space corresponding to the first identifier, or grant the first instance the role of a user but not owner of the physical space corresponding to the first identifier, when the mapping relationship already includes the first identifier and the corresponding virtual address.

16. The apparatus according to any one of claims 11 to 15, characterized in that, The mapping module is further configured to, upon termination of the inference service corresponding to the first model inference request, release the second instance's permission to access the physical space corresponding to the second identifier; wherein, When the second instance is the owner of the physical space corresponding to the second identifier, and there are other instances that are users of the physical space corresponding to the second identifier, the owner role of the second instance is transferred to one of the other instances, or the virtual space of the first virtual address is released; or, When the second instance is the owner of the physical space corresponding to the second identifier, and there are no other instances that are users of the physical space corresponding to the second identifier, the physical space of the first virtual address is released.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The mapping module is also used for: Based on the first model parameters, the first identifier is determined by a checksum or hash mapping device.

18. The apparatus according to any one of claims 11 to 17, characterized in that, The first instance is a virtual machine or a container.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The acquisition module is further configured to acquire a first identifier corresponding to the first model parameter based on the second model inference request; the first model inference request and the second model inference request are triggered by different instances that are isolated from each other; The mapping module is further configured to determine the first virtual address corresponding to the first identifier according to the preset mapping relationship; The inference module is further configured to obtain first model parameters from the physical space corresponding to the first virtual address and execute model inference as indicated by the second model inference request based on the first model parameters.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The first identifier is generated after model training and recorded in the model file; The acquisition module is specifically used for: Read the first identifier corresponding to the first model parameter from the model file.

21. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 10.

22. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 10.

23. A system comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other. The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 1 to 10.

24. A chip, characterized in that, It includes at least one processing unit and an interface circuit, the interface circuit being used to provide program instructions or data to the at least one processing unit, the at least one processing unit being used to execute the program instructions to implement the method of any one of claims 1 to 10.

Images