A data processing method and apparatus

Abstract

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: in the process of generating an inference result corresponding to an inference request, a cloud processing platform can obtain a target tensor identifier corresponding to the inference request; in the case where it is determined that a target tensor corresponding to the target tensor identifier is located in a storage shard, a target processing node in which the target tensor is located is obtained; the target tensor is loaded from the storage shard of the target processing node to the memory shard of the target processing node, so as to obtain the target tensor from the memory shard of the target processing node; and according to the target tensor, an inference result corresponding to the inference request is generated. Cross-node network tensor loading can be avoided, the tensor loading speed can be improved, the network bandwidth competition between tensor loading traffic and inference request traffic can be reduced, and the inference request processing efficiency can be improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In the process of inference using artificial intelligence models such as Large Language Models (LLM), in order to reduce the repeated calculation of key tensors and value tensors in the attention mechanism, thereby saving computing resources and reducing inference latency, a key-value cache (KV Cache) mechanism can be used. This means storing the already calculated key-value tensors (key tensors and value tensors) for reuse in subsequent inference processes.

[0003] Currently, the storage of key-value tensors involves a two-layer architecture of memory and storage. Due to the large data volume of key-value tensors, it is often necessary to combine them with cloud storage technology, that is, to store key-value tensors in a cloud processing platform. For cloud processing platforms, the coupling relationship between bandwidth and capacity, as well as the scarcity of cross-node networks, need to be considered. Therefore, how to quickly retrieve key-value tensors from the corresponding memory or storage of the cloud processing platform to improve model inference speed is a key issue. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, applied to a cloud processing platform, the cloud processing platform including at least one processing node, each processing node including storage shards and memory shards, the method comprising: Obtain the target tensor identifier corresponding to the inference request; If it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, the target processing node where the target tensor is located is obtained; The target tensor is loaded from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; Based on the target tensor, generate the inference result corresponding to the inference request.

[0006] According to a second aspect of the embodiments of this specification, a cloud processing platform is provided, the cloud processing platform including at least one processing node, a model inference node and a tensor management node, wherein each processing node includes a storage shard and a memory shard. The model inference node is used to obtain the target tensor identifier corresponding to the inference request and send the target tensor identifier to the tensor management node; The tensor management node is used to receive the target tensor identifier and, if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, send a tensor loading instruction to the target processing node where the target tensor is located. The target processing node is configured to, in response to the tensor loading instruction, load the target tensor from the storage shard of the target processing node to the memory shard of the target processing node, and return loading completion information to the tensor management node. The tensor management node is used to receive the loading completion information and send the target processing node identifier of the target processing node to the model inference node; The model inference node is used to receive the target processing node identifier and read the target tensor corresponding to the target tensor identifier from the memory slice of the target processing node corresponding to the target processing node identifier.

[0007] According to a third aspect of the embodiments of this specification, a data processing apparatus is provided, located in a cloud processing platform, the cloud processing platform including at least one processing node, each processing node including a storage shard and a memory shard, including: The first acquisition module is configured to acquire the target tensor identifier corresponding to the inference request; The second acquisition module is configured to acquire the target processing node where the target tensor is located when it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard. A loading module is configured to load the target tensor from a storage shard of the target processing node into a memory shard of the target processing node, so as to obtain the target tensor from the memory shard of the target processing node; The generation module is configured to generate the inference result corresponding to the inference request based on the target tensor.

[0008] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0009] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0010] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0011] In one embodiment of this specification, during the process of generating the inference result corresponding to the inference request, the cloud processing platform can obtain the target tensor identifier corresponding to the inference request; if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, the platform can obtain the target processing node where the target tensor is located; the platform can load the target tensor from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; and the platform can generate the inference result corresponding to the inference request based on the target tensor.

[0012] The above method sets up memory shards and storage shards in each processing node. When the target tensor requested by the inference request is located in the storage shard, the loading of the target tensor from the storage shard to the memory shard can be achieved within a single processing node. Then, the target tensor can be obtained from the memory shard. This avoids tensor loading across the network, improves tensor loading speed, and reduces network bandwidth contention between tensor loading traffic and inference request traffic, thereby improving inference request processing efficiency. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the structure of a cloud processing platform provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the structure of a cloud processing platform provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 5 This is an architecture diagram of a data processing system provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Remote Direct Memory Access (RDMA) is a technology that can bypass a remote central processing unit (CPU) or kernel to provide low-latency, high-throughput data transfer.

[0022] Key-Value Cache (KV Cache): In Transformers, to avoid repeatedly calculating the key-value matrix corresponding to historical tokens each time a new token is generated, the historical key-value matrix can be cached. This allows for the reuse of the cached historical key-value matrix, reducing computational complexity and saving computational resources. In one or more embodiments of this specification, a token refers to the basic unit when the model processes data. It can be a character, a word, a punctuation mark, etc. In this specification, a word is a token.

[0023] Memory Shard: In one or more embodiments of this specification, a memory shard can be understood as a logical or physical sub-unit deployed on each processing node after the full memory resources are divided according to predetermined rules. It is used to carry key-value cached memory-state data, i.e., hot data. For example, if the distributed memory pool corresponds to 5GB of full memory resources, and the cloud processing platform includes two processing nodes, then a 2GB memory shard can be configured on processing node 1, and a 3GB memory shard can be configured on processing node 2. It should be noted that the memory shard in this specification can expose an access interface through a remote direct memory access protocol, allowing the inference engine on other processing nodes to directly read the key-value tensors stored in the memory shard.

[0024] Storage Shard: In one or more embodiments of this specification, a storage shard can be understood as a logical or physical sub-unit deployed on each processing node after the full storage resources are divided according to predetermined rules. It is used to carry key-value cached storage-state data migrated from memory shards on the processing nodes, i.e., cold data. For example, assuming the full storage resources are 10GB and the cloud processing platform contains two processing nodes, a 4GB storage shard can be mounted on processing node 1, and a 6GB storage shard can be mounted on processing node 2. It should be noted that the storage resources in this specification are cloud storage resources, such as cloud block storage.

[0025] Co-location deployment on the same machine: In one or more embodiments of this specification, co-location deployment on the same machine can be understood as deploying memory shards and storage shards on the same processing node. That is, each memory shard and the storage shard in its processing node form a one-to-one binding relationship, or in other words, each memory shard and storage shard are bound together based on the processing node. This allows the process of loading cold data from the storage shard to the memory shard to be completed within the same processing node in the cloud processing platform, without going through a cross-node network.

[0026] In cloud storage technologies, cloud providers typically deploy memory-based instances, high-performance computing nodes, and large-capacity storage nodes in physical partitions, placing them on separate machines and racks. This creates two independent resource pools for memory and storage. For example... Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a cloud processing platform provided in one embodiment of this specification. Figure 1 The diagram illustrates the structure of an existing cloud processing platform, showing that it comprises several storage nodes and memory nodes. Consequently, data transfer between storage and memory nodes requires traversing the cloud cluster network, resulting in cross-node network traffic. Furthermore, cloud block storage is relatively expensive, and its bandwidth and capacity are strongly coupled; requiring more bandwidth often necessitates larger capacities, increasing costs.

[0027] Based on this, this specification provides a structural diagram of a cloud processing platform, such as... Figure 2As shown, the cloud processing platform comprises several processing nodes, each containing a memory shard and a storage shard. In one or more embodiments of this specification, the cloud processing platform can be used to store tensors during model inference, specifically key-value tensors, i.e., the key matrix and value matrix in the attention mechanism corresponding to the Transformer. The cloud processing platform can include a memory layer and a storage layer. The memory layer, as a bandwidth layer, provides high-throughput access for hot data (tensors that have been used frequently recently), while the storage layer, as a capacity layer, provides elastic large capacity for cold data (tensors that have been used less frequently recently). Furthermore, the memory layer and storage layer can be respectively divided into multiple memory shards and storage shards, and each memory shard and storage shard can be bound one-to-one to a processing node. Thus, each processing node of the cloud processing platform includes one memory shard and one storage shard.

[0028] The processing node can be any device with computing capabilities, including but not limited to computing nodes, host machines, virtual machine instances, etc.

[0029] It should be noted that this manual does not impose specific restrictions on how memory shards and storage shards are configured in each processing node. They can be configured according to actual needs, as long as both memory shards and storage shards are configured in each processing node.

[0030] In this way, during the process of generating the inference result corresponding to the inference request based on the cloud processing platform, the cloud processing platform can obtain the target tensor identifier corresponding to the inference request; if it is determined that the target tensor corresponding to the target tensor identifier is located in the storage shard, the platform can obtain the target processing node where the target tensor is located; the platform can load the target tensor from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; and the platform can generate the inference result corresponding to the inference request based on the target tensor.

[0031] The above method sets up memory shards and storage shards in each processing node, i.e., it performs co-location deployment on the same machine. Therefore, when the target tensor requested by the inference request is located in the storage shard, the loading of the target tensor from the storage shard to the memory shard can be realized within a single processing node, and then the target tensor can be obtained from the memory shard. This avoids tensor loading across the network of nodes, improves tensor loading speed, and reduces network bandwidth competition between tensor loading traffic and inference request traffic, thereby improving the efficiency of inference request processing.

[0032] Specifically, the migration of key-value tensors between memory shards and storage shards in the cloud processing platform, including unloading key-value tensors from memory shards to storage shards and loading key-value tensors from storage shards to memory shards, can all be completed within the same processing node. The data migration path can be the internal bus of the processing node, such as the Peripheral Component Interconnect Express (PCIe) bus, thus eliminating the need to traverse cross-node networks, improving data migration efficiency, increasing inference request processing efficiency, and saving costs.

[0033] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0034] See Figure 3 , Figure 3 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0035] Step 302: Obtain the target tensor identifier corresponding to the inference request.

[0036] In practical applications, users can initiate inference requests on the model client side. Specifically, users can input the information to be inferred in the model client's user interface, which will then generate an inference request carrying that information. This request is sent to the server, which has a pre-deployed machine learning model. Based on the inference request, the server can determine the target tensor identifier corresponding to the request and then send that identifier to the cloud processing platform to obtain the target tensor. Alternatively, the model client can send the inference request to the cloud processing platform, which can also have a pre-deployed machine learning model. The cloud processing platform can then determine the target tensor identifier based on the inference request.

[0037] In one or more embodiments of this specification, obtaining the target tensor identifier corresponding to the inference request includes: Obtain a reasoning request, wherein the reasoning request carries information to be reasoned; The information to be reasoned is input into a preset machine learning model to obtain the word sequence corresponding to the information to be reasoned. Based on the lexical sequence, the target tensor identifier corresponding to the information to be inferred is determined.

[0038] In practical applications, the cloud processing platform can obtain the inference request sent by the model client, and then input the information to be inferred carried in the inference request into the preset machine learning model. The preset machine learning model can analyze and process the information to be inferred and output the word sequence corresponding to the information to be inferred.

[0039] In one or more embodiments of this specification, the preset machine learning model can be any model based on the Transformer architecture, such as a large language model, and this specification does not impose any specific limitations.

[0040] In one or more embodiments of this specification, a lexical sequence can be divided into multiple lexical subsequences, and then the tensor identifier corresponding to each lexical subsequence can be calculated to obtain the target tensor identifier corresponding to the information to be inferred.

[0041] Specifically, in one or more embodiments of this specification, determining the target tensor identifier corresponding to the information to be reasoned based on the lexical sequence includes: The word sequence is divided into multiple word subsequences; For each word subsequence, obtain the tensor identifier of the previous word subsequence, and determine the tensor identifier of the word subsequence using a hash algorithm based on the word subsequence and the tensor identifier of the previous word subsequence. Based on the tensor identifiers corresponding to each word subsequence, the target tensor identifier corresponding to the information to be inferred is obtained.

[0042] In practical applications, when determining the tensor identifier of a word subsequence using a hash algorithm based on the tensor identifiers of the word subsequence and the previous word subsequence, the tensor identifiers of the word subsequence and the previous word subsequence can be concatenated. Then, a hash algorithm is used to calculate the hash value corresponding to the concatenated sequence, and the hash value is used as the tensor identifier corresponding to the word subsequence. Thus, the target tensor identifier corresponding to the information to be inferred can be obtained based on the tensor identifier corresponding to each word subsequence.

[0043] When dividing a word sequence into multiple word subsequences, the division can be based on a preset number of words, such as dividing the word sequence into multiple word subsequences by specifying a certain number of words as one word subsequence. The specified number can be 2, 3, etc. Random division is also possible, as well as division according to words, sentences, etc., in the vocabulary list corresponding to the preset words. This manual does not impose specific limitations.

[0044] It should be noted that the hash algorithm can be set based on actual needs, and this manual does not impose specific restrictions.

[0045] The method described above, which divides the word sequence corresponding to the information to be reasoned into multiple word subsequences and uses a hash algorithm to generate the target tensor identifier corresponding to the information to be reasoned, can improve semantic accuracy and thus improve reasoning accuracy.

[0046] Step 304: If it is determined that the target tensor corresponding to the target tensor identifier is located in the storage shard, obtain the target processing node where the target tensor is located.

[0047] In practical applications, after obtaining the target tensor identifier, it can be first determined whether the target tensor corresponding to the target tensor identifier has been stored in the cloud processing platform. In one or more embodiments of this specification, the cloud processing platform may store a first correspondence, wherein the first correspondence represents the correspondence between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier.

[0048] In one or more embodiments of this specification, if it is determined that the target tensor identifier is not stored in the cloud processing platform, the target tensor can be calculated to store the calculated target tensor in the cloud processing platform, and the first correspondence can be updated.

[0049] Specifically, if it is determined that the target tensor identifier does not exist in the first correspondence, a target tensor corresponding to the inference request is generated based on a preset machine learning model, and the target shard type and target processing node identifier corresponding to the target tensor are determined to obtain the target storage information of the target tensor. The target storage information and the target tensor identifier are added to the first correspondence, and the target tensor is stored in the target fragment in the processing node corresponding to the target processing node identifier, wherein the target fragment corresponds to the target fragment type.

[0050] In practical applications, if it is determined that the target tensor identifier does not exist in the first correspondence, that is, if it is determined that the target tensor identifier is not stored in the cloud processing platform, the word sequence corresponding to the information to be inferred can be input into the preset machine learning model, so that the preset machine learning model can perform internal calculations to generate the target tensor corresponding to the word sequence.

[0051] It should be understood that how the preset machine learning model performs internal calculations to generate the target tensor corresponding to the word sequence is a capability inherent in the machine learning model itself, and this manual will not elaborate on it in detail.

[0052] Then, the cloud processing platform can determine the storage information of the target tensor. Specifically, the cloud processing platform can determine the target shard type and target processing node identifier corresponding to the target tensor based on preset storage rules. The shard type includes storage shards and memory shards.

[0053] In one or more embodiments of this specification, the preset storage rules can be set based on actual needs. For example, the preset storage rules can be random rules, that is, the target processing node and target shard where the target tensor should be stored can be randomly determined. Another example is that the preset rules can be: determine the target processing node identifier corresponding to the target tensor according to the storage capacity of each processing node. Specifically, the node identifier of the processing node with the largest storage capacity can be used as the target processing node identifier, and then the target tensor can be stored in the memory shard or storage shard of the target processing node corresponding to the target processing node identifier. That is to say, the target shard type can be either a memory shard or a storage shard.

[0054] Finally, after determining the target fragment type and target processing node identifier corresponding to the target tensor, the target tensor can be stored in the target processing node corresponding to the target processing node identifier, specifically in the target fragment of the target processing node, where the type of the target fragment is the target fragment type. For example, assuming the target fragment type is a storage fragment and the target processing node identifier is node 1, the target tensor can be stored in the storage fragment of node 1.

[0055] Furthermore, the target processing node identifier and target fragment type can be used as target storage information of the target tensor, and the correspondence between the target tensor identifier and the target storage information can be stored in the first correspondence.

[0056] Using the above method, if it is determined that the target tensor corresponding to the target tensor identifier is not stored in the cloud processing platform, the target tensor can be calculated, generated, and stored so that it can be reused in subsequent steps.

[0057] Step 306: Load the target tensor from the storage shard of the target processing node into the memory shard of the target processing node, so as to obtain the target tensor from the memory shard of the target processing node.

[0058] In one or more embodiments of this specification, if it is determined that a target tensor corresponding to a target tensor identifier is stored in the cloud processing platform, the target storage information corresponding to the target tensor identifier can be queried and obtained based on the first correspondence. Then, it can be determined whether the target tensor is stored in a memory segment based on the target segment type in the target storage information.

[0059] Specifically, based on the target fragment type, if it is determined that the target tensor corresponding to the target tensor identifier is located in a memory fragment, the target processing node where the target tensor is located is obtained based on the target processing node identifier; Obtain the target tensor corresponding to the target tensor identifier from the memory slice of the target processing node.

[0060] In practical applications, when the target tensor is determined to be located in a memory shard, it can be read directly from the target processing node. Specifically, the nodes deployed by the preset machine learning model in the cloud processing platform can use a one-sided RDMA operation to read the target tensor from the target processing node. That is, in one or more embodiments of this specification, obtaining the target tensor from the memory shard of the target processing node includes: using a one-sided RDMA operation to read the target tensor from the memory shard of the target processing node to obtain the target tensor.

[0061] It should be noted that RDMA is the aforementioned technology that can bypass a remote central processing unit or kernel to provide low-latency, high-throughput data transmission. Therefore, in one or more embodiments of this specification, a one-sided RDMA operation can be understood as an operation where one node directly accesses the memory region of another node. During the execution of a one-sided RDMA operation, the CPU of the other node does not need to intervene in processing network requests, thus enabling low-latency communication between the two nodes. Specifically, in a one-sided RDMA operation, the initiating node can initiate read / write requests for data in the memory of the remote node. The CPU of the remote node is unaware of this data read / write request, i.e., the entire one-sided RDMA operation, and does not require the remote node to process the data. Because a one-sided RDMA operation does not require the remote node's CPU to participate in the data read / write process, the data read / write time is shorter, and the CPU utilization is lower.

[0062] In one or more embodiments of this specification, the target storage information of the target tensor corresponding to the target tensor identifier is obtained by querying according to the first correspondence relationship, wherein the first correspondence relationship represents the correspondence between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier, and the target storage information includes the target processing node identifier and the target fragment type; If it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, the target processing node where the target tensor is located is obtained, including: Based on the target shard type, if it is determined that the target tensor corresponding to the target tensor identifier is located in the storage shard, the target processing node where the target tensor is located is obtained based on the target processing node identifier.

[0063] In practical applications, when the target shard type obtained based on the first correspondence query is a storage shard, it confirms that the target tensor is stored in a storage shard. Therefore, the target tensor can be loaded from the storage shard of the target processing node into the memory shard. Specifically, the target tensor can be loaded from the storage shard of the target processing node into the memory shard via the internal bus of the processing node, such as the PCIe bus. This allows the process of loading the tensor from the storage shard to the memory shard to be completed within a single processing node, without needing to traverse a cross-node network, thus improving data migration efficiency and inference request processing efficiency.

[0064] In one or more embodiments of this specification, after the target tensor is cached in a memory shard of the target processing node, the target tensor can be retrieved from the memory shard of the target processing node using one-sided RDMA.

[0065] Step 308: Generate the inference result corresponding to the inference request based on the target tensor.

[0066] In one or more embodiments of this specification, after obtaining the target tensor, the target tensor can be input into a preset machine learning model, so that the preset machine learning model does not need to perform the calculation process of the target tensor, but directly reads and uses it to generate the inference result corresponding to the inference request based on the target tensor.

[0067] It should be noted that this specification does not impose specific restrictions on the reasoning requests and reasoning results. For example, if the preset machine learning model can be a question-answering model, then the information to be reasoned in the reasoning request can be the question information, and the reasoning result can be the response information. The specific data modality of the question information and the response information can be one or more of the following: image, text, and video.

[0068] In practical applications, a cloud processing platform can include model inference nodes. These nodes can be understood as those that deploy a pre-defined model to perform inference and generate inference results. The inference node can read the target tensor from a memory shard of the target processing node and generate inference results based on the pre-defined model and the target tensor. Alternatively, the cloud processing platform can send the generated target tensor to a third-party server that has the pre-defined model deployed on it, enabling the third-party server to generate inference results based on the pre-defined model and the target tensor.

[0069] In one embodiment of this specification, during the process of generating the inference result corresponding to the inference request, the cloud processing platform can obtain the target tensor identifier corresponding to the inference request; if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, the platform can obtain the target processing node where the target tensor is located; the platform can load the target tensor from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; and the platform can generate the inference result corresponding to the inference request based on the target tensor.

[0070] The above method sets up memory shards and storage shards in each processing node. When the target tensor requested by the inference request is located in the storage shard, the loading of the target tensor from the storage shard to the memory shard can be achieved within a single processing node. Then, the target tensor can be obtained from the memory shard. This avoids tensor loading across the network, improves tensor loading speed, and reduces network bandwidth contention between tensor loading traffic and inference request traffic, thereby improving inference request processing efficiency.

[0071] In one or more embodiments of this specification, the method further includes: During the inference process based on the tensors stored in the cloud processing platform, for each processing node, if it is determined that the memory utilization rate of the processing node is greater than the first utilization threshold, the first specified tensor in the processing node is moved to the storage shard of the processing node, and the storage information corresponding to the first specified tensor is updated.

[0072] In practical applications, when the memory utilization rate of a memory shard is too high, or when there are too many tensors stored in a memory shard, the tensors in the memory shard can be transferred to the storage shard of the processing node where the memory shard is located.

[0073] Specifically, for any processing node, if it is determined that the memory utilization rate of the memory shard corresponding to the processing node is greater than the first utilization threshold, a background thread can be invoked to periodically scan the tensors in the memory shard of the processing node, determine the first specified tensor, and migrate the first specified tensor to the storage shard of the processing node.

[0074] The first specified tensor can be determined based on preset rules, such as the usage of the tensor. Specifically, the first specified tensor can be a tensor that is not frequently used recently in the memory slice, or it can be determined in combination with the size of the tensor. Specifically, if it is determined that the memory capacity occupied by the tensor is greater than the preset memory capacity, the tensor that is not frequently used recently in the memory slice can be used as the first specified tensor. The first threshold can be determined based on the actual situation, and this manual does not impose specific restrictions.

[0075] In practical applications, the storage information corresponding to the first specified tensor can be updated based on the position migration of the first specified tensor, that is, the fragmentation type corresponding to the first specified tensor can be updated.

[0076] By using the above method, infrequently used tensors in memory shards are evicted to storage shards, reserving sufficient memory space for more frequently used tensors, thereby improving cache hit rate and speeding up inference.

[0077] In one or more embodiments of this specification, other methods can also be used to evict tensors from memory. For example, for any processing node, the access frequency of each tensor in the memory slice of that processing node can be obtained, and tensors with access frequencies lower than a preset access frequency threshold can be designated as first specified tensors and evicted from the memory slice to the storage slice. Alternatively, the size of the space occupied by each tensor can be obtained, and tensors with a space occupied greater than a preset space occupied threshold can be designated as first specified tensors and evicted from the memory slice to the storage slice.

[0078] In one or more embodiments of this specification, the method further includes: During the inference process based on the tensors stored in the cloud processing platform, for each processing node, if it is determined that the storage utilization rate of the processing node is greater than the second utilization threshold, the second specified tensor in the processing node is deleted from the storage shard of the processing node, and the storage information corresponding to the second specified tensor is updated.

[0079] In practical applications, when the storage utilization rate of a storage shard is too high, or when there are too many tensors stored in a storage shard, the tensors in the storage shard can be deleted. Specifically, for any processing node, if it is determined that the storage utilization rate of the storage shard corresponding to that processing node is greater than a second preset threshold, a background thread can be invoked to periodically scan the tensors in the storage shard of that processing node, identify the second specified tensor, and delete the second specified tensor from the storage shard of that processing node.

[0080] The second threshold can be determined based on the actual situation, and this manual does not impose specific restrictions.

[0081] The second specified tensor can be determined based on preset rules, such as tensor usage, specifically a tensor that is not frequently used in a storage fragment. Alternatively, it can be determined based on tensor size; specifically, if the tensor occupies more storage capacity than a preset storage capacity, the tensor with a storage capacity greater than the preset storage capacity can be used as the second specified tensor. By using the above method, infrequently used tensors in the storage fragments are evicted, reserving enough storage space for more frequently used tensors.

[0082] In one or more embodiments of this specification, the cloud processing platform may include model inference nodes, tensor management nodes, and multiple processing nodes.

[0083] The model inference node is used to obtain inference requests and can request tensor address mapping or memory allocation from the tensor management node. It can also read and write memory slices of the processing node through one-sided RDMA.

[0084] The tensor management node receives and processes requests from the model inference node. It handles requests for memory allocation, metadata queries, migration / deletion, and more. It can perform migrations of tensors from memory shards to storage shards and vice versa, and manages the primary mapping relationship—maintaining the correspondence between tensor identifiers and tensor storage information. This primary mapping relationship can be dynamically updated based on tensor migrations, deletions, and writes.

[0085] Furthermore, in practical applications, the tensor management node can also maintain a Least Recently Used (LRU) linked list to update hot and cold data, evict data, etc.

[0086] In practical applications, the data layer of a cloud processing platform may include a distributed, decoupled memory layer and storage layer. In one or more embodiments of this specification, the data layer consists of processing nodes, each used to store tensor data. The memory layer may consist of a master node and child nodes. The master node is responsible for memory resource management and coordination, while the child nodes provide actual physical memory and are accessible externally via RDMA. The storage layer can be understood as a local storage shard for each child node. Cold data in the child node's memory shard can be asynchronously written to disk to the cloud block storage mounted locally on that child node, and can be reloaded into the memory shard when needed. Thus, each processing node includes a storage shard and a memory shard. The processing node is used to store tensors, and this processing node can specifically be the aforementioned child node. The master node may not store tensor data but is used to manage the memory shards of each processing node.

[0087] In one or more embodiments of this specification, when it is determined that the target tensor identifier is not stored in the cloud processing platform, the target tensor can be calculated to store the calculated target tensor in the cloud processing platform. The tensor management node can generate storage information for the target tensor and can send the storage information corresponding to the target tensor, including the target shard type and the target processing node identifier, to the request acquisition node. The target processing node can use one-sided RDMA to write the target tensor into the memory shard.

[0088] In practical applications, a locking mechanism can also be used during the tensor writing process. After completing a one-sided RDMA operation, the lock is released to complete the write and prevent other nodes from accessing the tensor.

[0089] In one or more embodiments of this specification, if it is determined that the target tensor identifier is stored in a memory shard of the cloud processing platform, the target tensor can be read directly from the memory of the target processing node using one-sided RDMA.

[0090] In one or more embodiments of this specification, to balance low cost and high throughput bandwidth, the storage layer adopts the following storage medium organization: the storage shard of each processing node consists of multiple Cloud Block Storage (CBS) volumes. In practical applications, low-bandwidth, low-unit-cost cloud block storage volumes can be selected. For example, a CBS volume with a capacity of 100GB and a bandwidth of 100MB / s can be selected.

[0091] Furthermore, multiple CBS blocks on a processing node can be striped and aggregated using Redundant Array of Independent Disks (RAID-0) technology to form a single logical storage volume. Through RAID-0 striping, the total bandwidth of the aggregated logical storage volume equals the sum of the bandwidths of each individual CBS volume, and the total capacity equals the sum of the capacities of each individual CBS volume. For example, if a processing node is configured with four CBS volumes, each with a bandwidth of 100MB / s and a capacity of 100GB, then after RAID-0 aggregation, a single logical storage volume with a total bandwidth of 400MB / s and a total capacity of 400GB can be obtained.

[0092] Through the RAID-0 aggregation described above, multiple low-cost, small-capacity CBS volumes can be combined to obtain bandwidth comparable to or even higher than that of a single high-specification CBS volume, while avoiding the cost waste of purchasing large-capacity storage to obtain high bandwidth.

[0093] Furthermore, native file systems, such as the ext4 file system, can be mounted on the logical storage volume formed by RAID-0 aggregation. Specifically, the logical storage volume can be formatted using the `mkfs.ext4` command and mounted to a specified directory. By using a native file system, software overhead can be minimized, eliminating the need to introduce an additional user-space storage engine.

[0094] In one or more embodiments of this specification, in order to avoid the failure of the tensor management node, the first correspondence in the tensor management node can be backed up, such as persisted to an external cloud database, so that after the failure of the tensor management node is repaired, the tensor storage information query service can continue to be provided for the nodes and processing nodes deployed in the model based on the first correspondence backed up in the external cloud database.

[0095] Corresponding to the above method embodiments, this specification also provides a cloud processing platform, which includes at least one processing node, a model inference node, and a tensor management node.

[0096] In one or more embodiments of this specification, each processing node in the cloud processing platform includes storage shards and memory shards. The processing node is used to store tensors, such as key-value tensors, generated during the inference process of a pre-defined machine learning model.

[0097] A model inference node can be understood as a node used to perform inference, or a node where a pre-defined machine learning model is deployed. In practical applications, a pre-defined machine learning model, such as a question-answering model, can be deployed on the model inference node. Users can input information in the model client to initiate an inference request. The model inference node can then execute the inference request based on the request from the model client and obtain the inference result through the pre-defined machine learning model. During the inference process performed by the pre-defined machine learning model, tensor reuse is involved. The model inference node can obtain the necessary tensors from the processing node through the tensor management node.

[0098] A tensor management node can be understood as a node used to manage tensors and their storage locations, as well as to manage the various processing nodes. In one or more embodiments of this specification, the tensor management node can maintain the correspondence between tensor identifiers and tensor storage locations.

[0099] It should be noted that the preset machine learning model can be set according to actual needs. For example, it can be a text processing model specifically a question-answering model, an image processing model, or a multimodal model, etc.

[0100] In one or more embodiments of this specification, the model inference node is used to obtain the target tensor identifier corresponding to the inference request and send the target tensor identifier to the tensor management node; The tensor management node is used to receive the target tensor identifier and, if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, send a tensor loading instruction to the target processing node where the target tensor is located. The target processing node is configured to, in response to the tensor loading instruction, load the target tensor from the storage shard of the target processing node to the memory shard of the target processing node, and return loading completion information to the tensor management node. The tensor management node is used to receive the loading completion information and send the target processing node identifier of the target processing node to the model inference node; The model inference node is used to receive the target processing node identifier and read the target tensor corresponding to the target tensor identifier from the memory slice of the target processing node corresponding to the target processing node identifier.

[0101] In one or more embodiments of this specification, the model inference node can first obtain an inference request and determine the target tensor identifier corresponding to the inference request. Then, it can generate a tensor retrieval request carrying the target tensor identifier and send the tensor retrieval request to the tensor management node. The tensor management node can receive the tensor retrieval request carrying the target tensor identifier and determine the target storage information corresponding to the target tensor identifier based on a first stored relationship. This first relationship is the correspondence between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier. The target storage information may include a target processing node identifier and a target shard type, which determines which type of shard on which processing node the target tensor is stored.

[0102] In one or more embodiments of this specification, the tensor management node can generate a tensor loading instruction for the target tensor when it is determined that the target tensor is stored in a storage shard, i.e., when the target shard type is determined to be a storage shard, and send the tensor loading instruction to the target processing node. This tensor loading instruction instructs the target processing node to load the target tensor from the storage shard into the memory shard. The target processing node can then receive the tensor loading instruction and load the target tensor from its own storage shard into its own memory shard according to the instruction, so that the model inference node can quickly read the target tensor from the target processing node's memory. Furthermore, after the target processing node completes the loading, it can return loading completion information to the tensor management node, which can then send the target processing node identifier from the target storage information to the target processing node based on this loading completion information.

[0103] In one or more embodiments of this specification, a target processing node may receive a target processing node identifier and read a target tensor from the memory slice of the target processing node corresponding to the target processing node identifier. It may then execute an inference request based on the target tensor using a preset machine learning model to obtain inference results.

[0104] It should be noted that the technical solution of this cloud processing platform and the technical solution of the aforementioned data processing method are based on the same concept. For details not described in detail in the technical solution of the cloud processing platform, please refer to the description of the technical solution of the aforementioned data processing method, which will not be repeated here.

[0105] Furthermore, it is worth noting that the data processing methods provided in this manual are applicable to cache reuse scenarios for any type of tensor in a preset machine learning model. For example, the tensor mentioned above can be a key-value tensor, which can be used for cache reuse of key-value tensors, cache reuse of feature maps during model inference, cache reuse of embedded features, and so on.

[0106] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 4 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: The first acquisition module 402 is configured to acquire the target tensor identifier corresponding to the inference request; The second acquisition module 404 is configured to acquire the target processing node where the target tensor is located when it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard. The loading module 406 is configured to load the target tensor from the storage shard of the target processing node into the memory shard of the target processing node, so as to obtain the target tensor from the memory shard of the target processing node; The generation module 408 is configured to generate the inference result corresponding to the inference request based on the target tensor.

[0107] Optionally, the first acquisition module 402 is further configured to acquire a reasoning request, wherein the reasoning request carries information to be reasoned; input the information to be reasoned into a preset machine learning model to obtain a lexical sequence corresponding to the information to be reasoned; and determine the target tensor identifier corresponding to the information to be reasoned based on the lexical sequence.

[0108] Optionally, the first acquisition module 402 is further configured to divide the lexical sequence into multiple lexical subsequences; for each lexical subsequence, acquire the tensor identifier of the previous lexical subsequence, and determine the tensor identifier of the lexical subsequence using a hash algorithm based on the tensor identifier of the lexical subsequence and the tensor identifier of the previous lexical subsequence; and obtain the target tensor identifier corresponding to the information to be inferred based on the tensor identifiers corresponding to each lexical subsequence.

[0109] Optionally, the second acquisition module 404 is further configured to query and acquire the target storage information of the target tensor corresponding to the target tensor identifier according to the first correspondence relationship, wherein the first correspondence relationship represents the correspondence relationship between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier, and the target storage information includes the target processing node identifier and the target fragment type. The second acquisition module 404 is further configured to, based on the target fragment type, and if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage fragment, obtain the target processing node where the target tensor is located based on the target processing node identifier.

[0110] Optionally, the second acquisition module 404 is further configured to, based on the target fragment type, if it is determined that the target tensor corresponding to the target tensor identifier is located in a memory fragment, obtain the target processing node where the target tensor is located based on the target processing node identifier; and obtain the target tensor corresponding to the target tensor identifier from the memory fragment of the target processing node.

[0111] Optionally, the second acquisition module 404 is further configured to, when it is determined that the target tensor identifier does not exist in the first correspondence, generate a target tensor corresponding to the inference request based on a preset machine learning model, and determine the target fragment type and target processing node identifier corresponding to the target tensor to obtain the target storage information of the target tensor; add the target storage information and the target tensor identifier to the first correspondence, and store the target tensor in the target fragment of the processing node corresponding to the target processing node identifier, wherein the target fragment corresponds to the target fragment type.

[0112] Optionally, the device further includes a detection module; The detection module is configured to, during the inference process based on tensors stored in the cloud processing platform, for each processing node, if it is determined that the memory usage of the processing node is greater than a first usage threshold, move a first specified tensor in the processing node to the storage shard of the processing node and update the storage information corresponding to the first specified tensor.

[0113] Optionally, the detection module is further configured to, during the inference process based on tensors stored in the cloud processing platform, for each processing node, if it is determined that the storage utilization rate of the processing node is greater than a second utilization threshold, delete the second specified tensor in the processing node from the storage shard of the processing node and update the storage information corresponding to the second specified tensor.

[0114] In one embodiment of this specification, during the process of generating the inference result corresponding to the inference request, the cloud processing platform can obtain the target tensor identifier corresponding to the inference request; if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, the platform can obtain the target processing node where the target tensor is located; the platform can load the target tensor from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; and the platform can generate the inference result corresponding to the inference request based on the target tensor.

[0115] The aforementioned device sets up memory shards and storage shards in each processing node. Thus, when the target tensor requested by the inference request is located in the storage shard, the loading of the target tensor from the storage shard to the memory shard can be achieved within a single processing node, and the target tensor can be obtained from the memory shard. This avoids tensor loading across the node network, improves tensor loading speed, and reduces network bandwidth contention between tensor loading traffic and inference request traffic, thereby improving inference request processing efficiency.

[0116] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0117] See Figure 5 , Figure 5 This specification illustrates an architecture diagram of a data processing system according to one embodiment of the present specification. The data processing system may include a client 100 and a server 200. Client 100 is used to send inference requests to server 200; Server 200 is configured to receive the inference request and obtain the target tensor identifier corresponding to the inference request based on the inference request; if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, it obtains the target processing node where the target tensor is located; loads the target tensor from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; generates the inference result corresponding to the inference request based on the target tensor; and sends the inference result to client 100. Client 100 is also used to receive the inference results sent by server 200.

[0118] By applying the scheme of the embodiments in this specification, by setting memory shards and storage shards in each processing node, when the target tensor requested by the inference request is located in the storage shard, the loading of the target tensor from the storage shard to the memory shard can be realized within a single processing node, and then the target tensor can be obtained from the memory shard. This avoids tensor loading across the node network, improves tensor loading speed, and reduces network bandwidth competition between tensor loading traffic and inference request traffic, thereby improving inference request processing efficiency.

[0119] The data processing system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and the server 200 can be referred to as cloud devices, specifically the cloud processing platform. The cloud processing platform includes at least one processing node, and each processing node includes storage shards and memory shards. Multiple clients 100 can establish communication connections through the server 200. In the data processing scenario, the server 200 is used to provide data processing services between the multiple clients 100. Each client 100 can act as a sender or receiver, communicating through the server 200.

[0120] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In data processing scenarios, users can publish data streams to server 200 through client 100, server 200 can generate inference results based on the data stream, and push the inference results to other clients that have established communication.

[0121] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0122] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on a computing device and depends on the device or certain apps on the device to run. The computing device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured on the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0123] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0124] It is worth noting that the data processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the data processing methods provided in the embodiments of this specification. In other embodiments, the data processing methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0125] Figure 6A structural block diagram of a computing device 600 according to an embodiment of this application is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0126] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0127] In one embodiment of this application, the aforementioned components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0128] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0129] The processor 620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0130] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0131] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0133] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0134] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0137] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0139] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to a cloud processing platform, the cloud processing platform comprising at least one processing node, each processing node comprising storage shards and memory shards, the method comprising: Obtain the target tensor identifier corresponding to the inference request; According to the first correspondence, the target storage information of the target tensor corresponding to the target tensor identifier is obtained by querying. The first correspondence represents the correspondence between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier. The target storage information includes the target processing node identifier and the target fragment type. Based on the target shard type, if it is determined that the target tensor corresponding to the target tensor identifier is located in the storage shard, the target processing node where the target tensor is located is obtained based on the target processing node identifier; The target tensor is loaded from the storage shard of the target processing node into the memory shard of the target processing node to obtain the target tensor from the memory shard of the target processing node; Based on the target tensor, generate the inference result corresponding to the inference request.

2. The method as described in claim 1, wherein obtaining the target tensor identifier corresponding to the inference request includes: Obtain a reasoning request, wherein the reasoning request carries information to be reasoned; The information to be reasoned is input into a preset machine learning model to obtain the word sequence corresponding to the information to be reasoned. Based on the lexical sequence, the target tensor identifier corresponding to the information to be inferred is determined.

3. The method as described in claim 2, wherein determining the target tensor identifier corresponding to the information to be inferred based on the lexical sequence includes: The word sequence is divided into multiple word subsequences; For each word subsequence, obtain the tensor identifier of the previous word subsequence, and determine the tensor identifier of the word subsequence using a hash algorithm based on the word subsequence and the tensor identifier of the previous word subsequence. Based on the tensor identifiers corresponding to each word subsequence, the target tensor identifier corresponding to the information to be inferred is obtained.

4. The method of claim 1, further comprising: Based on the target fragment type, if it is determined that the target tensor corresponding to the target tensor identifier is located in a memory fragment, the target processing node where the target tensor is located is obtained based on the target processing node identifier; Obtain the target tensor corresponding to the target tensor identifier from the memory slice of the target processing node.

5. The method of claim 1, further comprising: If it is determined that the target tensor identifier does not exist in the first correspondence, a target tensor corresponding to the inference request is generated based on a preset machine learning model, and the target fragment type and target processing node identifier corresponding to the target tensor are determined to obtain the target storage information of the target tensor. The target storage information and the target tensor identifier are added to the first correspondence, and the target tensor is stored in the target fragment in the processing node corresponding to the target processing node identifier, wherein the target fragment corresponds to the target fragment type.

6. The method according to any one of claims 1 to 5, wherein the method further comprises: During the inference process based on the tensors stored in the cloud processing platform, for each processing node, if it is determined that the memory utilization rate of the processing node is greater than the first utilization threshold, the first specified tensor in the processing node is moved to the storage shard of the processing node, and the storage information corresponding to the first specified tensor is updated.

7. The method according to any one of claims 1 to 5, wherein the method further comprises: During the inference process based on the tensors stored in the cloud processing platform, for each processing node, if it is determined that the storage utilization rate of the processing node is greater than the second utilization threshold, the second specified tensor in the processing node is deleted from the storage shard of the processing node, and the storage information corresponding to the second specified tensor is updated.

8. A cloud processing platform, the cloud processing platform comprising at least one processing node, a model inference node, and a tensor management node, wherein, Each processing node includes storage shards and memory shards; The model inference node is used to obtain the target tensor identifier corresponding to the inference request and send the target tensor identifier to the tensor management node; The tensor management node is configured to receive the target tensor identifier, query and obtain the target storage information of the target tensor corresponding to the target tensor identifier according to a first correspondence relationship, wherein the first correspondence relationship represents the correspondence between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier, and the target storage information includes a target processing node identifier and a target shard type; based on the target shard type, if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, obtain the target processing node where the target tensor is located based on the target processing node identifier, and send a tensor loading instruction to the target processing node where the target tensor is located; The target processing node is configured to, in response to the tensor loading instruction, load the target tensor from the storage shard of the target processing node to the memory shard of the target processing node, and return loading completion information to the tensor management node; The tensor management node is used to receive the loading completion information and send the target processing node identifier of the target processing node to the model inference node; The model inference node is used to receive the target processing node identifier and read the target tensor corresponding to the target tensor identifier from the memory slice of the target processing node corresponding to the target processing node identifier.

9. A data processing apparatus located on a cloud processing platform, the cloud processing platform comprising at least one processing node, each processing node comprising a storage shard and a memory shard, comprising: The first acquisition module is configured to acquire the target tensor identifier corresponding to the inference request; The second acquisition module is configured to query and acquire the target storage information of the target tensor corresponding to the target tensor identifier according to the first correspondence relationship, wherein the first correspondence relationship represents the correspondence between the tensor identifier and the storage information of the tensor corresponding to the tensor identifier, and the target storage information includes the target processing node identifier and the target shard type; and based on the target shard type, if it is determined that the target tensor corresponding to the target tensor identifier is located in a storage shard, obtain the target processing node where the target tensor is located based on the target processing node identifier. A loading module is configured to load the target tensor from a storage shard of the target processing node into a memory shard of the target processing node, so as to obtain the target tensor from the memory shard of the target processing node; The generation module is configured to generate the inference result corresponding to the inference request based on the target tensor.

10. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images