Inference method, device, equipment, cluster, product and medium

Abstract

The invention provides a reasoning method and device, equipment, a cluster, a product and a medium. The method comprises the following steps: a gateway selects a first calculation instance from a plurality of calculation instances based on a first reasoning request, and sends the first reasoning request to the first calculation instance; the first calculation instance performs reasoning according to the first reasoning request to obtain a first reasoning result and an intermediate parameter; the first calculation instance stores the first reasoning request, the first reasoning result and the intermediate parameter in a memory pool; the first calculation instance returns the first reasoning request to the gateway; the gateway selects a second calculation instance from a plurality of calculation instances based on the first reasoning request, and sends the first reasoning request to the second calculation instance; the second calculation instance reads the first reasoning result and the intermediate parameter from the memory pool based on the first reasoning request for storage, and continues to conduct reasoning based on the first reasoning result and the intermediate parameter to obtain a second reasoning result, and the second reasoning result comprises the first reasoning result.

Description

Technical Field

[0001] The present application relates to artificial intelligence, and in particular to a reasoning method, apparatus, device, cluster, product and medium. Background Art

[0002] An "inference request" usually refers to a request or instruction to perform an inference operation on a computing instance. For example, in the fields of artificial intelligence and machine learning, when a computing instance is needed to analyze, predict, or draw conclusions about data, an inference request is sent to the relevant computing instance. As the number of inference requests increases, computing instances often need to process a large number of inference requests at the same time, and the load on the computing instance is very heavy. In some special cases, when a computing instance is performing a calculation, the inference is suddenly interrupted. At this time, the interrupted inference request will return to the waiting queue and wait to be processed by the computing instance again, which will cause all previous calculations to be wasted, resulting in a heavier load on the computing instance. Summary of the invention

[0003] The present application provides an inference method, apparatus, device, cluster, product and medium, which can avoid the waste of previous calculations and reduce the load of computing instances.

[0004] In a first aspect, a reasoning method is provided. A gateway selects a first computing instance from multiple computing instances based on a first reasoning request, and sends the first reasoning request to the first computing instance. The first computing instance performs reasoning according to the first reasoning request to obtain a first reasoning result and an intermediate parameter. The intermediate parameter is a parameter generated by the first computing instance in the process of calculating the first reasoning result. For example, the intermediate parameter is a key-value matrix, and the key-value matrix includes a key matrix and a value matrix, and the key matrix and the value matrix are generated by performing different linear transformations on the first reasoning request. The first computing instance stores the first reasoning request, the first reasoning result, and the intermediate parameter in a memory pool. The first computing instance returns the first reasoning request to the gateway. The gateway selects a second computing instance from the multiple computing instances based on the first reasoning request, and sends the first reasoning request to the second computing instance. The second computing instance reads the first reasoning result and the intermediate parameter storage from the memory pool based on the first reasoning request, and continues to perform reasoning based on the first reasoning result and the intermediate parameter to obtain a second reasoning result, wherein the number of tags included in the second reasoning result is greater than the number of tags included in the first reasoning result, and the second reasoning result includes the first reasoning result.

[0005] In the above solution, the first computing instance has completed some complex or time-consuming calculations, and passing its results to the second computing instance can avoid repeated calculations and save time and resources.

[0006] In some optional real number examples, the gateway selects the first computing instance from multiple computing instances based on the first reasoning request specifically as follows: calculating the score of each computing instance in the multiple computing instances under the first reasoning request, and selecting the first computing instance from the multiple computing instances according to the score of each computing instance of the multiple computing instances, wherein the score of the first computing instance is higher than the scores of part or all of the other computing instances in the multiple computing instances. Taking the calculation of the score of the first computing instance as an example, the first score of the first computing instance under the first reasoning request can be calculated according to the remaining video memory capacity ratio of the first computing instance, the number of inference request queues of the first computing instance, and the length of the first inference request. The remaining video memory capacity ratio of the first computing instance is the ratio of the remaining video memory capacity of the first computing instance to the available video memory capacity of the first computing instance, wherein the remaining video memory capacity of the first computing instance is the predicted remaining video memory capacity, or the actually collected remaining video memory capacity.

[0007] In the above scheme, when calculating the score of the first computing instance, the gateway not only considers the number of inference requests queued for the first computing instance and the length of the first inference request, but also considers the proportion of the remaining video memory capacity of the first computing instance to comprehensively obtain the optimal node.

[0008] In a second aspect, an inference method is provided. A second computing instance receives a first inference request sent by a gateway, and reads a first inference result and an intermediate parameter corresponding to the first inference request from a memory pool according to the first inference request, wherein the first inference result and the intermediate parameter are stored in the memory pool by the first computing instance. The second computing instance continues to infer based on the first inference result and the intermediate parameter to obtain a second inference result, wherein the number of tags included in the second inference result is greater than the number of tags included in the first inference result, and the second inference result includes the first inference result.

[0009] In some possible designs, the first inference result and the intermediate parameter are calculated by a first computing instance based on the first inference request.

[0010] In some possible designs, the second computing instance is selected by the gateway from multiple computing instances based on the first inference request.

[0011] In some possible designs, the intermediate parameters include a key matrix and a value matrix, and the key matrix and the value matrix are generated by performing different linear transformations on the first reasoning request. The second computing instance continues to reason based on the first reasoning result, the key matrix and the value matrix to obtain a second reasoning result.

[0012] The second computing instance continues to perform reasoning based on the first reasoning result and the intermediate parameter to obtain a second reasoning result, including: In a third aspect, an inference device is provided. The device comprises: A receiving module, used to receive a first inference request sent by a gateway, and read a first inference result and an intermediate parameter corresponding to the first inference request from a memory pool according to the first inference request, wherein the first inference result and the intermediate parameter are stored in the memory pool by a first computing instance; An inference module is used to continue inference based on the first inference result and the intermediate parameter to obtain a second inference result, wherein the number of tags included in the second inference result is greater than the number of tags included in the first inference result, and the second inference result includes the first inference result.

[0013] In a fourth aspect, a computing device is provided, wherein the computing device comprises a processor and a memory; The processor is used to execute instructions stored in the memory so that the computing device performs the operating steps of the method described in any one of the first aspects.

[0014] In a fifth aspect, a computing device cluster is provided, characterized in that it includes a plurality of computing devices, each computing device includes a processor and a memory; The processor of at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operating steps of the method described in any one of the first aspects.

[0015] In a sixth aspect, a computer program product comprising instructions is provided, wherein when the instructions are executed by a computing device cluster, the computing device cluster executes the operation steps of the method described in any one of the first aspects.

[0016] In a seventh aspect, a computer-readable storage medium is provided, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the operating steps of the method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a structural diagram of a scheduling system provided by this application; Figure 2 This is an example diagram of a scheduling system provided by the present application deployed on a cloud platform; Figure 3 It is a flowchart of a reasoning method provided by this application; Figure 4 It is a structural schematic diagram of an inference device provided by the present application; Figure 5 It is a structural schematic diagram of a computing device provided by the present application; Figure 6 It is a structural diagram of a computing device cluster provided by this application; Figure 7 It is a structural diagram of another computing device cluster provided by this application. DETAILED DESCRIPTION

[0018] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of a scheduling system provided by this application. Figure 1 As shown, the scheduling system includes a gateway 110, a computing instance and a memory pool 140. Among them, the computing instance includes a first computing instance 120 and a second computing instance 130. The gateway 110, the first computing instance 120, the second computing instance 130 and the memory pool 140 can be connected to each other in communication, and the above communication connection can be a wired connection or a wireless connection. The wired connection includes Ethernet, occupied lines, cables, etc., and the wireless connection includes a wireless local area network (wireless fidelity, Wi-Fi), a cellular network, etc., and can also include the Internet (internet) that supports both wired and wireless connections, an internal local area network (local area network, LAN), etc., and of course, other connection types can also be included, which are not specifically limited in this application.

[0019] The gateway 110 is the single entry point of the scheduling system. It acts as an intermediary between the front-end client (not shown) and the back-end computing instance, receives reasoning requests from the client, then routes these reasoning requests to the appropriate computing instance, and returns the response to the client. It can be a computing device or a computing device cluster consisting of multiple computing devices, or a chip or a chip system consisting of multiple chips, where the computing device can be a bare metal server (BMS), a virtual machine, or a container. BMS refers to a general physical server, such as an ARM server or an X86 server; a virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. All work that can be done in a physical computer can be done in a virtual machine. When creating a virtual machine in a computing device, part of the hard disk and memory capacity of the physical machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent basic input / output system (BIOS), hard disk and operating system, and can be operated like a physical machine; a container is a portable software unit that can combine an application and all its dependencies into a software package that is not limited by the underlying host operating system, so there is no need to build a complex environment, simplifying the process from application development to deployment. Edge computing devices are hardware devices close to data sources or data users. The chip can be implemented using a central processing unit (CPU), or a processor implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above-mentioned PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC) or any combination thereof, and this application does not make specific restrictions.It may also include processors or accelerator cards that have good performance when executing machine learning tasks and can improve the speed and efficiency of neural network training, such as graphics processing units (GPUs), neural network processing units (NPUs), data processing units (DPUs), tensor processing units (TPUs), etc., which are not specifically limited in this application.

[0020] A computing instance generally refers to an independent computing resource allocation unit. It can be understood as a virtual or physical computing environment with a specific configuration (such as the number of CPU cores, memory size, storage capacity, etc.) that can run specific applications, services or tasks. Computing instances (including the first computing instance 120 and the second computing instance 130) can be deployed in computing devices and computing device clusters. Among them, a computing device cluster refers to a system that connects multiple computing devices together through a network and works together to complete large-scale computing tasks or provide high-availability services. In other words, a computing device is a basic unit that constitutes a computing device cluster, and a computing device cluster is an organizational form in which multiple computing devices work together. When the first computing instance 120 and the second computing instance 130 are deployed on the same computing device, the first computing instance 120 and the second computing instance 130 can be deployed in different virtual machines or containers of the computing device, respectively, to ensure that the resources (such as CPU, memory, storage, etc.) between the first computing instance 120 and the second computing instance 130 are isolated from each other to avoid mutual interference. When the first computing instance 120 and the second computing instance 130 are deployed in a computing device cluster, the first computing instance 120 and the second computing instance 130 can be respectively deployed in different virtual machines, containers, and computing devices of the computing device cluster, so as to ensure that the resources (such as CPU, memory, storage, etc.) between the first computing instance 120 and the second computing instance 130 are isolated from each other to avoid mutual interference. In a specific embodiment, the first computing instance 120 and the second computing instance 130 can be used for reasoning, specifically, can be used for scientific computing and simulation, such as climate modeling, physical phenomenon simulation, chemical process analysis, etc.; can be used for data analysis and big data processing, such as processing massive amounts of data, performing data mining, machine learning, statistical analysis, etc.; can be used for artificial intelligence and deep learning, such as training neural network models, performing image recognition, speech processing, natural language processing, etc.; can be used for financial modeling and risk analysis, such as predicting market trends, assessing portfolio risks, performing credit scoring, etc.; can be used for bioinformatics. Analysis, for example, analysis of gene sequences, protein structure prediction, disease diagnosis models, etc.; it can be used for engineering design and simulation, for example, mechanical component design, electronic circuit simulation, building structure analysis, etc.; it can be used for image and video processing, for example, image rendering, special effects production, video encoding and decoding, etc.; it can be used for online games and virtual reality, for example, supporting server-side computing for massively multiplayer online games, and real-time rendering of virtual reality environments, etc.; it can be used for e-commerce and network services, for example, processing online transactions, recommendation systems, search engines, etc.; it can be used for autonomous driving, for example, training autonomous driving models, conducting real-time traffic analysis and decision-making, etc.

[0021] The memory pool 140 can be accessed by any computing instance. For example, the memory pool 140 can be accessed by the first computing instance, and can also be accessed by the second computing instance. The memory pool 140 is deployed on a computing device or a computing device cluster, and can also be deployed on a storage device or a storage array composed of multiple storage devices. Among them, the description of the computing device and the computing device cluster can refer to the above content, and will not be repeated here. The storage device can be a hard disk drive (HDD), a solid state disk (SSD), a mechanical hard disk (HDD), a U disk (universal serial bus, USB), a flash memory (flash), an SD card (secure digital memory Card, SDcard), a memory stick, etc., which is not specifically limited in this application. The storage array can be a redundant array of independent disks (RAID), a network attached storage (NAS), a storage area network (SAN), etc., which is not specifically limited in this application.

[0022] Optionally, the gateway 110 and the computing instance (including the first computing instance 120 and the second computing instance 130) can be deployed on the same computing device, or the gateway 110 and the computing instance (including the first computing instance 120 and the second computing instance 130) can be deployed on different computing devices in the same computing device cluster, or the gateway 110 and the computing instance (including the first computing instance 120 and the second computing instance 130) can be deployed on different computing devices in different computing device clusters, which is not specifically limited in this application.

[0023] Optionally, the computing instances (including the first computing instance 120 and the second computing instance 130) and the memory pool 140 can be deployed on the same computing device, or on different computing devices in the same computing device cluster, or on different computing device clusters, or the memory pool 140 can be deployed on a storage device and the computing instances can be deployed on a computing device. The storage device and the computing device can belong to the same or different computing device clusters, which is not specifically limited in this application.

[0024] As a possible implementation, Figure 1 In addition to being implemented in the above-mentioned forms, the architecture of the scheduling system shown can also be deployed in the form of a data center or a cloud platform, etc. Specifically, the scheduling system of the present application is used to provide cloud services to users, which will not be illustrated one by one here.

[0025] The above describes in detail possible deployment methods of the gateway 110, the first computing instance 120, the second computing instance 130, and the memory pool 140. In actual deployment, any of the above methods can be used for flexible deployment in combination with specific application scenarios and business requirements. The following is an exemplary description of the actual deployment method of the gateway 110, the first computing instance 120, the second computing instance 130, and the memory pool 140 in combination with specific application scenarios.

[0026] As an example of an application scenario in which a scheduling system is deployed on a local device, a user can use the computing instance to perform data query, real-time problem solving, technical support, data analysis and other operations. In this scenario, the gateway 110, the first computing instance 120, the second computing instance 130 and the memory pool 140 can be deployed on a local device. For example, the gateway 110, the first computing instance 120, the second computing instance 130 and the memory pool 140 are deployed on a local service or server cluster. The user can use an office computer to log in to the scheduling system and send an inference request to the computing instance. The computing instance performs inference based on the inference request and stores the inference request and the inference result in the memory pool 140, which are then returned to the user.

[0027] As an example of an application scenario where a scheduling system is deployed on a cloud platform, the scheduling system can be deployed on a cloud platform. For example, Figure 2 This is an example diagram of a scheduling system deployed on a cloud platform provided by this application. Figure 2 As shown, a user can initiate a purchase request for a cloud service through a client. The cloud service includes a sub-service that uses a computing instance for reasoning. After the client sends the purchase request to the cloud platform, the cloud platform can provide the client with cloud service usage rights for the computing instance (including the first computing instance and the second computing instance) and the memory pool, so that the user can send the reasoning request to the computing instance through the client. The computing instance can perform reasoning based on the reasoning request, obtain the reasoning result, and return it to the client.

[0028] Among them, the cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, security resources, etc., to meet the computing needs of the scheduling system under different scales and loads, and these computing resources can be dynamically scaled according to the usage requirements of the scheduling system to ensure the stable operation of the scheduling system and provide users with reliable reasoning services.

[0029] Optionally, the memory pool 140 and the computing instance may be services provided by the same cloud platform. Figure 2The cloud platform shown includes computing instances and a memory pool 140. Alternatively, the memory pool 140 and the computing instances may be services provided by different cloud platforms, in which case data communication between the memory pool 140 and the computing instances may be achieved through a hybrid cloud architecture, that is, the memory pool 140 is deployed in data center A, the computing instances are deployed in data center B, and data centers A and B achieve data communication between the computing instances and the memory pool through their respective cloud platforms.

[0030] It is worth mentioning that Figure 2 In the example shown, the scheduling system is deployed and implemented in a cloud platform. Figure 2 This is merely an exemplary description of the scheduling system involved in this application and is not specifically limited in this application.

[0031] Optionally, the scheduling system can also be implemented through deployment on multiple cloud platforms. For example, the gateway is deployed in a local server, the first computing instance is deployed in a first cloud platform, the second computing instance is deployed in a second cloud platform, and the memory pool is deployed in a third cloud platform. The above examples are for illustration only and are not specifically limited in this application.

[0032] Optionally, the scheduling system can also be implemented through multiple non-cloud platform deployments. For example, the gateway and the first computing instance are deployed on the internal office server of Company A, and the second computing instance is deployed on the internal office server of Company B. Company A and Company B have a cooperative relationship. The above examples are for illustration only and are not specifically limited in this application.

[0033] Optionally, the scheduling system can also be implemented through multiple cloud platforms and non-cloud platforms. For example, the gateway and the memory pool are deployed on the internal office server of the enterprise, the first computing instance is deployed on the first cloud platform, and the second computing instance is deployed on the second cloud platform. The above examples are for illustration only and are not specifically limited in this application.

[0034] It should be understood that the present application does not limit the first computing instance, the second computing instance, and the memory pool. Figure 2 The system structure shown is used as an example for explanation.

[0035] It should be understood that the above application scenarios are used for illustration purposes only. The gateways, computing instances, and memory pools can be flexibly deployed according to actual business needs, and examples are not given here one by one.

[0036] The above describes the possible system architecture of the present application. The following briefly describes how the system architecture implements the scheduling method provided by the present application.

[0037] The gateway 110 is used to receive a first reasoning request, select a first computing instance from multiple computing instances according to the first reasoning request, and send the first reasoning request to the first computing instance 120. The reasoning request is an instruction or input submitted to a computing instance with reasoning capability to require it to analyze, infer and draw conclusions. The reasoning request can be an instruction to translate a specified sentence, for example, please translate "What day is today?", etc., it can be a question raised by a user, for example, "What are the attractions in Beijing?", etc., it can be an instruction to summarize a paragraph, for example, please write a summary of no more than 300 words based on the content of the above paragraph, etc., it can be an instruction to generate text, images, etc., for example, please generate an image of a swallow, etc. In a specific implementation, the above-mentioned first reasoning request can be a natural language text input by the user through the client, or it can be a natural language text generated by the client based on speech recognition technology after the user inputs voice through the client, or it can be a text extracted by image recognition, or it can be a natural language text extracted by video text extraction technology, or data of user questions in other formats, which is not specifically limited in this application.

[0038] The gateway 110 may select the first computing instance from the multiple computing instances according to the first reasoning request in the following manner: first, respectively calculate the score of each computing instance under the first reasoning request, and then select the first computing instance from the multiple computing instances according to the score of each computing instance. Here, the score of the first computing instance may be the highest score of the multiple computing instances, or the score of the first computing instance may be the second highest score of the multiple computing instances, or the score of the first computing instance may be the third highest score of the multiple computing instances, and so on.

[0039] The following will take the first computing instance as an example to describe in detail how to determine the score of the computing instance: the score of the first computing instance is calculated according to the remaining video memory capacity of the first computing instance, the number of inference requests queued for the first computing instance, and the length of the first inference request. It can be understood that the scores of other computing instances can also refer to the calculation process of the score of the first computing instance.

[0040] The remaining video memory capacity ratio of the first computing instance is equal to the remaining video memory capacity of the first computing instance divided by the available video memory capacity of the first computing embodiment. Among them, the remaining video memory capacity of the first computing instance is equal to the available video memory capacity of the first computing embodiment minus the capacity already used by the first computing instance. In a specific example, each computing instance can occupy a certain available video memory capacity, for example, 10G, 20G, etc., for storing reasoning requests and / or tags output according to reasoning requests, etc. The available video memory capacity of different computing instances can be independent of each other, that is, each computing instance can only access its own available video memory capacity, and does not access the available video memory capacity of other computing instances. Different computing embodiments can occupy the same available video memory capacity or different available video memory capacities, which are not specifically limited here. Here, the larger the remaining video memory capacity ratio of the first computing instance, the higher the score of the first computing instance, and the smaller the remaining video memory capacity ratio of the first computing instance, the lower the score of the first computing instance. Here, the remaining video memory capacity of the first computing instance can be the remaining video memory capacity actually collected, or it can be the predicted remaining video memory capacity, which is not specifically limited here.

[0041] The number of inference request queues of a computing instance is the number of inference requests that are assigned to the computing instance and have not been processed by the computing instance and are in the queue. Here, each computing instance can be assigned a certain number of inference requests. If the computing instance is unable to process the assigned inference requests in time, the inference requests that are unable to be processed in time will be queued and wait for the computing instance to process. The number of inference request queues of a computing instance is related to the number of computing instances assigned and the processing capacity of the computing instance. Under the condition that other conditions are the same, the more inference requests assigned to the computing instance, the more inference request queues of the computing instance, and the fewer inference requests assigned to the computing instance, the fewer inference request queues of the computing instance; under the condition that other conditions are the same, the stronger the processing capacity of the computing instance, the fewer the number of inference request queues of the computing instance, and the weaker the processing capacity of the computing instance, the more the number of inference request queues of the computing instance. Here, the more the number of inference request queues of the first computing instance, the lower the score of the first computing instance; the fewer the number of inference request queues of the first computing instance, the higher the score of the first computing instance.

[0042] The length of the first reasoning request is the length of the reasoning request, which usually refers to the scale or size of the input content that constitutes the reasoning request. If it is a text-related reasoning request, the length may be measured by the number of characters, words, sentences, etc. If it is an image-related reasoning request, the length may be measured by the resolution, number of pixels, color depth, etc. of the image. For example, if the first reasoning request is a file-related reasoning request, the length of the first reasoning request is the number of words in the input text length. Here, the longer the length of the first reasoning request, the lower the score of the first computing instance; the shorter the length of the first reasoning request, the higher the score of the first computing instance.

[0043] Since indicators such as the number of inference requests queued for the computing instance and the remaining video memory capacity of the first computing instance change in real time, the number of inference requests queued for the computing instance and the remaining video memory capacity of the first computing instance can be collected by the gateway when needed, or the computing instance can report them to the gateway periodically, which is not specifically limited here.

[0044] It can be understood that in the above example, the influencing factors of the score of the first computing instance include the remaining video memory capacity of the first computing instance, the number of inference requests queued for the first computing instance, and the length of the first inference request to calculate the score of the first computing instance. In actual applications, the influencing factors of the score of the first computing instance may include more or fewer influencing factors. For example, the influencing factors of the score of the first computing instance may include the remaining video memory capacity of the first computing instance and the number of inference requests queued for the first computing instance.

[0045] In a specific embodiment, the score of the computing instance is calculated according to the remaining video memory capacity ratio of the computing instance, the number of inference requests queued for the computing instance, and the length of the first inference request, which can be implemented by using the following formula: score=ratio×0.2+1 / amount×0.5+1 / length×0.3; Among them, score is the score of the calculation instance, ratio is the ratio of the remaining video memory capacity of the calculation instance, and length is the length of the first inference request. It can be understood that 0.2, 0.5, and 0.3 in the above formula are only specific examples. In actual applications, other values ​​can also be used, such as 0.15, 0.5, and 0.35, etc., as long as the sum of these values ​​is 1.

[0046] The first computing instance 120 performs reasoning according to the first reasoning request to obtain a first reasoning result and intermediate parameters.

[0047] The first inference result is the front part of the inference result. The inference result is a plurality of tags obtained by the first computing instance through inference according to the first inference request. Among them, tags usually refer to the basic units of text. For example, for the sentence "Hello, how are you?", the tags are "Hello" "," "how" "are" "you" "?", and for another example, for "The weather is nice today.", the tags are "today" "weather" "nice" ".".

[0048] For example, assuming that the first reasoning request is "What are the attractions in Beijing?", the first computing instance performs reasoning based on the first reasoning request "What are the attractions in Beijing?" to obtain a reasoning result containing multiple tags, such as "The Palace Museum: the royal palace of the Ming and Qing dynasties in China, a world cultural heritage, with rich cultural relics and magnificent buildings." Then, the first reasoning result can be "The Palace Museum: the royal palace of the Ming and Qing dynasties in China, a world cultural heritage", "The Palace Museum: the royal palace of the Ming and Qing dynasties in China", and so on.

[0049] The intermediate parameter is a parameter obtained by the first computing instance when calculating the first inference result. Taking the first computing instance using the self-attention model as an example, the intermediate parameter can be a key value matrix. The formula of the self-attention model is shown in formula (1).

[0050]

[0051] in, For the first reasoning request, is the query matrix corresponding to the first reasoning request, used to indicate the current focus or information requirement of the model, is the key matrix corresponding to the first reasoning request, including identification information of each position in the first reasoning request, is the value matrix corresponding to the first reasoning request, used to store the actual value or information content corresponding to the key matrix, , , The three matrices are Generated by performing three different linear transformations, is the dimension of the model. Therefore, the key-value matrix can include a key matrix and a value matrix.

[0052] It can be understood that the key value matrix here may include the key value matrix when the latest mark is calculated and the key value matrix obtained by calculating the mark before the latest mark. However, in order to reduce the waste of computing resources and storage resources, only the most recent part of the key value matrix can be stored. If the required calculation accuracy is higher, more key value matrices can be stored, and if the required calculation speed is faster, fewer key value matrices can be stored.

[0053] When the first computing instance 120 performs reasoning according to the first reasoning request, it has two characteristics: (1) When the first computing instance 120 performs reasoning, it performs reasoning in batches. In addition to receiving the first reasoning request sent by the gateway, the first computing instance 120 can also receive other reasoning requests sent by the gateway. The first reasoning request and other reasoning requests are stored in the running queue of the first computing instance, and reasoning is performed in batches. When the first computing instance 120 performs reasoning on the first reasoning request and other reasoning requests, if the first computing instance receives a new reasoning request, the new reasoning request is stored in the waiting queue of the first computing instance. When the first reasoning request and other reasoning requests are completed, the new reasoning request in the waiting queue is transferred to the running queue for reasoning. Since the first computing instance will process multiple reasoning requests at the same time, if the reasoning results output by other reasoning requests are too long and occupy a large amount of video memory, the first reasoning request will be interrupted.

[0054] (2) When the first computing instance 120 outputs the inference result, it outputs it token by token. For example, it may first output "Palace Museum", then "Museum", then ":", then "China", and so on. Because when the first computing instance generates the inference result, it predicts a token each time, and continues to generate the next token based on the new token until a complete sentence is generated or a certain stop condition is reached, the inference result is output token by token.

[0055] Therefore, if the first reasoning request outputs the first reasoning result after reasoning, the first computing instance will interrupt the reasoning of the first reasoning request because other reasoning requests occupy a large amount of video memory. Here, the first reasoning result is used to indicate that the reasoning of the first reasoning request by the first computing instance has not been completed. If the first reasoning request is not interrupted, the first computing instance will continue to output tags after outputting the first reasoning result until all the reasoning results are output. Continuing with the first reasoning request of "What are the attractions in Beijing?" as an example, after the first computing instance outputs tags such as "The Palace Museum: a royal palace of the Ming and Qing dynasties in China, a world cultural heritage, with rich cultural relics and magnificent architecture", it can continue to output tags such as "with rich cultural relics and magnificent architecture".

[0056] The reasons for the interruption of computing instances may include the following: hard disk failure, such as CPU overheating, memory failure, hard disk damage, power supply problems, etc.; network problems, such as network connection interruption, high network latency, packet loss, etc., resulting in the inability to transmit the data required for computing in time; insufficient resources, such as insufficient memory, high CPU usage, and storage space exhaustion; software errors, such as loopholes or errors in the computing program, resulting in runtime crashes or exceptions; operating system failures, such as operating system crashes, blue screens, etc.; concurrent conflicts, such as conflicts when multiple processes or threads access and modify shared resources at the same time, resulting in computing interruptions; external interference, such as electromagnetic interference, physical vibration, etc., affecting the normal operation of computing devices; malicious attacks, such as attacks by viruses and malware, destroying the operating environment of computing instances, etc. In addition, computing instances can also be interrupted by users actively. For example, if the user thinks that the running speed of the first computing instance is too slow, the load of the first computing instance is too heavy, and the first computing instance needs to be checked, etc., the user can also actively interrupt and hand over the remaining reasoning work to other computing instances for processing. Alternatively, the working status of the first computing instance is normal, but a new computing instance is added to the system. The first computing instance can also interrupt the reasoning work of the first computing instance and hand over the remaining reasoning work to the newly added computing instance for processing.

[0057] The first computing instance 120 is used to send the first reasoning request, the first reasoning result and the intermediate parameters to the memory pool 140. Accordingly, the memory pool 140 is used to receive the first reasoning request, the first reasoning result and the intermediate parameters sent by the first computing instance.

[0058] The first reasoning request, the first reasoning result, and the intermediate parameters can be stored in the memory pool in the form of a database (e.g., a relational database or a non-relational database, etc.), a file (e.g., a text file, a binary file, JavaScript object notation, extensible markup language, etc.), a cache system, a distributed storage system (e.g., a Hadoop distributed file system, an object storage service), a data warehouse, etc.

[0059] Taking the storage in the memory pool in the form of a non-relational database as an example, the first inference request is hashed to obtain a hash value, and the storage location of the first inference request in the hash table is determined according to the hash value, and the hash value is used as the key, and the first inference result and the intermediate parameters can be stored as the value. In this way, it is convenient to find the first inference result and the intermediate parameters according to the first inference request. Alternatively, the first inference request can be directly used as the key and the first inference result can be stored as the value.

[0060] Taking storage in a memory pool in the form of a relational database as an example, the first inference request, the first inference result, and the intermediate parameters can be respectively used as data in a table of different columns in the same row of the table, or the first inference request, the first inference result, and the intermediate parameters can be respectively used as data in a table of different rows in the same column of the table.

[0061] Taking the storage in the memory pool in the form of a file as an example, the first inference request can be used as the name of the file, and the first inference result and the intermediate parameters can be stored as the content of the file, so as to facilitate finding the first inference result and the intermediate parameters according to the first inference request. Alternatively, the first inference request can be hashed to obtain a hash value, and the hash value can be used as the name of the file, and the first inference request, the first inference result and the intermediate parameters can be stored as the content of the file.

[0062] It can be understood that the above storage method is only a specific example. In actual application, storage can also be performed in other ways, which are not specifically limited here.

[0063] In addition to storing the first reasoning request, the first reasoning result, and the intermediate parameters in the memory pool, other related data may also be stored in the memory pool. Here, the related data may include one or more of the parameters of the model of the first computing instance, description information of the first reasoning request (such as the time of reasoning, the execution environment, the related user or business context, etc.), data source information (for example, the data collection channel, the collection time, etc.), data quality assessment indicators, data annotation information (for example, the annotator, the annotation time, the annotation version, etc.), and the like.

[0064] The first reasoning result obtained by the first computing instance during the reasoning process of the first reasoning request can be stored in the memory pool. However, once the first computing instance completes the reasoning of the first reasoning result and obtains the complete reasoning result, the first reasoning request and the complete reasoning result can continue to be stored in the memory pool, or the first reasoning request and the complete reasoning result can be deleted from the memory pool immediately and transferred to other storage devices, for example, to the private memory of the first computing instance, or the first reasoning request and the complete reasoning result can be stored in the memory for a short time and then transferred to other storage devices. As for the intermediate parameters, after obtaining the complete reasoning result, you can choose to delete the intermediate parameters or not.

[0065] The first computing instance 120 is used to return the first reasoning request to the gateway 110. Accordingly, the gateway 110 is used to receive the first reasoning request returned by the first computing instance 120.

[0066] The gateway 110 is used to select a second computing instance 130 from multiple computing instances according to the first reasoning request, and send the first reasoning request to the second computing instance 130 .

[0067] The gateway 110 may select the second computing instance 130 from the multiple computing instances according to the first reasoning request in the following ways: (1) The user specifies a second computing instance from multiple computing instances. For example, the second computing instance may be an idle computing instance, or the second computing instance may be a computing instance with relatively strong processing capability or other performance indicators (for example, bandwidth, storage capacity), or the second computing instance may be a newly added computing instance.

[0068] (2) The gateway 110 calculates the score of each computing instance (here, the computing instance other than the first computing instance) under the first reasoning request, and then selects a second computing instance from the multiple computing instances according to the score of each computing instance. For each computing instance, the score of the computing instance can be calculated according to the remaining video memory capacity of the computing instance, the number of inference requests queued for the computing instance, and the length of the first inference request. The gateway can use the same formula as the last time the gateway calculated the score of the computing instance to calculate the score of the computing instance, or it can use a different formula. For example, when the gateway last calculated the score of the computing instance, in the formula used, the coefficient corresponding to the remaining video memory capacity ratio of the first computing instance is 0.2, the coefficient corresponding to the number of inference requests queued for the first computing instance is 0.5, and the coefficient corresponding to the length of the first inference request is 0.2. Then, when the gateway calculates the score of the computing instance this time, in the formula used, the coefficient corresponding to the remaining video memory capacity ratio of the first computing instance can be adjusted to 0.3, the coefficient corresponding to the number of inference requests queued for the first computing instance is adjusted to 0.45, and the coefficient corresponding to the length of the first inference request is adjusted to 0.15. When calculating the score of the computing instance this time, the gateway can use the remaining video memory capacity of each computing instance and the number of inference request queues of each computing instance used in the last calculation of the score of the computing instance for calculation, or it can re-collect the remaining video memory capacity of each computing instance and the inference request queue of each computing instance, and use the newly collected remaining video memory capacity of each computing instance and the inference request queue of each computing instance to calculate the score of each computing instance.

[0069] After receiving the first reasoning request, the gateway may process the first reasoning request immediately, or rewrite the first reasoning request into the waiting queue for queuing. Alternatively, after receiving the first reasoning request, the gateway may set a priority level for the first reasoning request, and determine when to process the first reasoning request based on the priority level of the first reasoning request. The priority level of the first reasoning request may be set as needed, for example, the priority level of the first reasoning request may be any one of high, medium, low, etc.

[0070] The second computing instance 130 searches the memory pool for the first reasoning result according to the first reasoning request, and continues to reason according to the first reasoning result to obtain the second reasoning result.

[0071] The second computing instance 130 searches the memory pool for the first reasoning result according to the first reasoning request and may include the following implementations: (1) When the first reasoning request and the first reasoning result are stored in a non-relational database, the hash value of the first reasoning request can be calculated, and the possible position in the hash table can be determined according to the hash value. If a matching key is found at the position, the value corresponding to the key (i.e., the partial reasoning request and the intermediate parameter) is returned; or the first reasoning request and each key are compared respectively, the key that is the same as the first reasoning request is found, and the value corresponding to the key (i.e., the partial reasoning request and the intermediate parameter) is returned.

[0072] (2) When the first reasoning request and the first reasoning result are stored in a relational database, the first reasoning request and each element in the row or column storing the reasoning request can be compared to find the reasoning request in the row or column that is the same as the first reasoning request, and return the first reasoning result and intermediate parameters corresponding to the reasoning request.

[0073] (3) When the first reasoning request and the first reasoning result are stored in the form of a file, the hash value of the first reasoning request can be calculated, the corresponding file can be found according to the hash value, the reasoning request and the first reasoning result and intermediate parameters corresponding to the reasoning request can be read from the file, and then the first reasoning request can be compared with the reasoning request read from the file. If the first reasoning request is the same as the reasoning request read from the file, the first reasoning result and intermediate parameters corresponding to the reasoning request are returned; alternatively, the corresponding file can be found according to the first reasoning request, and the first reasoning result and intermediate parameters can be read from the file.

[0074] It can be understood that the above implementation method is only a specific example. In actual applications, when the storage methods of the first reasoning request, the first reasoning result and the intermediate parameters are different, the first reasoning result found from the memory pool will also change accordingly, which is not specifically limited here.

[0075] The process of the second computing instance 130 continuing to reason based on the first reasoning result and the intermediate parameters to obtain the second reasoning result can be: the second computing instance 130 can continue to reason based on the last mark in the first reasoning result and the intermediate parameters to obtain a new mark and a new intermediate parameter, and then use the new mark, the intermediate parameter and the new intermediate parameter to reason to obtain an updated mark, and so on, until the last mark is obtained. It can be understood that the above scheme is explained by using the example of using the n-1th mark and all the intermediate parameters before the n-1th mark when reasoning the nth mark. If the n-1th mark and the k intermediate parameters before the n-1th mark are used when reasoning the nth mark, then, each time a new intermediate parameter is calculated, an old intermediate parameter will be discarded, and the other calculation methods are the same.

[0076] See also Figure 3 , Figure 3 This is a flowchart of a reasoning method provided by this application. Figure 3 As shown, the reasoning method of the present application includes the following steps: S101: The gateway selects a first computing instance from multiple computing instances based on a first reasoning request.

[0077] The process of the gateway selecting the first computing instance from multiple computing instances based on the first reasoning request includes: the gateway calculates the score of each computing instance in the multiple computing instances under the first reasoning request. The gateway selects the first computing instance from the multiple computing instances according to the score of each computing instance of the multiple computing instances. Among them, the score of the first computing instance is higher than the scores of some or all of the computing instances in the multiple computing instances. In other words, the score of the first computing instance can be the highest score among the multiple computing instances, the second highest score among the multiple computing instances, the third highest score among the multiple computing instances, and so on.

[0078] Taking the first calculation instance as an example, the score of the first calculation instance can be calculated in the following way: The first score of the first computing instance under the first inference request is calculated according to the remaining video memory capacity ratio of the first computing instance, the number of inference requests queued for the first computing instance, and the length of the first inference request. The remaining video memory capacity ratio of the first computing instance is the ratio of the remaining video memory capacity of the first computing instance to the available video memory capacity of the first computing instance. The remaining video memory capacity of the first computing instance is the predicted remaining video memory capacity, or the actually collected remaining video memory capacity.

[0079] Since indicators such as the number of inference requests queued for the computing instance and the remaining video memory capacity of the first computing instance change in real time, the number of inference requests queued for the computing instance and the remaining video memory capacity of the first computing instance can be collected by the gateway when needed, or the computing instance can report them to the gateway periodically, which is not specifically limited here.

[0080] S102: The gateway sends a first reasoning request to the first computing instance. Correspondingly, the first computing instance receives the first reasoning request sent by the gateway.

[0081] S103: The first computing instance performs reasoning according to the first reasoning request to obtain a first reasoning result and intermediate parameters.

[0082] The intermediate parameters are parameters generated by the first computing instance in the process of calculating the first reasoning result. The first reasoning result is the front part of the reasoning result. The reasoning result is a plurality of tags obtained by the first computing instance according to the first reasoning request. Among them, tags usually refer to the basic units of text. For example, for the sentence "Hello, how are you?", the tags are "Hello" "," "how" "are" "you" "?", and for another example, for "The weather is nice today.", the tags are "today" "weather" "nice" ".".

[0083] For example, assuming that the first reasoning request is "What are the attractions in Beijing?", the first computing instance performs reasoning based on the first reasoning request "What are the attractions in Beijing?" to obtain a reasoning result containing multiple tags, such as "The Palace Museum: the royal palace of the Ming and Qing dynasties in China, a world cultural heritage, with rich cultural relics and magnificent buildings". Then, the first reasoning result can be "The Palace Museum: the royal palace of the Ming and Qing dynasties in China, a world cultural heritage", "The Palace Museum: the royal palace of the Ming and Qing dynasties in China", and so on.

[0084] The intermediate parameter is a parameter obtained when the first computing instance calculates the first reasoning result. The intermediate parameter may be a key-value matrix, which includes a key matrix and a value matrix, and the key matrix and the value matrix are generated by performing different linear transformations on the first reasoning request.

[0085] S104: The first computing instance sends the first reasoning request, the first reasoning result, and the intermediate parameters to the memory pool. Correspondingly, the memory pool receives the first reasoning request, the first reasoning result, and the intermediate parameters sent by the first computing instance.

[0086] The memory pool can be used by both the first computing instance and the second computing instance at the same time. That is, the data in the memory pool can be read and written by the first computing instance, and can also be read and written by the second computing instance.

[0087] Since the memory pool can be used by multiple computing instances at the same time, in addition to storing the first reasoning request and the first reasoning result output by the first computing instance, the memory pool also needs to store the reasoning requests assigned to the first computing instance for processing by the first computing instance and the corresponding first reasoning results, as well as the reasoning requests assigned to these computing instances for processing by other computing instances and the corresponding first reasoning results. Therefore, the data that needs to be stored in the memory pool is often very large. Therefore, it is necessary to manage the data in the memory pool, for example, check the data in the memory pool, delete the data that is no longer needed or compress the data that can be compressed to free up space; re-evaluate and reasonably adjust the size setting of the memory pool according to actual needs; migrate some older or less frequently used data to other storage media, such as disk storage; increase the physical memory of the memory pool; reduce the amount of cached data or shorten the retention time of cached data, etc.

[0088] S105: The first computing instance returns the first inference request to the gateway. Correspondingly, the gateway receives the first inference request returned by the first computing instance.

[0089] S106: The gateway selects a second computing instance from multiple computing instances based on the first reasoning request.

[0090] After receiving the first reasoning request, the gateway may process the first reasoning request immediately, or rewrite the first reasoning request into the waiting queue for queuing. Alternatively, after receiving the first reasoning request, the gateway may set a priority level for the first reasoning request, and determine when to process the first reasoning request based on the priority level of the first reasoning request. The priority level of the first reasoning request may be set as needed, for example, the priority level of the first reasoning request may be any one of high, medium, low, etc.

[0091] S107: The gateway sends the first reasoning request to the second computing instance. Correspondingly, the second computing instance receives the first reasoning request sent by the gateway.

[0092] S108: The second computing instance reads the first reasoning result and the intermediate parameter storage from the memory pool based on the first reasoning request, and continues to reason based on the first reasoning result and the intermediate parameter to obtain a second reasoning result.

[0093] The number of tags included in the second inference result is greater than the number of tags included in the first inference result, and the second inference result contains the first inference result.

[0094] The present application also provides a scheduling device, such as Figure 4 As shown, including: The receiving module 210 is used to receive a first inference request sent by the gateway, and read a first inference result and an intermediate parameter corresponding to the first inference request from a memory pool according to the first inference request, wherein the first inference result and the intermediate parameter are stored in the memory pool by the first computing instance; The reasoning module 220 is used to continue reasoning based on the first reasoning result and the intermediate parameter to obtain a second reasoning result, wherein the number of tags included in the second reasoning result is greater than the number of tags included in the first reasoning result, and the second reasoning result includes the first reasoning result.

[0095] The receiving module 210 and the reasoning module 220 may be implemented by software or hardware. For example, the implementation of the reasoning module is described below by taking the reasoning module as an example. Similarly, the implementation of the receiving module may refer to the implementation of the reasoning module.

[0096] As an example of a software functional unit, a reasoning module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the reasoning module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Generally, a region may include multiple AZs.

[0097] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0098] As an example of a hardware functional unit, the reasoning module may include at least one computing device, such as a server, etc. Alternatively, the reasoning module may also be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a device implemented by a programmable logic device (PLD). The above-mentioned PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0099] The multiple computing devices included in the reasoning module can be distributed in the same region or in different regions. The multiple computing devices included in the reasoning module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the reasoning module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, etc.

[0100] It should be noted that, in other embodiments, the receiving module 210 can be used to execute the receiving action in step S108 of the scheduling method; the reasoning module 220 can be used to execute the reasoning action in step S108 of the scheduling method, the receiving module 210 can execute the receiving action and the reasoning action in step S108 of the scheduling method, and the reasoning module 220 can execute the receiving action and the reasoning action in step S108 of the scheduling method.

[0101] The present application also provides a chip system, which includes a processor and a power supply circuit, the power supply circuit is used to supply power to the processor, and the processor is used to execute the operation steps corresponding to the scheduling method. For the sake of brevity, it is not repeated here. Among them, the processor can be implemented by a GPU, and can also be implemented by a computing device such as a DPU, NPU, XPU, SoC, offload card, and acceleration card.

[0102] The present application also provides a computing device 300. Figure 5 As shown, computing device 300 includes: bus 302, processor 304, memory 306 and communication interface 308. Processor 304, memory 306 and communication interface 308 communicate through bus 302. Computing device 300 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 300.

[0103] The bus 302 may be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. Among them, the unified bus is also called a Lingqu bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 302 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 302 may include a path for transmitting information between various components of the computing device 300 (e.g., the memory 306, the processor 304, and the communication interface 308). Among them, the unified bus may also be called a Lingqu bus.

[0104] The processor 304 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, etc.

[0105] The memory 306 may include a volatile memory, such as a random access memory (RAM). The processor 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). In addition, the memory 306 may also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0106] It is worth noting that the same type of storage medium can be configured in the same computing device to implement the function of memory 306, or two or more types of storage media can be configured to implement the function of memory 306, and this application does not limit this.

[0107] The memory 306 stores executable program codes, and the processor 304 executes the executable program codes to respectively implement the functions of the aforementioned reasoning module and receiving module, thereby implementing the scheduling method. That is, the memory 306 stores instructions for executing the scheduling method.

[0108] Alternatively, the memory 306 stores executable codes, and the processor 304 executes the executable codes to respectively implement the functions of the aforementioned scheduling device, thereby implementing the scheduling method. That is, the memory 306 stores instructions for executing the scheduling method.

[0109] The communication interface 308 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 300 and other devices or a communication network.

[0110] As a possible implementation, the computing device 300 may also include a chip system, the chip system includes a processor and a power supply circuit, the power supply circuit is used to power the processor, and the processor is used to execute the operation steps corresponding to the scheduling method. For the sake of brevity, it is not repeated here. Among them, the processor can be implemented by a GPU, or by a computing device or AI chip such as a DPU, NPU, XPU, SoC, offload card, accelerator card, etc.

[0111] As a possible implementation, the computing device 300 may include multiple types of processors 304, that is, the computing device 300 is a heterogeneous device, for example, the computing device 300 includes a CPU and a GPU, and the operation steps corresponding to the scheduling method may be performed by at least one of the processors 304. For the sake of brevity, it will not be described in detail here.

[0112] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0113] like Figure 6 As shown, the computing device cluster includes at least one computing device 300. The memory 306 in one or more computing devices 300 in the computing device cluster may store the same instructions for executing the scheduling method.

[0114] In some possible implementations, the memory 306 of one or more computing devices 300 in the computing device cluster may also store partial instructions for executing the scheduling method. In other words, the combination of one or more computing devices 300 may jointly execute instructions for executing the scheduling method.

[0115] It should be noted that the memory 306 in different computing devices 300 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the scheduling device. That is, the instructions stored in the memory 306 in different computing devices 300 can implement the functions of one or more modules in the reasoning module and the receiving module.

[0116] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 7 A possible implementation is shown. Figure 7 As shown, two computing devices 300A and 300B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 306 in the computing device 300A stores instructions for executing the functions of the reasoning module. At the same time, the memory 306 in the computing device 300B stores instructions for executing the functions of the receiving module.

[0117] Figure 7The connection mode between the computing device clusters shown may be based on the consideration that the scheduling method provided in the present application requires, for example, a large amount of data storage and a large amount of calculation, and therefore it is considered that the functions implemented by the receiving module are handed over to the computing device 300B for execution.

[0118] It should be understood that Figure 7 The functions of the computing device 300A shown in FIG. 300A may also be completed by multiple computing devices 300. Similarly, the functions of the computing device 300B may also be completed by multiple computing devices 300.

[0119] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 5 and Figure 6 The connection mode of the computing device cluster is different in that the memory 306 in one or more computing devices 300 in the computing device cluster may store the same instructions for executing the scheduling method.

[0120] In some possible implementations, the memory 306 of one or more computing devices 300 in the computing device cluster may also store partial instructions for executing the scheduling method. In other words, the combination of one or more computing devices 300 may jointly execute instructions for executing the scheduling method.

[0121] It should be noted that the memory 306 in different computing devices 300 in the computing device cluster can store different instructions for executing part of the functions of the scheduling system. That is, the instructions stored in the memory 306 in different computing devices 300 can implement the functions of one or more devices in the scheduling device.

[0122] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a scheduling method or a scheduling method.

[0123] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the scheduling method, or instruct the computing device to execute the scheduling method.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A reasoning method, characterized in that: include: The gateway selects a first computing instance from multiple computing instances based on the first reasoning request, and sends the first reasoning request to the first computing instance; The first computing instance performs reasoning according to the first reasoning request to obtain a first reasoning result and an intermediate parameter, wherein the intermediate parameter is a parameter generated by the first computing instance in a process of calculating the first reasoning result; The first computing instance stores the first reasoning request, the first reasoning result, and the intermediate parameter in a memory pool; The first computing instance returns the first inference request to the gateway; The gateway selects a second computing instance from the multiple computing instances based on the first inference request, and sends the first inference request to the second computing instance; The second computing instance reads the first inference result and the intermediate parameter storage from the memory pool based on the first inference request, and continues to infer based on the first inference result and the intermediate parameters to obtain a second inference result, wherein the number of tags included in the second inference result is greater than the number of tags included in the first inference result, and the second inference result contains the first inference result.

2. The method according to claim 1, characterized in that: The gateway selects a first computing instance from multiple computing instances based on the first reasoning request, including: Calculating a score of each computing instance in the plurality of computing instances under the first reasoning request; A first computing instance is selected from the plurality of computing instances according to a score of each computing instance of the plurality of computing instances, wherein the score of the first computing instance is higher than scores of some or all of the other computing instances in the plurality of computing instances.

3. The method according to claim 2, characterized in that Calculating a score of each computing instance in the plurality of computing instances under the first reasoning request includes: A first score of the first computing instance under the first inference request is calculated according to the remaining video memory capacity ratio of the first computing instance, the number of inference requests queued for the first computing instance, and the length of the first inference request.

4. The method according to claim 3, characterized in that The remaining video memory capacity ratio of the first computing instance is the ratio of the remaining video memory capacity of the first computing instance to the available video memory capacity of the first computing instance, wherein the remaining video memory capacity of the first computing instance is the predicted remaining video memory capacity, or the actually collected remaining video memory capacity.

5. The method according to any one of claims 1 to 4, characterized in that: The intermediate parameters include a key matrix and a value matrix, where the key matrix and the value matrix are generated by performing different linear transformations on the first inference request respectively; The second computing instance continues to reason based on the first reasoning result and the intermediate parameter to obtain a second reasoning result, including: the second computing instance continues to reason based on the first reasoning result, the key matrix and the value matrix to obtain a second reasoning result.

6. A method of reasoning, characterized in that include: The second computing instance receives the first inference request sent by the gateway, and reads the first inference result and the intermediate parameter corresponding to the first inference request from the memory pool according to the first inference request, wherein the first inference result and the intermediate parameter are stored in the memory pool by the first computing instance; The second computing instance continues to reason based on the first reasoning result and the intermediate parameters to obtain a second reasoning result, wherein the number of tags included in the second reasoning result is greater than the number of tags included in the first reasoning result, and the second reasoning result contains the first reasoning result.

7. The method according to claim 6, characterized in that The first inference result and the intermediate parameter are calculated by the first computing instance based on the first inference request.

8. The method according to claim 6 or 7, characterized in that: The second computing instance is selected by the gateway from multiple computing instances according to the first inference request.

9. The method according to claim 6 or 7, characterized in that: The intermediate parameters include a key matrix and a value matrix, where the key matrix and the value matrix are generated by performing different linear transformations on the first inference request respectively; The second computing instance continues to reason based on the first reasoning result and the intermediate parameter to obtain a second reasoning result, including: the second computing instance continues to reason based on the first reasoning result, the key matrix and the value matrix to obtain a second reasoning result.

10. An inference device, characterized in that: The device comprises: A receiving module, used to receive a first inference request sent by a gateway, and read a first inference result and an intermediate parameter corresponding to the first inference request from a memory pool according to the first inference request, wherein the first inference result and the intermediate parameter are stored in the memory pool by a first computing instance; An inference module is used to continue inference based on the first inference result and the intermediate parameter to obtain a second inference result, wherein the number of tags included in the second inference result is greater than the number of tags included in the first inference result, and the second inference result includes the first inference result.

11. A computing device, characterized in that: The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory so that the computing device performs the operation steps of the method according to any one of claims 6 to 9.

12. A computing device cluster, characterized in that: comprising a plurality of computing devices, each computing device comprising a processor and a memory; The processor of at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method according to any one of claims 6 to 9.

13. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the operation steps of the method according to any one of claims 6 to 9.

14. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method according to any one of claims 6 to 9.

Images