Task processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a task processing method and device, a storage medium and a program product, and relates to the field of computers.The method comprises the steps that under the condition that a first task is added to a first queue, the first task is sent to a first node matched with the first queue, the first node stores a first key value resource matched with the first task to a first storage space, and processes the first task based on the first key value resource; under the condition that the execution duration of the first task reaches a first preset duration, adding a second task matched with the first task to a first queue; and when it is determined that the execution duration of the first task reaches a second predetermined duration, the second task is sent to the first node, and the first node stores a second key value resource matched with the second task in the first storage space and processes the second task based on the second key value resource and a target resource in the first key value resource. By adopting the scheme, the technical problem of waste of computing resources is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and specifically to a task processing method and device, a storage medium, and a program product. Background Art

[0002] Retrieval-Augmented Generation (RAG) systems aim to improve the accuracy and relevance of content generated by artificial intelligence (AI) by integrating information retrieval and language generation. They are particularly applicable to areas such as conversational AI, information retrieval, and automatic content generation. Existing technologies for document evaluation tasks in RAG systems, such as the retrieval evaluator introduced in Corrective Retrieval-Augmented Generation (CRAG), perform a Cartesian combination of scoring on retrieved documents and user questions to optimize recall results. While this process improves the accuracy of RAG systems, under a simple concurrency mechanism, the scoring tasks for documents with the same or similar questions are indiscriminately assigned to different Graphics Processing Unit (GPU) nodes. Each GPU node is required to independently maintain and update the key-value cache (kvcache) corresponding to its processing tasks, resulting in repeated execution of the same computation on multiple nodes, which wastes computing resources.

[0003] Therefore, there is a technical problem of wasting computing resources in the prior art. Summary of the Invention

[0004] The embodiments of the present application provide a task processing method and device, a storage medium, and a program product to at least solve the technical problem of wasting computing resources.

[0005] According to one embodiment of the present application, a task processing method is provided, comprising: when a first task is added to a first queue, sending the first task to a first node matching the first queue, so that the first node stores a first key-value resource matching the first task in a first storage space, and processing the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object; when the execution duration of the first task reaches a first predetermined duration, adding a second task matching the first task to the first queue, wherein the second task is used to evaluate the correlation between the second query statement and the second object, and both the first query statement and the second query statement include a target shared prefix; when it is determined that the execution duration of the first task reaches a second predetermined duration, sending the second task to the first node, so that the first node stores a second key-value resource matching the second task in the first storage space, and processing the second task based on the second key-value resource and a target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching a reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

[0006] According to another embodiment of the present application, a task processing device is provided, comprising: a first sending unit, for sending the first task to a first node matching the first queue when the first task is added to the first queue, so that the first node stores a first key-value resource matching the first task in a first storage space, and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object; an adding unit, for adding a second task matching the first task to the first queue when the execution time of the first task reaches a first predetermined time, wherein the second task is used to The correlation between the second query statement and the second object is evaluated, and both the first query statement and the second query statement include a target shared prefix; the second sending unit is used to send the second task to the first node when it is determined that the execution time of the first task reaches the second predetermined time, so that the first node stores the second key-value resource matching the second task in the first storage space, and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching the reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

[0007] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of the method described in each embodiment of the present application when executed by a processor.

[0010] In an embodiment of the present application, when a first task is added to a first queue, the first task is sent to a first node that matches the first queue so that the first node stores a first key-value resource that matches the first task in a first storage space and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object. Then, when the execution time of the first task reaches a first predetermined time, a second task that matches the first task is added to the first queue, wherein the second task is used to evaluate the correlation between the second query statement and the second object, and both the first query statement and the second query statement include a target shared prefix. Then, when it is determined that the execution time of the first task reaches a second predetermined time, the second task is sent to the first node so that the first node stores a second key-value resource that matches the second task in the first storage space and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource that matches a reference element in the second query statement and a key-value resource that matches the second object, and the target resource is a key-value resource that matches the target shared prefix. That is to say, by adopting the embodiment of the present application, on the one hand, by identifying the target shared prefix contained in the first query statement and the second query statement, multiple tasks can share the key-value resources on the GPU node during execution. This sharing mechanism reduces the necessity of each GPU node to maintain an independent kvcache, thereby significantly reducing the video memory occupancy and computational complexity, avoiding repeated execution of the same calculation, and thus saving computing resources. On the other hand, a dynamic scheduling strategy based on execution time is introduced. When the execution time of the first task reaches the first predetermined time, the second task will be pre-scheduled to the first queue. This pre-scheduling mechanism ensures that the GPU node can prepare to process the next task, avoiding delays and waste of resources between tasks. In summary, by adopting the embodiment of the present application, the technical effect of saving computing resources is achieved, and the technical problem of wasting computing resources in the task processing method provided in the related art is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flowchart of a method for processing a task according to an embodiment of the present application;

[0012] Figure 2 is a flowchart of a method for processing a task according to an embodiment of the present application;

[0013] Figure 3 is a schematic diagram of processing a task according to an embodiment of the present application;

[0014] Figure 4 This is a structural block diagram of a task processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] As an optional solution, the above task processing method has the following specific steps: Figure 1 Shown include:

[0018] S102, when adding the first task to the first queue, sending the first task to the first node matching the first queue, so that the first node stores the first key-value resource matching the first task in the first storage space, and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object.

[0019] It should be noted that the processing method of the above task can be but is not limited to being used in the document relevance evaluation scenario in the large-scale Retrieval-Augmented Generation (RAG) system. Specifically, it can be applied in the RAG system. When a user makes a query request (for example, "What are the latest developments in AI technology?"), the system retrieves multiple document fragments from the database and evaluates the relevance between these document fragments and the user query.

[0020] Furthermore, the relevance evaluation between the first task, the first query statement, and the first object can be, but is not limited to, used to indicate a basic function in the RAG system, which is intended to evaluate the relevance between a piece of text or document (the first object) and a question posed by a user (the first query statement). For example, a user may ask, "What are the latest developments in AI technology?" The system will retrieve multiple documents from the database and evaluate their relevance to the question.

[0021] Optionally, the first queue can be used, but is not limited to, to temporarily store tasks awaiting processing. When a first task is ready for processing, it is sent to the first node that matches the queue, i.e., the GPU node responsible for executing the task. The first node then uses the key-value resources in its graphics memory to process the task in the queue.

[0022] Furthermore, the first key-value resource can be, but is not limited to, a data structure used to store important information during the GPU inference calculation process, namely a key-value pair cache (KV Cache). This information is reused during task generation and stored in the GPU's video memory. Specifically, the first node can be, but is not limited to, utilizing large language models (LLMs) to process query and document scoring tasks. Because LLMs utilize a self-attention mechanism, when they calculate a portion of an input sequence, they generate a set of intermediate results that can be used for subsequent sequence calculations. The KV Cache is used to store these intermediate calculation results, including the key and value vectors in the attention mechanism. These vectors need to be calculated for each new word generated during the decoding process, and the cache mechanism allows for the reuse of previously calculated results, avoiding repeated calculations for the same or similar inputs.

[0023] Optionally, the first storage space may be, but is not limited to, a video memory used to indicate a GPU used by the first node, and is used to store key-value resources related to tasks processed by the first node to facilitate task processing.

[0024] S104. When the execution time of the first task reaches a first predetermined time, a second task matching the first task is added to the first queue, wherein the second task is used to evaluate the correlation between the second query statement and the second object, and both the first query statement and the second query statement include a target shared prefix.

[0025] Optionally, the target shared prefix can be used, but is not limited to, to indicate keywords or phrases that appear in both the first and second query statements. These keywords or phrases often constitute the context or subject of the query. By maximizing the utilization of the target shared prefix, the efficiency of the GPU inference framework can be improved.

[0026] It should be noted that the first and second predetermined durations are used to control the scheduling and execution of tasks. The first predetermined duration refers to the time when the second task is added to the first queue, typically when the first task has begun execution but has not yet completed. The second predetermined duration determines when the second task is actually sent to the first node for processing.

[0027] S106. When it is determined that the execution time of the first task reaches the second predetermined time, the second task is sent to the first node so that the first node stores the second key-value resource matching the second task in the first storage space, and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching the reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

[0028] It should be noted that the second key-value resource matched with the second task may be, but is not limited to, used to indicate KV Cache related to other elements in the second query statement except the target shared prefix and KV Cache related to the second object, including the key (Key) and value (Value) vectors of these elements. The target resource may be, but is not limited to, used to indicate KV Cache related to the target shared prefix, including the key (Key) and value (Value) vectors of the target shared prefix.

[0029] As an optional example, the above steps can be explained with reference to, but not limited to, the following examples:

[0030] Consider a RAG system that evaluates the relevance between user queries and documents retrieved from a database. The user submits two queries: "What are the latest developments in AI technology?" and "What are the recent innovations in AI technology?" Both queries contain the shared prefix "latest AI technology" and can be considered homogeneous reasoning tasks.

[0031] S1, processing of the first task: First task: The task is to evaluate the correlation between "What are the latest developments in AI technology?" and a certain document. First queue: A queue is used to temporarily store tasks waiting to be processed, which is initially empty. First node: A GPU node, used to execute tasks in the queue. First, add the first task to the first queue. Then, send the first task to the first node. The first node stores the first key-value resource (containing information related to "the latest AI technology") that matches the first task to the first storage space (i.e., the GPU's video memory). The first node processes the first task based on these key-value resources and evaluates the correlation between the first query statement and the first object.

[0032] S2. Pre-scheduling the second task: The second task is to evaluate the relevance between the query "What are the latest innovations in AI technology?" and another document. When the execution time of the first task reaches a first predetermined duration (e.g., 50% of the expected execution time), the system recognizes that the targets of the first and second queries share the prefix "latest AI technology" and pre-adds the second task to the first queue. At this point, the second task has not yet begun processing, but is ready to wait for the completion of the first task.

[0033] S3, formal processing of the second task: When the execution time of the first task reaches the second predetermined time (for example, 100% of the expected execution time means that the task has been completed), the second task will be sent to the first node. The first node stores the second key-value resource (containing information related to "what innovations have been made recently") that matches the second task in the first storage space, and integrates it with the target resource (shared prefix) in the first key-value resource for processing the second task. The first node processes the second task based on the second key-value resource and the existing target resource, that is, evaluates the correlation between the second query statement and the second object.

[0034] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0035] In an embodiment of the present application, when a first task is added to a first queue, the first task is sent to a first node that matches the first queue so that the first node stores a first key-value resource that matches the first task in a first storage space and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object. Then, when the execution time of the first task reaches a first predetermined time, a second task that matches the first task is added to the first queue, wherein the second task is used to evaluate the correlation between the second query statement and the second object, and both the first query statement and the second query statement include a target shared prefix. Then, when it is determined that the execution time of the first task reaches a second predetermined time, the second task is sent to the first node so that the first node stores a second key-value resource that matches the second task in the first storage space and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource that matches a reference element in the second query statement and a key-value resource that matches the second object, and the target resource is a key-value resource that matches the target shared prefix. That is to say, by adopting the embodiment of the present application, on the one hand, by identifying the target shared prefix contained in the first query statement and the second query statement, multiple tasks can share the key-value resources on the GPU node during execution. This sharing mechanism reduces the necessity of each GPU node to maintain an independent kvcache, thereby significantly reducing the video memory occupancy and computational complexity, avoiding repeated execution of the same calculation, and thus saving computing resources. On the other hand, a dynamic scheduling strategy based on execution time is introduced. When the execution time of the first task reaches the first predetermined time, the second task will be pre-scheduled to the first queue. This pre-scheduling mechanism ensures that the GPU node can prepare to process the next task, avoiding delays and waste of resources between tasks. In summary, by adopting the embodiment of the present application, the technical effect of saving computing resources is achieved, and the technical problem of wasting computing resources in the task processing method provided in the related art is solved.

[0036] As an optional solution, when the first task is added to the first queue, before the first task is sent to the first node matching the first queue, the method further includes:

[0037] S1, obtaining N reference tasks sent by a target device, wherein the N reference tasks are used to evaluate the relevance between N query statements and N objects, where N is greater than 1 and is an integer.

[0038] It should be noted that the target device described above can be, but is not limited to, a device for instructing task generation, typically a client device or a component in the RAG system, responsible for generating tasks for evaluating the relevance between a query statement and an object. The N reference tasks described above are a series of tasks, each of which includes a query statement and an object (e.g., a document fragment) for evaluating the relevance between the two.

[0039] S2. Determine each reference task in the N reference tasks as the current reference task and perform the following steps until N target tasks are obtained, wherein the N target tasks include the first task and the second task: move the target element in the current query statement corresponding to the current reference task from the first position in the current query statement to the second position to obtain the current target task, wherein the number of target elements in the N query statements is less than the predetermined number, the N target tasks include the current target task, and the first position is located before the second position.

[0040] Optionally, the current reference task may be, but is not limited to, used to indicate the task currently being analyzed and processed when processing the reference task. Furthermore, the target element may be, but is not limited to, used to indicate a keyword or phrase related to the evaluation task, which typically does not appear repeatedly in multiple query statements. The first position and second position refer to the original position and target position in the query statement, respectively. By moving the position of the target element, the input structure can be optimized to share more prefix information.

[0041] S3, obtain the i-th query statement among the N query statements corresponding to the N target tasks, and obtain the j-th query statement among the N query statements corresponding to the N target tasks, where i is less than N, i is a positive integer, j is a positive integer, j is greater than i, and j is less than or equal to N.

[0042] S4: Determine a shared prefix included in the i-th query statement and the j-th query statement, wherein the i-th query statement includes each element in the shared prefix, and the j-th query statement includes each element in the shared prefix.

[0043] S5, determining the number of elements included in the shared prefix, and storing the number of elements in an element number record table.

[0044] It should be noted that the shared prefix can be used, but not limited to, to indicate common components across multiple query statements. This can be utilized by downstream reasoning systems to reduce computation and memory usage. The element count table can be used, but not limited to, to indicate a data structure that records the number of elements in each shared prefix to quantify the degree of prefix sharing and facilitate subsequent task scheduling optimization.

[0045] Optionally, the above determination of the shared prefix included in the i-th query statement and the j-th query statement may include, but is not limited to, separately determining the shared prefix included in the i-th query statement and each query statement located after the i-th query statement in the N query statements. For example, assuming that N is 4 and i is 2, then in the above steps, it is necessary to separately determine the shared prefixes corresponding to the second query statement, the third query statement, and the fourth query statement. Then, the number of elements corresponding to each of the two shared prefixes is determined, and the number of elements corresponding to each of the above two shared prefixes is recorded in the element number record table.

[0046] As an optional example, the above steps can be explained with reference to, but not limited to, the following examples:

[0047] Assume the target device is a RAG system that needs to evaluate whether 100 documents (objects) are relevant to 10 user questions (queries). The goal is to optimize this process to reduce the memory usage of the kvcache (key-value cache) during GPU inference.

[0048] Step 1: Get N reference tasks:

[0049] First, we obtain 10 reference tasks, each of which consists of a user question (query) and a set of documents (objects) for evaluating relevance. For example, a reference task might be “evaluate the relevance of document A to question Q1.”

[0050] Step 2: Determine N reference tasks as current reference tasks and reconstruct:

[0051] Next, the system iterates through the 10 reference tasks. For each current reference task, it identifies the target element, which is a keyword or phrase related to the assessment task. For example, if the question is about "AI technology," "AI technology" will be extracted as the target element.

[0052] Then, the query structure is modified, moving the target element ("AI technology") from the first position in the query to the second position to form the current target task. For example, the original query is "Please tell me the latest progress in AI technology", which is adjusted to "Please tell me the latest progress in AI technology", so that "latest progress" becomes a shared prefix.

[0053] Step 3: Determine the shared prefix and record it:

[0054] After reconstruction is complete, N target tasks are obtained. For query statements i and j, determine the shared prefix between them, such as "Please tell me about the latest progress." Count the number of elements in the shared prefix and store this number in the element count table.

[0055] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0056] Using the embodiments of the present application, on the one hand, by determining and optimizing shared prefixes, downstream inference nodes can reuse prefix information stored in the kvcache (key-value cache) when processing tasks with similar query statements, avoiding unnecessary video memory allocation and data loading. This means that video memory usage will be significantly reduced, especially when processing a large number of similar tasks, where the video memory savings are particularly significant. On the other hand, the optimized request structure reduces the amount of computation required by the GPU when processing tasks because it shares more prefix information. This is because the prefix information only needs to be calculated once, and subsequent tasks can directly use these calculation results, thus saving the computing resources originally required to independently calculate the prefix information for each task and improving the computational efficiency of the GPU inference framework. On the other hand, by recording the number of elements in the shared prefix, the system can more intelligently schedule tasks and ensure balanced resource utilization. For example, when an inference node's kvcache resources are heavily used, the system can prioritize scheduling tasks with fewer new elements to that node, avoiding video memory overflow while ensuring that the resources of other nodes are fully utilized.

[0057] As an optional solution, after sending the first task to the first node matching the first queue so that the first node stores the first key-value resource matching the first task in the first storage space and processes the first task based on the first key-value resource, the method further includes:

[0058] S1. Obtain the number of N-1 elements of the shared prefix included between the first query statement and N-1 reference query statements from the element number record table, wherein the reference query statement is the other query statements except the first query statement among the N query statements corresponding to the N target tasks.

[0059] It should be noted that the aforementioned N-1 reference query statements can be, but are not limited to, used to indicate the remaining N-1 query statements in the N target tasks, excluding the first query statement currently being processed. These statements will be used to compare and evaluate the shared prefix with the first query statement to determine the best matching task.

[0060] Furthermore, the shared prefix can be used, but is not limited to, to indicate a sequence of keywords or phrases that appear in two or more query statements. These shared elements form a common prefix that can be utilized by the GPU reasoning framework to achieve resource sharing in the kvcache.

[0061] S2, sorting the N-1 number of elements, and determining the largest number of elements among the N-1 number of elements as the target number of elements.

[0062] Optionally, the target number of elements is the number of shared prefix elements contained in the statement that shares the most prefix elements with the first query statement among the N-1 reference query statements.

[0063] S3: Determine the target task corresponding to the reference query statement corresponding to the target number of elements as the second task matching the first task.

[0064] It should be noted that the above-mentioned second task is the task that shares the most prefix elements (i.e., the reference query statement corresponding to the number of target elements) with the first task among the N target tasks, and is selected as the next processing task after the completion of the first task to maximize resource sharing and scheduling efficiency.

[0065] As an optional example, the above steps can be explained with reference to, but not limited to, the following examples:

[0066] Assume that there are five query statements (N=5) below, which are used to retrieve and evaluate the relevance of documents:

[0067] Query statement 1: Retrieve research reports on the latest AI technology.

[0068] Query statement 2: Find the latest application cases of AI technology in the medical field.

[0069] Query statement 3: List the latest AI technology development trends.

[0070] Query statement 4: Get the patent list of the latest AI technology.

[0071] Query statement 5: Understand the latest applications of AI technology in the field of education.

[0072] S1, obtain N reference tasks and determine the shared prefix:

[0073] The shared prefix "latest Al technology" in the query statement is identified and used as the target shared prefix element.

[0074] S2, restructure the query to maximize the shared prefix:

[0075] Move the shared prefix element to the front of the query statement and obtain five reconstructed target tasks:

[0076] Current objective task 1: Latest Al technology\n\nSearch for relevant research reports.

[0077] Current Objective 2: Find the latest Al technology application cases in the medical field.

[0078] Current Objective 3: Latest Al Technology\n\nList development trends.

[0079] Current Objective 4: Latest Al Technology\n\nObtain patent list.

[0080] Current Objective 5: Understand the latest AI technology and its application in education.

[0081] S 3, build an element number record table and determine the target element number:

[0082] Assume that the number of elements is recorded in the table, and t1 to t5 represent the five target tasks after reconstruction. For example, the number of shared prefix elements between t1 and t2 is 5, which means that the two tasks share all the prefix elements.

[0083] S4, determining a second task that matches the first task:

[0084] Assume that the first task is t1. Obtain the number of shared prefix elements (N-1 elements) of the other four reference query statements related to t1 from the element number record table. After sorting these numbers, it is found that the largest number of elements is 5. The reference query statement corresponding to this number is the query statement corresponding to t2. Therefore, the target task t2 is determined as the second task matching the first task.

[0085] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0086] According to an embodiment of the present application, an element quantity record table is used to quantify the number of shared prefix elements between each query statement, which enables the system to clearly identify which query statement (reference query statement) shares the most prefix elements with the first query statement currently being processed. By determining the target task (second task) with the largest number of shared prefix elements and scheduling it to the same node as the first task for processing, the system can effectively utilize the resources in kvcache (key-value cache), reduce video memory usage, avoid unnecessary cache updates and data exchanges, and thus save computing resources.

[0087] As an optional solution, when the execution duration of the first task reaches the first predetermined duration, adding the second task matching the first task to the first queue includes:

[0088] When the execution duration of the first task reaches a first predetermined duration and the resource occupancy rate of the first storage space is less than a predetermined threshold, the second task is added to the first queue, wherein the resource occupancy rate is based on the amount of resources stored in the first storage space and the total amount of resources that can be stored in the first storage space.

[0089] It should be noted that the resource utilization ratio may be, but is not limited to, indicating the ratio of the amount of resources currently used by the first storage space (e.g., video memory) to the total amount of resources in the storage space. The upper predetermined threshold may be, but is not limited to, indicating an upper limit on resource utilization set by the system. When the resource utilization ratio of the first storage space is below this threshold, the system determines that there are sufficient resources to process new tasks.

[0090] As an optional example, the above steps can be explained with reference to, but not limited to, the following examples:

[0091] Suppose we are processing a document evaluation task based on CRAG (Corrective Retrieval-Augmented Generation), where the system needs to evaluate the relevance of multiple documents to user questions. During the task scheduling process, the above steps are implemented as follows:

[0092] S1, Initialization Phase: The system receives multiple task requests, each of which consists of a query statement and a document object to be evaluated. For example, consider two tasks: Task 1 (evaluating question Q1 and document D1) and Task 2 (evaluating question Q2 and document D2).

[0093] S2, Task Processing: Task1 is first added to the first queue and then sent to the first node for processing. During the processing, the system constructs the first key-value resources, which are stored in the kvcache, i.e. the first storage space.

[0094] S3, execution time monitoring: the system continuously monitors the execution time of the first task Task 1. When the execution time of Task 1 reaches a first predetermined time, the system starts preparing to schedule Task 2.

[0095] S4, resource occupancy evaluation: At the same time, the system checks the resource occupancy of the first storage space to determine whether there are sufficient resources to process Task 2. The resource occupancy is calculated based on the amount of resources stored in the first storage space and the total amount of resources that can be stored in the first storage space.

[0096] S5, Scheduling Decision: If Task 1's execution duration reaches a first predetermined duration and the resource occupancy rate of the first storage space is less than a predetermined threshold, the system deems the first node capable of processing Task 2, which shares a prefix with Task 1. For example, suppose Q1 is "Evaluate relevance in document D1" and Q2 is "Evaluate relevance in document D2." After optimization, the shared prefix of the two tasks is "Evaluate relevance in documents."

[0097] S6, Second Task Scheduling: The system adds Task2 to the first queue and sends it to the first node. Because Task2 and Task1 share a prefix, the first node can reuse the key-value resources generated when processing Task1 and only needs to generate new key-value pairs for the newly added elements, thus reducing resource consumption and inference latency.

[0098] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0099] By adopting the embodiment of the present application, the resource occupancy rate of the first storage space is controlled by setting a predetermined threshold, ensuring that even under high load conditions of concurrent task execution, resource occupancy can be maintained within a reasonable range, thereby avoiding system performance degradation due to excessive resource consumption. This dynamic control mechanism improves resource utilization while ensuring the stability and response speed of the system. When the execution time of the first task reaches the first predetermined time and the resource occupancy rate is lower than the predetermined threshold, the second task is automatically added to the first queue and sent to the matching first node for processing. Load balancing between tasks is ensured, avoiding idle or excessive use of resources on some nodes due to improper task scheduling, thereby improving the throughput and efficiency of the entire system.

[0100] As an optional solution, in the case of adding the first task to the first queue, after sending the first task to the first node matching the first queue, the method further includes:

[0101] When the resource occupancy rate of the first storage space is greater than or equal to a predetermined threshold, the second task is added to the second queue, so that the second node matching the second queue processes the second task.

[0102] It should be noted that the second queue can be, but is not limited to, used to indicate another task queue running in parallel with the first queue, for scheduling and managing pending tasks. The second node is another GPU inference node running in parallel with the first node, for processing tasks in the second queue.

[0103] As an optional example, the above steps can be explained with reference to, but not limited to, the following examples:

[0104] Assume the total size of the first storage space is 1GB and the preset resource utilization is 75%. That is, when video memory usage reaches 750MB, the system begins adjusting task scheduling. Task 1: Processing a query and document combination evaluation task, requiring 100MB of KVCache resources. Task 2: Processing another query and document combination evaluation task, requiring 150MB of KVCache resources.

[0105] When executing Task 1, the resource occupancy rate of the first storage space increases to 100 MB, which is lower than the threshold of 750 MB, so the task can be executed normally.

[0106] When Task 2 is ready to execute, if its resource requirements (150MB) are added, the resource utilization of the first storage space will reach 250MB. At this time, the system checks whether it has exceeded the threshold. Since 250MB is lower than 750MB, Task 2 can still be sent to the first node for processing through the first queue.

[0107] If the resource requirements of the subsequent task (task 3) make the resource occupancy rate of the first storage space close to or exceed 750MB, the system will schedule task 3 to the second queue and send it to the second node for processing to avoid resource overload of the first node, thereby ensuring balanced and efficient use of system resources.

[0108] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0109] In this embodiment of the present application, a predetermined threshold is set to monitor the resource occupancy rate of the first storage space. When the predetermined threshold is reached or exceeded, the system automatically reallocates subsequent tasks (such as the second task) to another queue (the second queue) and the corresponding node (the second node). This effectively avoids resource overload on a single node or storage space, ensures the proper allocation of KVCache and other key resources, thereby reducing resource contention and waiting time, and improving resource utilization efficiency.

[0110] As an optional solution, after the first node stores the first key-value resource matching the first task in the first storage space and processes the first task based on the first key-value resource, the method further includes:

[0111] When it is determined that the first node has completed the first task, the first key-value resource is removed from the first storage space.

[0112] As an optional example, the above steps can be explained with reference to, but not limited to, the following examples:

[0113] Task completion detection: The system requires a mechanism to detect the task execution status of downstream inference nodes (such as GPU nodes). When a task (the first task) is completed, the system can identify the task status change from in progress to completed through the task status monitoring function (such as task_status).

[0114] Key-value resource identification: Before a task begins execution, the system stores the key-value resources associated with the task (primary key-value resources) in a primary storage space (e.g., GPU memory). These resources include key-value pairs representing the task's input sequence, intermediate computation results, and output sequence. During storage, the system assigns these resources an identifier associated with the task (e.g., task_id or task_token) for subsequent management and tracking.

[0115] Resource removal logic: Once the system detects that the first task has completed, it triggers resource removal logic. This is typically implemented by a resource management component, such as a memory manager. Based on the task identifier (task_id or task_token), this component searches the first storage space for the key-value resources associated with the first task and removes (releases) them from the video memory. This process ensures that only resources for tasks in progress or about to be completed remain in the video memory, avoiding resource waste.

[0116] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0117] By adopting the embodiment of the present application, after the first node completes the first task, the corresponding first key-value resource is removed, which can immediately release the cache resources that are no longer needed in the GPU video memory, avoid excessive cache occupation, and make room for the execution of subsequent tasks, thereby ensuring the stability and responsiveness of the GPU inference system in high-concurrency scenarios.

[0118] Optionally, the processing method of the above task may include, but is not limited to:

[0119] When the first node processes a task, the cache time of the first key-value resource is dynamically adjusted to reduce resource usage; when the first task is completed, the system automatically removes the first key-value resource from the first storage space, but retains key information related to the target shared prefix for subsequent task processing.

[0120] It should be noted that, using the above steps, when the first node processes the first task (Task1), the system will dynamically adjust the cache time of the first key-value resource according to the task execution status and resource occupancy. For example, if the system detects that the current GPU video memory occupancy is high, but Task1 is about to be completed, the system will shorten the cache time of the key-value resource that matches Task1 to avoid occupying too many resources, while ensuring that these resources are available before Task1 is completed. Once Task1 is completed, the system will automatically remove the first key-value resource from the first storage space, but will retain key information related to the target shared prefix (such as "the latest Al technology"). This means that the system will analyze the first key-value resource and only retain the key-value part that matches the shared prefix, while removing other irrelevant or redundant key-value information. This reduces the occupancy of video memory while ensuring that the shared prefix information can be reused by subsequent tasks (such as Task2), thereby improving the efficiency of kvcache.

[0121] Alternatively, as an optional example, it is possible but not limited to: Figure 2 The following example illustrates how to handle the above tasks:

[0122] Execute step S202, and when adding the first task to the first queue, send the first task to the first node matching the first queue, so that the first node stores the first key-value resource matching the first task in the first storage space, and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object.

[0123] Then, step S204 is executed to determine whether the resource occupancy rate of the first storage space is less than a predetermined threshold value when the execution time of the first task reaches the first predetermined time.

[0124] Then, if the resource occupancy rate of the first storage space is less than a predetermined threshold, steps S206-1 to S206-1-1 are executed:

[0125] Step S206 - 1 : adding the second task to the first queue, wherein the resource occupancy rate is obtained based on the amount of resources stored in the first storage space and the total amount of resources that can be stored in the first storage space.

[0126] Step S206-1-1, when it is determined that the execution time of the first task reaches the second predetermined time, the second task is sent to the first node, so that the first node stores the second key-value resource matching the second task in the first storage space, and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching the reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

[0127] When the resource occupancy rate of the first storage space is greater than or equal to the predetermined threshold, step S206 - 2 is executed to add the second task to the second queue, so that the second node matching the second queue processes the second task.

[0128] Then, step S208 is executed to remove the first key-value resource from the first storage space when it is determined that the first node has completed the first task.

[0129] Optionally, as an optional example, the task scheduling method of the above task processing method can be but not limited to referring to Figure 3 Specifically, Figure 3 The code snippet shown describes a parallel task scheduling algorithm for large language models (LLMs) to optimize the resource utilization efficiency of GPU inference services based on RAG (Retrieval-Augmented Generation) systems. The following is a detailed explanation of the code:

[0130] S1 receives the following three parameters:

[0131] inputs: A list of strings containing all the questions that need to be sent to the LLMs for reasoning.

[0132] vllms: A list of strings containing the addresses of downstream inference services, that is, the service node addresses where LLMs are located.

[0133] gpu_tokens: A list of integers representing the remaining available token capacity of each inference service node (vllm). The token capacity here can be understood as the amount of text information that the GPU inference node can process, which is usually related to the video memory usage.

[0134] S2, build prefix relationship table:

[0135] The length of the input list is obtained and a two-dimensional common prefix table, comm_prefix_tbl, is initialized to store the maximum common prefix lengths between all pairs of questions in the input list. The input list is then sorted, and all pairs of questions are iterated over, calculating and recording the maximum common prefix length for each pair. This provides important information for subsequent dynamic task scheduling.

[0136] S3, initialize task queue and scheduling:

[0137] A mask array is initialized to mark which tasks have been scheduled. Simultaneously, the same number of WorkerQueues are created based on the length of the vllms, each corresponding to a different inference service node (vllms). In the following loop, the algorithm randomly selects an unscheduled task, pushes it into a queue, and updates the mask array, gpu_tokens list, and common prefix table comm_prefix_tbl to prevent the same task from being scheduled multiple times.

[0138] S4, dynamic task scheduling:

[0139] Before all tasks are scheduled, the algorithm continues to execute the following logic:

[0140] The gather_task_status(queues) function is used to collect the status of tasks in all queues, including completed and ongoing tasks. The algorithm then iterates over the tasks in each queue and updates the gpu_tokens list to reflect the real-time occupation and release of video memory. Next, the algorithm iterates over the task queues, checks the last task (back_task) in each queue, and attempts to find the next task (next_task_idx) with the largest common prefix. If the remaining resources (gpu_tokens) of the target node meet the scheduling conditions (the remaining tokens meet the 80% water mark), the next task found is scheduled to the corresponding queue, and the mask and common prefix tables are updated.

[0141] For example, the above steps can be explained based on, but not limited to, the following examples:

[0142] Assume we have the following parameters:

[0143] inputs: contains 5 question inputs (question 1 to question 5).

[0144] vllms: Contains the addresses of three inference service nodes (node ​​1 to node 3).

[0145] gpu_tokens: The remaining token capacity of each node is 16,000, 12,000, and 14,000 respectively.

[0146] S1, initialization phase: Calculate and fill the comm_prefix_tbl, assuming that the maximum common prefix length between problem 1 and problem 2 is 8, the maximum common prefix length between problem 1 and problem 3 is 5, the maximum common prefix length between problem 4 and problem 5 is 10, and so on. Create three task queues, representing the queues of pending tasks of the three inference service nodes.

[0147] S2, randomly scheduled tasks: Assume that the algorithm first selects problem 1 and pushes it into the queue of node 1. Next, problem 3 is randomly selected and scheduled into the queue of node 2, and problem 5 is scheduled into node 3. Update mask, gpu_tokens, and comm_prefix_tbl to reflect the status after scheduling.

[0148] S3, dynamic scheduling phase: During each loop, the algorithm checks task status, updates g pu_tokens, and releases resources for completed tasks. When Node 1's resources meet the scheduling criteria, the algorithm attempts to find the task with the largest common prefix with Problem 1. For example, Problem 2 will be scheduled to Node 1's queue. Similarly, when resources on Nodes 2 and 3 allow, the algorithm attempts to schedule the next task with the largest common prefix with Problems 3 and 5. For example, Problem 4 might be scheduled to Node 3 because it shares the largest common prefix with Problem 5.

[0149] It should be noted that the above example is an optional example provided to facilitate explanation of the method for processing the above task, and there is no limitation on the specific implementation of the method for processing the above task.

[0150] This embodiment also provides a task processing device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0151] Figure 4 is a structural block diagram of a task processing device according to an embodiment of the present application, such as Figure 4 As shown, the device includes:

[0152] a first sending unit 402 configured to, when adding the first task to the first queue, send the first task to a first node that matches the first queue, so that the first node stores a first key-value resource that matches the first task in a first storage space and processes the first task based on the first key-value resource, wherein the first task is configured to evaluate a correlation between the first query statement and the first object;

[0153] an adding unit 404 configured to add a second task matching the first task to the first queue if the execution duration of the first task reaches a first predetermined duration, wherein the second task is configured to evaluate a correlation between a second query statement and a second object, and both the first query statement and the second query statement include a target shared prefix;

[0154] The second sending unit 406 is used to send the second task to the first node when it is determined that the execution time of the first task reaches the second predetermined time, so that the first node stores the second key-value resource matching the second task in the first storage space, and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching the reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

[0155] As an optional solution, the above-mentioned device also includes: a first acquisition unit, used to acquire N reference tasks sent by the target device, wherein the N reference tasks are used to evaluate the correlation between N query statements and N objects, N is greater than 1 and N is an integer; a task reconstruction unit, used to determine each reference task in the N reference tasks as a current reference task and perform the following steps until N target tasks are obtained, wherein the N target tasks include a first task and a second task: moving the target element in the current query statement corresponding to the current reference task from the first position in the current query statement to the second position to obtain the current target task, wherein the number of target elements in the N query statements is less than a predetermined number, and the N target tasks The task includes the current target task, and the first position is located before the second position; the second acquisition unit is used to obtain the i-th query statement among the N query statements corresponding to the N target tasks, and obtain the j-th query statement among the N query statements corresponding to the N target tasks, wherein i is less than N, and i is a positive integer, j is a positive integer, j is greater than i and j is less than or equal to N; the first determination unit is used to determine the shared prefix included in the i-th query statement and the j-th query statement, wherein the i-th query statement includes each element in the shared prefix, and the j-th query statement includes each element in the shared prefix; the second determination unit is used to determine the number of elements included in the shared prefix, and store the number of elements in the element number record table.

[0156] As an optional solution, the above-mentioned device also includes: a third acquisition unit, used to obtain the N-1 number of elements of the shared prefix included between the first query statement and N-1 reference query statements from the element number record table, wherein the reference query statement is the other query statements other than the first query statement among the N query statements corresponding to the N target tasks; a third determination unit, used to sort the N-1 number of elements and determine the largest number of elements among the N-1 number of elements as the target number of elements; a fourth determination unit, used to determine the target task corresponding to the reference query statement corresponding to the target number of elements as the second task matching the first task.

[0157] As an optional solution, the above-mentioned adding unit includes: an adding module, used to add the second task to the first queue when the execution duration of the first task reaches a first predetermined duration and the resource occupancy rate of the first storage space is less than a predetermined threshold, wherein the resource occupancy rate is based on the amount of resources stored in the first storage space and the total amount of resources that can be stored in the first storage space.

[0158] As an optional solution, the above-mentioned device also includes: a first adding unit, which is used to add the second task to the second queue when the resource occupancy rate of the first storage space is greater than or equal to a predetermined threshold, so that the second node matching the second queue processes the second task.

[0159] As an optional solution, the above-mentioned device further includes: a removing unit, configured to remove the first key-value resource from the first storage space when it is determined that the first node has completed the first task.

[0160] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0162] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0163] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0164] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0165] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0166] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0167] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0168] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0169] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0170] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A task processing method, characterized in that: include: In the case of adding a first task to a first queue, sending the first task to a first node matching the first queue, so that the first node stores a first key-value resource matching the first task in a first storage space, and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object; When the execution time of the first task reaches a first predetermined time, adding a second task matching the first task to the first queue, wherein the second task is used to evaluate the correlation between a second query statement and a second object, and both the first query statement and the second query statement include a target shared prefix; When it is determined that the execution time of the first task reaches a second predetermined time, the second task is sent to the first node so that the first node stores a second key-value resource matching the second task in the first storage space, and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching the reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

2. The task processing method according to claim 1, characterized in that: In the case where the first task is added to the first queue, before the first task is sent to the first node matching the first queue, the method further includes: Obtaining N reference tasks sent by the target device, wherein the N reference tasks are used to evaluate the correlation between the N query statements and the N objects, where N is greater than 1 and is an integer; Determine each reference task in the N reference tasks as a current reference task and perform the following steps until N target tasks are obtained, wherein the N target tasks include the first task and the second task: move a target element in a current query statement corresponding to the current reference task from a first position in the current query statement to a second position to obtain a current target task, wherein the number of the target elements in the N query statements is less than a predetermined number, the N target tasks include the current target task, and the first position is located before the second position; Obtaining an i-th query statement among the N query statements corresponding to the N target tasks, and obtaining a j-th query statement among the N query statements corresponding to the N target tasks, wherein i is less than N, and i is a positive integer, j is a positive integer, j is greater than i, and j is less than or equal to N; Determining a shared prefix included in the i-th query statement and the j-th query statement, wherein the i-th query statement includes each element in the shared prefix, and the j-th query statement includes each element in the shared prefix; The number of elements included in the shared prefix is ​​determined, and the number of elements is stored in an element number record table.

3. The task processing method according to claim 2, characterized in that: After sending the first task to a first node matching the first queue so that the first node stores a first key-value resource matching the first task in a first storage space and processes the first task based on the first key-value resource, the method further includes: Acquire, from the element quantity record table, the number of N-1 elements of the shared prefix included between the first query statement and N-1 reference query statements, wherein the reference query statements are other query statements except the first query statement among the N query statements corresponding to the N target tasks; Sorting the N-1 numbers of elements, and determining the largest number of elements among the N-1 numbers of elements as the target number of elements; The target task corresponding to the reference query statement corresponding to the target number of elements is determined as the second task matching the first task.

4. The task processing method according to claim 1, characterized in that: When the execution duration of the first task reaches a first predetermined duration, adding a second task matching the first task to the first queue includes: When the execution duration of the first task reaches a first predetermined duration and the resource occupancy rate of the first storage space is less than a predetermined threshold, the second task is added to the first queue, wherein the resource occupancy rate is based on the amount of resources stored in the first storage space and the total amount of resources that can be stored in the first storage space.

5. The task processing method according to claim 4, characterized in that: In the case of adding the first task to the first queue, after sending the first task to the first node matching the first queue, the method further includes: When the resource occupancy rate of the first storage space is greater than or equal to the predetermined threshold, the second task is added to a second queue, so that a second node matching the second queue processes the second task.

6. A method for processing a task according to any one of claims 1 to 5, characterized in that: After the first node stores the first key-value resource matching the first task in the first storage space and processes the first task based on the first key-value resource, the method further includes: When it is determined that the first node has completed the first task, the first key-value resource is removed from the first storage space.

7. A task processing device, characterized in that: include: a first sending unit, configured to send the first task to a first node matching the first queue when adding the first task to the first queue, so that the first node stores a first key-value resource matching the first task in a first storage space, and processes the first task based on the first key-value resource, wherein the first task is used to evaluate the correlation between the first query statement and the first object; an adding unit, configured to add a second task matching the first task to the first queue when the execution duration of the first task reaches a first predetermined duration, wherein the second task is used to evaluate the correlation between a second query statement and a second object, and both the first query statement and the second query statement include a target shared prefix; A second sending unit is used to send the second task to the first node when it is determined that the execution time of the first task reaches a second predetermined time, so that the first node stores the second key-value resource matching the second task in the first storage space, and processes the second task based on the second key-value resource and the target resource in the first key-value resource, wherein the second key-value resource includes a key-value resource matching the reference element in the second query statement and a key-value resource matching the second object, and the target resource is a key-value resource matching the target shared prefix.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by a processor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images