Server, central server and task processing method based on large model
By allocating multiple GPUs in the server and dynamically scheduling resources, using the combination of small models and large models to handle complex tasks, the problem of low task processing accuracy and efficiency in the prior art is solved, and more efficient and accurate task processing is achieved.
Patent Information
- Application Number
- CN202411875163.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the accuracy and efficiency of task processing are low, especially when dealing with complex scenarios, the accuracy of small models is insufficient, while the implementation cost of large models is high and the inference time is long.
By allocating the first GPU for small model tasks and the second GPU for large model tasks in the server, and dynamically scheduling resources when receiving the large model fusion task, the small model is used for preliminary processing, and determining whether to re-check the large model based on the results.
The accuracy and efficiency of task processing are improved, resource utilization is optimized through the processing method of multi-GPU combination, and the accuracy of task processing is improved through the combination of size and size models.
Smart Images
Figure CN119847735A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a server, a central server, and a task processing method based on a large model. Background Art
[0002] In recent years, with the development of artificial intelligence technology, the corresponding task reasoning platform has also been developing continuously. As the business scope continues to expand, the reasoning task scenarios undertaken by the platform are also constantly enriched.
[0003] The advantage of traditional small-model reasoning platforms is that they have low implementation costs, but they have one model for each scenario, and require tailor-made specific models. The amount of data required for single-model development is huge, and the launch is slow. In addition, for complex scenarios, the accuracy of small models is low, making it difficult to respond quickly and the accuracy is difficult to meet user needs. With the rapid development of large models, small-model reasoning platforms can no longer meet today's reasoning needs. Large models have high-accuracy understanding capabilities for scenarios with fuzzy rules, such as fights, traffic accidents and other complex events. The amount of data required for single-model development is small, and the launch is fast, but it has the disadvantages of high implementation costs and long reasoning time.
[0004] Therefore, how to improve the accuracy and efficiency of reasoning task processing becomes an urgent problem to be solved. Summary of the invention
[0005] The embodiments of the present application provide a server, a central server and a task processing method based on a large model to solve the problems of low accuracy and efficiency of task processing in the prior art.
[0006] The present application provides a server, the server comprising a first GPU allocated for a small model task, and a second GPU allocated for a large model task;
[0007] The server is configured to:
[0008] If the algorithm type corresponding to the pending task sent by the central server is the fusion of large and small models, and the remaining resources of the first GPU cannot process the pending task based on the small model, then obtaining the remaining resources of the second GPU;
[0009] If the remaining resources of the second GPU can process the task to be processed based on the small model, then loading the target small model corresponding to the task to be processed in the second GPU, and determining a first result of the task to be processed based on the target small model;
[0010] According to the first result, determine the data to be reviewed of the target large model corresponding to the task to be processed, load the target large model in the second GPU, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0011] The present application provides a central server, which is configured as follows:
[0012] If the currently received pending task meets the task processing conditions, the current resource occupancy information of each server connected to itself is obtained;
[0013] Determine a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed;
[0014] The tasks to be processed are distributed to the target server so that the target server processes the tasks to be processed.
[0015] The present application provides a task processing method based on a large model, the method comprising:
[0016] If the algorithm type corresponding to the pending task sent by the central server is the fusion of large and small models, and the remaining resources of the first resource space cannot process the pending task based on the small model, then the remaining resources of the second resource space are obtained;
[0017] If the remaining resources of the second resource space can process the task to be processed based on the small model, then loading the target small model corresponding to the task to be processed in the second resource space, and determining the first result of the task to be processed based on the target small model;
[0018] According to the first result, the data to be reviewed of the target large model corresponding to the task to be processed is determined, the target large model is loaded in the second resource space, the data to be reviewed is reviewed based on the target large model, and the target result of the task to be processed is obtained.
[0019] The present application provides a task scheduling method, the method comprising:
[0020] If the currently received pending task meets the task processing conditions, the current resource occupancy information of each server connected to itself is obtained;
[0021] Determine a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed;
[0022] The tasks to be processed are distributed to the target server so that the target server processes the tasks to be processed.
[0023] The present application provides a task processing device based on a large model, the device comprising:
[0024] An acquisition module, configured to acquire the remaining resources of the second resource space if the algorithm type corresponding to the pending task sent by the central server is a fusion of large and small models, and the remaining resources of the first resource space cannot process the pending task based on the small model;
[0025] a first processing module, configured to load a target small model corresponding to the task to be processed in the second resource space if the remaining resources in the second resource space can process the task to be processed based on the small model, and determine a first result of the task to be processed based on the target small model;
[0026] The second processing module is used to determine the data to be reviewed of the target large model corresponding to the task to be processed based on the first result, load the target large model in the second resource space, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0027] The present application provides a task scheduling device, which includes:
[0028] The acquisition module is used to obtain the current resource occupancy information of each server connected to itself if the currently received pending task meets the task processing condition;
[0029] A determination module, used for determining a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed;
[0030] The scheduling module is used to distribute the tasks to be processed to the target server so that the target server processes the tasks to be processed.
[0031] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned large model-based task processing methods, or any of the above-mentioned task scheduling methods when executing a computer program stored in a memory.
[0032] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned large model-based task processing methods, or any of the above-mentioned task scheduling methods.
[0033] Since in the embodiment of the present application, the server is allocated with a first GPU for executing small model tasks and a second GPU for executing large model tasks, when processing tasks, if the algorithm type corresponding to the task to be processed received from the central server is large and small model fusion, and the remaining resources of the first GPU cannot process the task to be processed based on the small model, in order to be able to process the task to be processed in a timely manner, the remaining resources of the second GPU are obtained. If the remaining resources of the second GPU can process the task to be processed based on the small model, the target small model corresponding to the task to be processed is loaded in the second GPU, and the first result of the task to be processed is determined based on the target small model. According to the first result, the data to be reviewed of the target large model corresponding to the task to be processed is determined, and then the target large model is loaded in the second GPU, and the data to be reviewed is reviewed based on the target large model to obtain the target result of the task to be processed. The resource utilization rate and the efficiency of task processing are improved by the processing method of multi-GPU combination, and the accuracy of task processing is improved by the combination of large and small models. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0035] Figure 1 A schematic diagram of the structure of a server provided in an embodiment of the present application;
[0036] Figure 2 A schematic diagram of a task processing process provided in an embodiment of the present application;
[0037] Figure 3 A schematic diagram of an overall business architecture process provided for an embodiment of the present application;
[0038] Figure 4 A schematic diagram of a computing power registration page provided in an embodiment of the present application;
[0039] Figure 5 A schematic diagram of adding a page to an algorithm provided in an embodiment of the present application;
[0040] Figure 6 A schematic diagram of a task adding page provided in an embodiment of the present application;
[0041] Figure 7 A schematic diagram of a hierarchical scheduling strategy provided in an embodiment of the present application;
[0042] Figure 8A schematic diagram for comparing the effects of different solutions provided in the embodiments of the present application;
[0043] Fig. 9 A schematic diagram of a task processing process based on a large model provided in an embodiment of the present application;
[0044] Fig.10 A schematic diagram of a task scheduling process provided in an embodiment of the present application;
[0045] Fig.11 A schematic diagram of the structure of a task processing device based on a large model provided in an embodiment of the present application;
[0046] Fig.12 A schematic diagram of the structure of a task scheduling device provided in an embodiment of the present application;
[0047] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.
[0049] Most current artificial intelligence multimodal (video, image, text, etc.) task platforms use small models in the process of reasoning task scheduling. Although small models have high reasoning efficiency, it is difficult to improve the accuracy for complex events, which is a big pain point, and the cycle for developing a small model algorithm is long. With the development of large models, the reasoning accuracy for complex scenes can be quickly improved and the development efficiency is fast, but the implementation cost is high and the reasoning efficiency is low. The present invention overcomes the disadvantages of both, integrates the advantages of both, and proposes a large and small model fusion reasoning platform architecture design method. An embodiment of the present application provides a server, a central server and a task processing method based on a large model, wherein the server includes a first GPU assigned to a small model task, and a second GPU assigned to a large model task: the server is configured to: if a task to be processed is received from the central server, and the remaining resources of the first GPU cannot process the task to be processed based on the small model, then obtain the remaining resources of the second GPU; if the remaining resources of the second GPU can process the task to be processed based on the small model, then load the target small model corresponding to the task to be processed into the second GPU, and determine the first result of the task to be processed based on the target small model; based on the first result, determine the data to be reviewed of the target large model corresponding to the task to be processed, load the target large model into the second GPU, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0050] Figure 1 A structural diagram of a server provided for an embodiment of the present application, the server 10 includes a first GPU 101 assigned to a small model task, and a second GPU 102 assigned to a large model task; the server 10 is configured as follows: if a pending task sent by a central server is received, and the remaining resources of the first GPU 101 cannot process the pending task based on the small model, then the remaining resources of the second GPU 101 are obtained; if the remaining resources of the second GPU 102 can process the pending task based on the small model, then the target small model corresponding to the pending task is loaded in the second GPU 102, and a first result of the pending task is determined based on the target small model; according to the first result, the data to be reviewed of the target large model corresponding to the pending task is determined, the target large model is loaded in the second GPU 102, and the data to be reviewed is reviewed based on the target large model to obtain the target result of the pending task.
[0051] In order to improve the accuracy and efficiency of task processing, in an embodiment of the present application, when processing tasks, a distributed method can be used for processing. In an embodiment of the present application, the server 10 can receive the pending tasks sent by the central server, wherein the central server can send the pending tasks that need to be processed to the corresponding server 10 for task processing according to the actual task requirements. Among them, the number of pending tasks can be one or more. If the server 10 receives the pending tasks sent by the central server, and the algorithm type corresponding to the pending tasks is the fusion of large and small models, it means that the pending tasks need to be preprocessed using a small model first, and then the processing results of the small model are reviewed using a large model. In an embodiment of the present application, the current remaining resources of the first GPU 101 can be obtained, and it can be determined whether the remaining resources can load the small model corresponding to the pending tasks and perform corresponding processing. In order to be able to process the pending tasks in a timely manner and improve resource utilization, if it is determined that the remaining resources of the first GPU 101 cannot process the pending tasks based on the small model, the remaining resources of the second GPU 102 allocated for the large model task can be obtained. If the remaining resources of the second GPU 102 can process the task to be processed based on the small model, the second GPU 102 can be temporarily used to process the small model of the task to be processed. The target small model corresponding to the task to be processed can be loaded in the second GPU 102, and the first result of the task to be processed can be determined based on the target small model. The target small model corresponding to the task to be processed can be determined based on the information when the user adds the task. For example, when adding a task, the user can select the name of the algorithm required to process the task, and the small model involved in the algorithm corresponding to the algorithm name is the target small model that needs to be loaded.
[0052] Since the small model task processing efficiency is high but the accuracy is low, in order to accurately process the task to be processed, after obtaining the first result of the task to be processed determined by the target small model, the target large model corresponding to the task to be processed can be used for re-inspection.
[0053] In an embodiment of the present application, the data to be reviewed of the target large model corresponding to the task to be processed can be determined based on the first result. In an embodiment of the present application, different strategies for screening the data to be reviewed can be pre-configured for different tasks to be processed. Exemplarily, assuming that the current task to be processed is a target recognition task, 50 images to be processed of the task to be processed are currently received, and after preliminary recognition of the small target model, it is determined that there are targets to be recognized in 20 images to be processed, then the 20 images to be processed can be determined as data to be reviewed, and the target large model can be used to review the 20 images to be processed. Of course, the data to be reviewed can also be spliced with other prompt words or data and input into the target large model, and technicians in this field can configure it as needed.
[0054] After the data to be reviewed is determined, the target large model corresponding to the task to be processed can be loaded and processed in the second GPU 102, and the data to be reviewed can be reviewed based on the target large model to obtain the target result of the task to be processed. For example, the task to be processed and the data to be reviewed can be input into the target large model together, so that the target large model determines whether the first result of the target small model is correct; the task to be processed can also be input into the target large model to obtain the second result, and the target result of the task to be processed can be determined based on whether the first result is similar to the second result.
[0055] Since in the embodiment of the present application, the server is allocated with a first GPU for executing small model tasks and a second GPU for executing large model tasks, when processing tasks, if the algorithm type corresponding to the task to be processed received from the central server is large and small model fusion, and the remaining resources of the first GPU cannot process the task to be processed based on the small model, in order to be able to process the task to be processed in a timely manner, the remaining resources of the second GPU are obtained. If the remaining resources of the second GPU can process the task to be processed based on the small model, the target small model corresponding to the task to be processed is loaded in the second GPU, and the first result of the task to be processed is determined based on the target small model. According to the first result, the data to be reviewed of the target large model corresponding to the task to be processed is determined, and then the target large model is loaded in the second GPU, and the data to be reviewed is reviewed based on the target large model to obtain the target result of the task to be processed. The resource utilization rate and the efficiency of task processing are improved by the processing method of multi-GPU combination, and the accuracy of task processing is improved by the combination of large and small models.
[0056] In order to further improve the accuracy and efficiency of task processing, based on the above embodiment, in the embodiment of the present application, the server 10 is further configured as follows:
[0057] If the remaining resources of the first GPU can process the task to be processed based on the small model, loading the target small model in the first GPU, and determining a first result of the task to be processed based on the target small model;
[0058] According to the first result, determine the data to be reviewed of the target large model corresponding to the task to be processed, load the target large model on the second GPU, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0059] After the server 10 obtains the pending task sent by the central server, if it is determined that the remaining resources of the first GPU 101 can process the pending task based on the small model, the target small model can be loaded in the first GPU 101, and the first result of the pending task can be determined based on the target small model. In order to conduct a recheck based on the large model, after obtaining the first result of the target small model in the first GPU 101, the rechecked data of the target large model corresponding to the pending task can be determined based on the first result, and the second GPU 102 can be controlled to load the target large model corresponding to the pending task, and the data to be processed can be rechecked based on the target large model to obtain the target result of the pending task.
[0060] In an embodiment of the present application, if the remaining resources of the first GPU can process the task to be processed based on the small model, the target small model is loaded in the first GPU, and the first result of the task to be processed is determined based on the target small model. Then, the target large model is loaded in the second GPU, and the target large model is used for rechecking, thereby improving the accuracy and efficiency of task processing.
[0061] In order to ensure the normal progress of task processing, based on the above embodiments, in the embodiment of the present application, the server 10 is further configured as follows:
[0062] Determine whether the remaining resources of the second GPU are capable of rechecking the data to be rechecked. If so, continue to perform subsequent steps to determine the data to be rechecked of the target large model corresponding to the task to be processed based on the first result.
[0063] Since the remaining resources in the current second GPU 102 may be too few, it will be impossible to recheck the data to be rechecked. Therefore, in an embodiment of the present application, after obtaining the first result, it can be determined whether the remaining resources of the second GPU 102 can be used to recheck the data to be rechecked. If so, it means that the remaining resources are sufficient, and the subsequent step of loading the target large model corresponding to the task to be processed in the second GPU can be continued.
[0064] In the embodiment of the present application, before using the second GPU to recheck the task to be processed, it is determined whether the remaining resources of the second GPU can recheck the data to be rechecked. If so, the subsequent task is continued to be executed, thereby ensuring the normal progress of task processing.
[0065] In order to further improve the efficiency of task processing, based on the above embodiments, in the embodiment of the present application, the server is further configured as follows:
[0066] If the remaining resources of the second GPU cannot recheck the data to be rechecked, determine whether the remaining resources of the first GPU can recheck the data to be rechecked based on the target large model; if so, load the target large model in the first GPU, recheck the data to be rechecked based on the target large model, and obtain the target result of the task to be processed.
[0067] If the remaining resources of the second GPU 102 cannot recheck the data to be rechecked, in order to process the recheck task in time, the remaining resources of the first GPU 101 can be occupied for rechecking. In the embodiment of the present application, the remaining resources of the current first GPU 101 can be obtained, and it can be determined whether the remaining resources can recheck the data to be rechecked based on the large model. If so, it means that the first GPU 101 is currently relatively idle, and the target large model can be loaded in the first GPU 101, and the data to be rechecked can be rechecked based on the target large model to obtain the target result of the task to be processed.
[0068] In an embodiment of the present application, if the remaining resources of the second GPU cannot re-check the data to be re-checked, it is further determined whether the remaining resources of the first GPU can re-check the data to be re-checked. If so, the target large model is temporarily used in the first GPU to re-check the data to be re-checked, thereby improving resource utilization and task processing efficiency.
[0069] In order to further improve the efficiency of task processing, based on the above embodiments, in the embodiment of the present application, the server 10 is further configured as follows:
[0070] If the remaining resources of the first GPU are unable to recheck the data to be rechecked based on the target large model, the data to be rechecked corresponding to the task to be processed will be added to the large model task queue, so that the data to be rechecked can be obtained from the large model task queue for processing when the resources of the second GPU are idle.
[0071] If it is determined that the remaining resources of the first GPU 101 are unable to re-inspect the data to be re-inspected based on the large model, it means that the first GPU 101 and the second GPU 102 are currently busy. The data to be re-inspected corresponding to the task to be processed can be added to the large model task queue, so that the data to be re-inspected can be obtained from the large model task queue for processing when the resources of the second GPU 102 are idle.
[0072] In order to further improve the efficiency of task processing, based on the above embodiments, in the embodiment of the present application, the server 10 is further configured as follows:
[0073] If the remaining resources of the second GPU 102 cannot process the pending tasks based on the small model, the pending tasks are added to the small model task queue so that the pending tasks can be obtained from the small model task queue for processing when the resources of the first GPU 101 are idle.
[0074] If the remaining resources of the second GPU 102 cannot process the pending tasks based on the small model, it means that the first GPU 101 and the second GPU 102 are currently busy. The pending tasks can be added to the small model task queue so that the pending tasks can be obtained from the small model task queue for processing when the resources of the first GPU 101 are idle.
[0075] In order to improve resource utilization, based on the above embodiments, in the embodiment of the present application, the server 10 is further configured as follows:
[0076] After the to-be-processed task is processed based on the target small model, the target small model is unloaded in the corresponding GPU.
[0077] In order to prevent the idle model from occupying server resources all the time, in an embodiment of the present application, after the processing of the task to be processed based on the target small model is completed, the target small model can be unloaded in the corresponding GPU.
[0078] In order to improve resource utilization, based on the above embodiments, in the embodiment of the present application, the server 10 is further configured as follows:
[0079] After the to-be-processed task is processed based on the target large model, the target large model is unloaded in the corresponding GPU.
[0080] In order to prevent the idle model from occupying server resources all the time, in an embodiment of the present application, after the processing of the task to be processed based on the target large model is completed, the target large model is unloaded in the corresponding GPU.
[0081] In one possible implementation, if the algorithm type corresponding to the task to be processed received from the central server is only a small model, the remaining resources of the first GPU can be obtained; if the remaining resources of the first GPU cannot process the task to be processed based on the small model, the remaining resources of the second GPU are obtained; if the remaining resources of the second GPU can process the task to be processed based on the small model, the target small model corresponding to the task to be processed is loaded in the second GPU, and the target result of the task to be processed is determined based on the target small model.
[0082] In one possible implementation, if the algorithm type corresponding to the task to be processed received from the central server is only a large model, the remaining resources of the second GPU can be obtained; if the remaining resources of the second GPU cannot process the task to be processed based on the large model, the remaining resources of the first GPU are obtained; if the remaining resources of the first GPU can process the task to be processed based on the large model, the target large model corresponding to the task to be processed is loaded in the first GPU, and the target result of the task to be processed is determined based on the target large model.
[0083] The task execution process is described below with reference to a specific embodiment. Figure 2 A schematic diagram of a task processing process provided for an embodiment of the present application. Assuming that there are currently 25 offline video files, the central dispatch server determines that a certain server can undertake the processing task of these 25 video files, and dispatches the task to be processed to the server for reasoning; wherein the server pre-allocates 4 4090 GPU graphics cards for small model tasks, each 4090 can load 5 small models; and pre-allocates 4 4090 GPU graphics cards for large model tasks, each 4090 GPU can load 1 large model.
[0084] Currently, the GPU graphics card of the server used to load the small model is idle, so 20 small model analysis tasks are started. Through the batch processing mechanism, the 20 pending tasks are processed to obtain the first result of each small model output.
[0085] For the remaining 5 tasks to be processed, it is determined whether the GPU used to load the large model is idle. If it is idle, the large model GPU is used to load the small model and process the remaining 5 tasks to be processed; if it is not idle, the 5 tasks to be processed are stored in the small model task queue and wait for loading and processing when the small model GPU is idle.
[0086] The first result after processing by the small model needs to be judged again by the large model. At this time, it can be judged whether the GPU used to load the large model can receive all the re-inspection tasks. If it can, it will be sent to the large model GPU through the batch processing mechanism to complete the secondary judgment; if not, it will be judged whether the small model GPU is idle and can load the large model. If so, the small model GPU can be used to load the large model to complete the judgment; otherwise, it will enter the large model task queue and wait for the large model GPU to be idle for loading and processing.
[0087] As mentioned above, the model needs to be loaded in advance before use so that it can respond in time when the pending tasks arrive; it should be unloaded after use to put the GPU resources in an idle state for flexible scheduling. In addition, for models that are used frequently, they can be kept in a loaded state to reduce the time loss caused by loading. The specific decision should be based on the actual situation.
[0088] On the basis of the above embodiments, in an embodiment of the present application, a central server is further provided, and the central server is configured as follows:
[0089] If the currently received pending task meets the task processing conditions, the current resource occupancy information of each server connected to itself is obtained;
[0090] Determine a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed;
[0091] The tasks to be processed are distributed to the target server so that the target server processes the tasks to be processed.
[0092] In order to achieve reasonable distribution of tasks, in an embodiment of the present application, a central server may also be configured, which is used to receive pending tasks sent by the front-end device and reasonably distribute the pending tasks. In an embodiment of the present application, if the pending tasks currently received by the central server meet the task processing conditions, the current resource occupancy information of each server connected to itself can be obtained to facilitate task distribution based on the resource occupancy information. Among them, the task processing condition can be that the accumulated number of pending tasks meets a preset number threshold, and the preset number threshold can be any integer, such as 1, 50, 500, etc.
[0093] After obtaining the resource occupancy information of each server 10, the target server for executing the pending task can be determined according to each resource occupancy information and the pending task. The specific allocation of the pending tasks can be configured by technicians in this field as needed. For example, the current pending tasks can be scheduled to the server node with the most sufficient resources; or after a batch of pending tasks arrive, the reasoning tasks can be distributed after comprehensive evaluation. Different scheduling algorithms are involved here: for example, the sliding average algorithm, the instantaneous weighted algorithm, the particle swarm algorithm, the improved particle swarm algorithm, etc.
[0094] After the target server is determined, the task to be processed can be distributed to the target server so that the target server processes the task to be processed, wherein the target server processes the received task to be processed, which is the process of task processing by the server 10 described in the above embodiments.
[0095] In the embodiment of the present application, according to the resource utilization of different servers 10, the central server distributes the tasks to be processed to achieve dynamic scheduling.
[0096] Figure 3 A schematic diagram of an overall business architecture process provided by an embodiment of the present application, such as Figure 3 As shown, the user can input the pending tasks on the platform front-end page or business system, and the pending tasks may include real-time video streams, offline video streams, pictures, etc. After obtaining the pending tasks, the pending tasks are sent to the inference platform, and the device used by the inference platform to execute the tasks may be the server 10 and the central server described in the above embodiments.
[0097] The inference platform performs a series of preprocessing on the data, such as decoding, frame extraction, pixel compression, etc. And according to the type of pending tasks configured by the business system, the corresponding model scheduling strategy is selected: only small model, only large model, and fusion of large and small models.
[0098] If the scheduling strategy is small model only, the preprocessed data is assigned to the small model algorithm module to obtain the inference result. This strategy is suitable for scenarios with large inference tasks, high inference efficiency requirements, and relatively low inference accuracy requirements.
[0099] If the scheduling strategy is large model only, the preprocessed data is assigned to the large model algorithm module to obtain the inference result. This strategy is suitable for scenarios with small inference tasks, relatively low inference efficiency requirements, and high inference accuracy requirements;
[0100] If the scheduling strategy is to integrate large and small models, the preprocessed data will be first assigned to the small model algorithm module to obtain the first result; then the first result will be sent to the large model algorithm module for secondary judgment to obtain the final inference result. This strategy is suitable for scenarios with large inference tasks, high inference efficiency requirements, and high inference accuracy requirements.
[0101] The inference results are sent to the message queue, decoupled from the front-end page of the inference platform or the business system, and wait for query and display by the front-end page of the inference platform or the business system.
[0102] In order to configure the parameters during task processing, in the embodiment of the present application, the user can register the computing power on the front-end page of the inference platform, enter the configuration information of each server, and then add the algorithm on the algorithm adding page and select the algorithm type, where the algorithm type includes only large model, only small model, and large and small model fusion. Finally, add the task, and after creating the task, you can process the task.
[0103] Figure 4 A schematic diagram of a computing power registration page provided in an embodiment of the present application, such as Figure 4 As shown, users can fill in the node name, node IP, system version, GPU information, CPU information, memory information, hard disk information, etc. on this page. For example, you can fill in the following example:
[0104] Node Name: Node1
[0105] Node IP: 10.18.1.1
[0106] System version: Ubuntu 16.04
[0107] GPU information: Tesla T4
[0108] CPU information: Intel(R)Xeon(R)Gold 6254CPU@3.10GHz
[0109] Memory information: 32G
[0110] Hard disk information: 500G.
[0111] Figure 5 A schematic diagram of adding a page to an algorithm provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, users can fill in information such as algorithm name, algorithm scenario, algorithm address, algorithm description, algorithm type, etc. on this page. For example, it can be filled in according to the following example:
[0112] Algorithm name: Tool detection
[0113] Algorithm scenario: Construction
[0114] Algorithm address: https: / / 1*.**.*.1:80 / ******
[0115] Algorithm description: Used for construction tool detection in construction site scenarios.
[0116] Figure 6 A schematic diagram of a task adding page provided in an embodiment of the present application is shown as follows: Figure 6 As shown, users can fill in the task name, data type, data name, data address, algorithm name and other information on this page. For example, you can fill in the following example:
[0117] Task name: Construction tool inspection at construction site No. 1
[0118] Data type: You can choose: All, Offline pictures, Offline videos, Real-time videos
[0119] Data name: Offline video data of construction site No. 1 on November 10
[0120] Data address: ftp: / / hos***a**e:port / d*ta
[0121] Algorithm name: Tool Detection.
[0122] Based on the above embodiments, the scheduling strategy for task processing provided by the present application can be summarized as a four-level hierarchical scheduling: Figure 7 A schematic diagram of a hierarchical scheduling strategy provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the process includes four levels of scheduling.
[0123] 1. Level 1 scheduling, also known as central scheduling, is a customized scheduling strategy for the central scheduling link. After registering the computing power, the tasks to be processed are distributed according to the resource utilization of different node servers to achieve dynamic scheduling;
[0124] 2. Secondary scheduling is a customized scheduling strategy for the task processing link. When the inference task is distributed from the central scheduling link to the task processing link, model arrangement is required, that is, to determine the model required for each task to be processed and the order of use, and then preload it to improve the response time. According to the current usage of server resources, dynamic loading is performed, and after the model is used, it is dynamically unloaded to prevent idle models from occupying server resources all the time. In addition, the secondary scheduling strategy also provides management functions for the data preprocessing link and shared memory to ensure the normal and efficient operation of the inference task. The data preprocessing link adopts pixel compression (reducing the data size and improving the transmission and algorithm reasoning efficiency while ensuring the accuracy of the algorithm), video stream multiplexing (avoiding the same preprocessing operation for the same data multiple times) and hardware acceleration strategy (improving the efficiency of data preprocessing); shared memory is responsible for creating and maintaining signaling communication channels and maintaining data transmission channels. Key data is processed in memory to further improve the response rate.
[0125] 3. The third-level scheduling is a customized scheduling strategy for algorithm logic. After the second-level scheduling strategy arranges and preloads the model, the third-level scheduling strategy implements specific tasks to be processed through conventional model calls (single algorithm single model) or heterogeneous computing (single algorithm multiple models serial or parallel).
[0126] 4. Four-level scheduling is a customized strategy for small model algorithm modules and large model algorithm modules, including storage (kv cache, paged memory allocation), scheduling strategy (time-sharing scheduling, continuous batch processing), decoding technology (predictive reasoning), business logic (model instance reuse) and other acceleration measures. It can at least increase the processing efficiency of small model and large model algorithm acceleration by 2 times.
[0127] In the embodiment of the present application, a four-level hierarchical scheduling strategy for platform computing power is proposed based on the overall business architecture process, and computing power strategy customization is performed for each link from data access to the platform to obtaining processing results, thereby reducing task processing time and improving task processing efficiency.
[0128] It should be noted that different application scenarios have different requirements. When adding algorithms, users can configure the algorithm type according to the application scenarios to which the corresponding algorithms are applicable. For example, for the algorithm of scenario 1, only the large model is used for processing, for the algorithm of scenario 2, only the small model is used for processing, and for the algorithm of scenario 3, a fusion of large and small models is used for processing. Figure 8 A schematic diagram for comparing the effects of different solutions provided in the embodiments of the present application is shown in FIG. Figure 8As shown in the figure, take the detection of large construction machinery on both sides of the railway as an example. This scene is complex and changeable, which is a great challenge for traditional small models. First, extract frames from the railway video to obtain 8 pictures (only 1 picture is expected to be a large construction machinery), and perform three detection schemes respectively. Among them, it takes 1 second for the large model to process 1 picture, and 200ms for the small model to process 1 picture.
[0129] (1) Solution 1: Only the small model is used, which takes 8*200ms=1.6s and detects two large machines. Result 1 is an excavator, which is detected correctly; Result 2 is a building material, which is mistakenly detected as a large construction machine by the small model;
[0130] (2) Solution 2: Only the large model is used. It takes 8*1s=8s and detects 1 result image, which is in line with expectations.
[0131] (3) Solution 3: Fusion of large and small models: The eight images are first screened by the small model to obtain two images, Result 1 and Result 2. These two images are then sent to the large model for secondary judgment to obtain the final expected result. This takes 8*200ms+2*1s=3.6s.
[0132] From the above comparison, it can be seen that the small model takes less time, but the detection accuracy is low; the large model has a high detection accuracy, but it takes a long time; therefore, by integrating the advantages of schemes 1 and 2 through the fusion of large and small models, scheme 3 is formed. For some application scenarios, the fusion of large and small models integrates the advantages of large and small models, achieves accurate detection of difficult events, fast response to emergencies, and reduces task processing costs and time.
[0133] On the basis of the above-mentioned embodiments, the embodiment of the present application further provides a task processing method based on a large model. The task processing method based on a large model is consistent with the processing process of the server in the above-mentioned embodiments, and will not be repeated here. Fig. 9 A schematic diagram of a task processing process based on a large model provided in an embodiment of the present application, the process includes:
[0134] S901: If the algorithm type corresponding to the pending task received from the central server is the fusion of large and small models, and the remaining resources of the first resource space cannot process the pending task based on the small model, then the remaining resources of the second resource space are obtained.
[0135] S902: If the remaining resources of the second resource space can process the task to be processed based on the small model, then load the target small model corresponding to the task to be processed into the second resource space, and determine the first result of the task to be processed based on the target small model.
[0136] S903: According to the first result, determine the data to be reviewed of the target large model corresponding to the task to be processed, load the target large model in the second resource space, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0137] In a possible implementation, the method further includes:
[0138] If the remaining resources of the first resource space can process the task to be processed based on the small model, loading the target small model in the first resource space, and determining a first result of the task to be processed based on the target small model;
[0139] According to the first result, the data to be reviewed of the target large model corresponding to the task to be processed is determined, the target large model is loaded in the second resource space, the data to be reviewed is reviewed based on the target large model, and the target result of the task to be processed is obtained.
[0140] In a possible implementation, after determining the first result of the task to be processed based on the target small model and before determining the to-be-processed data of the target large model corresponding to the task to be processed according to the first result, the method further includes:
[0141] Determine whether the remaining resources of the second GPU are capable of rechecking the data to be rechecked. If so, continue to perform subsequent steps to determine the data to be rechecked of the target large model corresponding to the task to be processed based on the first result.
[0142] In a possible implementation, the method further includes:
[0143] If the remaining resources of the second GPU cannot recheck the data to be rechecked, determine whether the remaining resources of the first GPU can recheck the data to be rechecked based on the target large model; if so, load the target large model in the first GPU, recheck the data to be rechecked based on the target large model, and obtain the target result of the task to be processed.
[0144] In a possible implementation, the method further includes:
[0145] If the remaining resources of the first GPU are unable to recheck the data to be rechecked based on the target large model, the data to be rechecked corresponding to the task to be processed will be added to the large model task queue, so that the data to be rechecked can be obtained from the large model task queue for processing when the resources of the second GPU are idle.
[0146] In a possible implementation, the method further includes:
[0147] If the remaining resources of the second GPU cannot process the pending tasks based on the target small model, the pending tasks are added to the small model task queue so that the pending tasks can be obtained from the small model task queue for processing when the resources of the first GPU are idle.
[0148] In a possible implementation, the method further includes:
[0149] After the to-be-processed task is processed based on the target small model, the target small model is unloaded in the corresponding GPU.
[0150] In a possible implementation, the method further includes:
[0151] After the to-be-processed task is processed based on the target large model, the target large model is unloaded in the corresponding GPU.
[0152] On the basis of the above embodiments, the embodiment of the present application further provides a task scheduling method. The task processing method based on the large model is consistent with the processing process of the central server in the above embodiments, and will not be repeated here. Fig.10 A schematic diagram of a task scheduling process provided in an embodiment of the present application, the process includes:
[0153] S1001: If the currently received task to be processed meets the task processing condition, the current resource occupancy information of each server connected to itself is obtained.
[0154] S1002: Determine a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed.
[0155] S1003: Distribute the pending tasks to the target server, so that the target server processes the pending tasks.
[0156] Based on the above embodiments, Fig.11 A schematic diagram of a task processing device based on a large model provided in an embodiment of the present application, the device comprising:
[0157] The acquisition module 1101 is used to acquire the remaining resources of the second resource space if the algorithm type corresponding to the pending task sent by the central server is the fusion of large and small models, and the remaining resources of the first resource space cannot process the pending task based on the small model;
[0158] The first processing module 1102 is configured to load a target small model corresponding to the task to be processed in the second resource space if the remaining resources in the second resource space can process the task to be processed based on the small model, and determine a first result of the task to be processed based on the target small model;
[0159] The second processing module 1103 is used to determine the data to be reviewed of the target large model corresponding to the task to be processed based on the first result, load the target large model in the second resource space, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0160] In a possible implementation, the first processing module 1102 is further configured to load the target small model in the first resource space if the remaining resources in the first resource space can process the task to be processed based on the small model, and determine a first result of the task to be processed based on the target small model;
[0161] The second processing module 1103 is also used to determine the data to be reviewed of the target large model corresponding to the task to be processed based on the first result, load the target large model in the second resource space, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
[0162] In a possible implementation, the second processing module 1103 is further used to determine whether the remaining resources of the second GPU are capable of rechecking the data to be rechecked. If so, continue to execute the subsequent steps to determine the data to be rechecked of the target large model corresponding to the task to be processed based on the first result.
[0163] In a possible implementation, the second processing module 1103 is further used to determine whether the remaining resources of the first GPU can recheck the data to be rechecked based on the target large model if the remaining resources of the second GPU cannot recheck the data to be rechecked; if so, load the target large model in the first GPU, recheck the data to be rechecked based on the target large model, and obtain the target result of the task to be processed.
[0164] In a possible implementation, the first processing module 1102 is further used to add the data to be reviewed corresponding to the task to be processed to a large model task queue if the remaining resources of the first GPU are unable to review the data to be reviewed based on the target large model, so that the data to be reviewed can be obtained from the large model task queue for processing when the resources of the second GPU are idle.
[0165] In a possible implementation, the first processing module 1102 is further used to add the pending task to the small model task queue if the remaining resources of the second GPU cannot process the pending task based on the target small model, so that the pending task can be obtained from the small model task queue for processing when the resources of the first GPU are idle.
[0166] In a possible implementation, the first processing module 1102 is further configured to unload the target small model from the corresponding GPU after completing processing of the task to be processed based on the target small model.
[0167] In a possible implementation, the first processing module 1102 is further configured to unload the target large model from the corresponding GPU after completing processing of the task to be processed based on the target large model.
[0168] Based on the above embodiments, Fig.12 A schematic diagram of a task scheduling device structure provided in an embodiment of the present application, the device comprising:
[0169] The acquisition module 1201 is used to acquire the current resource occupancy information of each server connected to the server if the currently received task to be processed meets the task processing condition;
[0170] A determination module 1202, configured to determine a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed;
[0171] The scheduling module 1203 is used to distribute the tasks to be processed to the target server so that the target server processes the tasks to be processed.
[0172] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, it includes: a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304;
[0173] The memory 1303 stores a computer program. When the program is executed by the processor 1301, the processor 1301 executes the steps of the large model-based task processing method or the steps of the task scheduling method described in the above embodiments.
[0174] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1302 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0175] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.
[0176] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based task processing method or the steps of the task scheduling method described in the above embodiments.
[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0178] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0181] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A server, characterized in that: The server includes a first GPU assigned to a small model task, and a second GPU assigned to a large model task; The server is configured to: If the algorithm type corresponding to the pending task sent by the central server is the fusion of large and small models, and the remaining resources of the first GPU cannot process the pending task based on the small model, then obtaining the remaining resources of the second GPU; If the remaining resources of the second GPU can process the task to be processed based on the small model, then loading the target small model corresponding to the task to be processed in the second GPU, and determining a first result of the task to be processed based on the target small model; According to the first result, determine the data to be reviewed of the target large model corresponding to the task to be processed, load the target large model in the second GPU, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
2. The server according to claim 1, characterized in that: The server is also configured to: If the remaining resources of the first GPU can process the task to be processed based on the small model, loading the target small model in the first GPU, and determining a first result of the task to be processed based on the target small model; According to the first result, determine the data to be reviewed of the target large model corresponding to the task to be processed, load the target large model on the second GPU, review the data to be reviewed based on the target large model, and obtain the target result of the task to be processed.
3. The server according to claim 1 or 2, characterized in that: The server is also configured to: Determine whether the remaining resources of the second GPU are capable of rechecking the data to be rechecked. If so, continue to perform subsequent steps to determine the data to be rechecked of the target large model corresponding to the task to be processed based on the first result.
4. The server according to claim 3, characterized in that: The server is also configured to: If the remaining resources of the second GPU cannot recheck the data to be rechecked, determine whether the remaining resources of the first GPU can recheck the data to be rechecked based on the target large model; if so, load the target large model in the first GPU, recheck the data to be rechecked based on the target large model, and obtain the target result of the task to be processed.
5. The server according to claim 4, characterized in that: The server is also configured to: If the remaining resources of the first GPU are unable to recheck the data to be rechecked based on the target large model, the data to be rechecked corresponding to the task to be processed will be added to the large model task queue, so that the data to be rechecked can be obtained from the large model task queue for processing when the resources of the second GPU are idle.
6. The server according to claim 1, characterized in that: The server is also configured to: If the remaining resources of the second GPU cannot process the pending tasks based on the target small model, the pending tasks are added to the small model task queue so that the pending tasks can be obtained from the small model task queue for processing when the resources of the first GPU are idle.
7. The server according to claim 1, characterized in that: The server is also configured to: After the to-be-processed task is processed based on the target small model, the target small model is unloaded in the corresponding GPU.
8. The server according to claim 1, characterized in that: The server is also configured to: After the to-be-processed task is processed based on the target large model, the target large model is unloaded in the corresponding GPU.
9. A central server, characterized in that: The central server is configured as follows: If the currently received pending task meets the task processing conditions, the current resource occupancy information of each server connected to itself is obtained; Determine a target server for executing the task to be processed according to each of the resource occupation information and the task to be processed; The tasks to be processed are distributed to the target server so that the target server processes the tasks to be processed.
10. A task processing method based on a large model, characterized in that: The method comprises: If the algorithm type corresponding to the pending task sent by the central server is the fusion of large and small models, and the remaining resources of the first resource space cannot process the pending task based on the small model, then the remaining resources of the second resource space are obtained; If the remaining resources of the second resource space can process the task to be processed based on the small model, then loading the target small model corresponding to the task to be processed in the second resource space, and determining the first result of the task to be processed based on the target small model; According to the first result, the data to be reviewed of the target large model corresponding to the task to be processed is determined, the target large model is loaded in the second resource space, the data to be reviewed is reviewed based on the target large model, and the target result of the task to be processed is obtained.