Task scheduling method for portrait platform under large model architecture

Abstract

The invention provides a task scheduling method for a portrait platform under a large model architecture, and the method comprises the steps: constructing a complete picture platform, carrying out the task triggering monitoring of all business links subordinate to the portrait platform, and obtaining a task created by the platform; performing fine-grained splitting on the created task to obtain a plurality of sub-tasks, placing the sub-tasks in a task pool, and identifying weight feature information of all the sub-tasks in the task pool in the corresponding tasks; on the basis of the weight feature information, a task execution queue about all the subtasks is generated, and a basis is provided for subsequently determining a task scheduling sequence; the execution task information of all the subordinate processes of the computing resource pool of the portrait platform is determined, the subordinate sub-tasks of the task execution queue are scheduled, the sub-tasks are distributed to the corresponding processes, and task fine-grained splitting and task scheduling are introduced into the portrait platform, so that the task execution efficiency is improved. The computing resources of the portrait platform are fully and efficiently utilized for task processing, and the problems of poor availability and poor expandability of the portrait platform caused by shortage of the computing resources are avoided.

Description

Technical Field

[0001] The present invention relates to the field of large model applications, and in particular to a task scheduling method for a portrait platform under a large model architecture. Background Art

[0002] The portrait platform can conveniently support the portrait analysis function based on the portrait data. The object of portrait analysis is not limited to the crowd or a single user, and the data source of portrait analysis can include the completed aggregated label data or behavior detail data. In order to maximize the performance of the portrait platform, it is necessary to optimize in different aspects such as data generation, function construction and architecture design. The existing portrait platform cannot meet the development needs of various industries. In particular, the portrait platform with a long execution time due to tight computing resources has problems such as poor usability and poor scalability, resulting in poor user experience and inaccurate resource cost accounting. Specifically, each business link of the existing portrait platform is executed in sequence. For this reason, it is necessary to detect whether the current business link needs to be executed. When it is determined that it needs to be executed, it will wait until the execution is successful before entering the next business link. However, business links such as crowd creation and portrait calculation rely heavily on big data offline calculation. When computing resources are tight, the corresponding execution time is long. At this time, it is necessary to loop in the code to wait for the current business link to be successfully executed, which leads to problems such as poor usability and poor scalability of the picture platform. Summary of the invention

[0003] The purpose of the present invention is to provide a task scheduling method for a portrait platform under a large model architecture, build a complete picture platform, monitor task triggers for all business links under the portrait platform, obtain tasks created by the platform, and obtain tasks generated by the platform in a timely and accurate manner; fine-grained splitting of the created tasks into several subtasks and placing them in a task pool, identifying the weight feature information of all subtasks in the task pool in their corresponding tasks, and realizing the identification and measurement of the importance and urgency of the subtasks; based on the weight feature information, generating a task execution queue for all subtasks, providing a basis for the subsequent determination of the task scheduling order; also determining the execution task information of all processes under the computing resource pool of the portrait platform, and scheduling the subtasks under the task execution queue, and assigning the subtasks to the corresponding processes, by introducing fine-grained task splitting and task scheduling in the portrait platform, the computing resources of the portrait platform are fully and efficiently utilized for task processing, avoiding the problems of poor availability and scalability of the portrait platform due to tight computing resources.

[0004] The present invention is achieved through the following technical solutions:

[0005] The task scheduling method of the portrait platform under the large model architecture includes:

[0006] Based on the big model architecture, a complete portrait platform is built; task trigger monitoring is performed on all business links under the portrait platform to obtain the tasks created by the portrait platform;

[0007] Fine-grained splitting of the created task to obtain a number of subtasks and placing them into the task pool of the portrait platform; identifying all subtasks in the task pool to obtain weight feature information of each subtask in its corresponding task;

[0008] Based on the weight feature information, a task execution queue for all subtasks is generated; the execution task information of all processes under the computing resource pool of the portrait platform is determined, and based on the execution task information, the subtasks under the task execution queue are scheduled and processed, so as to assign the subtasks to the corresponding processes.

[0009] Optionally, build a complete portrait platform based on the large model architecture, including:

[0010] Obtain the operation mode and operation performance requirements of the portrait platform to be built, and select a matching large model architecture based on the operation mode and the operation performance requirements;

[0011] Based on the matching large model architecture, a complete portrait platform is constructed.

[0012] Optionally, monitoring the task triggering of the portrait platform to obtain the task created by the portrait platform includes:

[0013] Monitor all business links under the portrait platform to obtain task trigger status information of all business links; wherein the task trigger status information includes platform user task trigger request information and pre-task trigger request information;

[0014] Based on the task triggering status information, the tasks created by the portrait platform during the execution of all business links are obtained.

[0015] Optionally, all business links under the portrait platform are monitored to obtain task triggering status information of all business links, including:

[0016] Monitor the business links of resource readiness detection, crowd creation, portrait calculation, portrait analysis report generation, crowd data storage and crowd post-operation on the portrait platform, and obtain the request information from platform users and the execution status information of the pre-task during the execution of each business link;

[0017] Based on the request information from the platform user and the preceding task execution status information, task triggering status information of each business link is obtained.

[0018] Optionally, the created task is finely divided to obtain a number of subtasks and placed in the task pool of the portrait platform, including:

[0019] Based on the K-means++ algorithm, the execution independence degree of the task content of the created task is identified to obtain the execution independence degree values ​​of all task contents under the created task;

[0020] Based on the execution independence value, the created task is finely split to obtain a number of subtasks; all subtasks are placed in the task pool of the portrait platform, and the status of all subtasks in the task pool is marked to distinguish whether each subtask is in an idle state or an occupied state.

[0021] Optionally, all subtasks in the task pool are marked with status to distinguish whether each subtask is in an idle state or an occupied state, including:

[0022] Scan all subtasks in the task pool to determine whether all subtasks in the current task pool are in an occupied state;

[0023] When the subtask is in an occupied state, marking the occupied state for the subtask;

[0024] When the subtask is in an idle state, mark it as an idle state;

[0025] Counting the occupied state flags and the idle state flags to obtain the corresponding numbers of the occupied state flags and the idle state flags;

[0026] When the number of occupied state marks exceeds the number of idle state marks, a secondary mark determination time interval is set;

[0027] The second marking determination time interval is obtained by the following formula:

[0028]

[0029] Wherein, T represents the time interval for determining the secondary mark; T0 represents the preset initial time interval; M represents the number of idle state marks; N represents the number of occupied state marks; n represents the number of secondary marks that appear in the historical mark record; P gi represents the conversion ratio of occupied state marking to idle state marking at the i-th secondary marking in the historical marking record; P ki represents the conversion ratio of the idle state mark to the occupied state mark at the i-th secondary mark in the historical mark record; f represents the compensation adjustment coefficient, and the compensation adjustment coefficient is obtained by the following formula:

[0030]

[0031] Where, f represents the compensation adjustment coefficient; P gb represents the standard deviation of the conversion ratio from occupied state marking to idle state marking in n secondary markings; P kb represents the standard deviation of the conversion ratio from idle state marking to occupied state marking in n secondary markings; P gmax represents the maximum conversion ratio of occupied state marking to idle state marking in n secondary markings; P kmax represents the maximum conversion ratio of the idle state mark to the occupied state mark in n secondary marks; P gi represents the conversion ratio of occupied state marking to idle state marking at the i-th secondary marking in the historical marking record; P ki represents the conversion ratio of the idle state mark to the occupied state mark during the i-th secondary mark in the historical mark record; M max represents the number of idle state marks corresponding to the maximum value of the conversion ratio of the occupied state mark to the idle state mark in the n secondary marks; N represents the number of occupied state marks corresponding to the maximum value of the conversion ratio of the occupied state mark to the idle state mark in the n secondary marks;

[0032] When the second marking determination time interval ends, all subtasks in the task pool are re-scanned to determine whether all subtasks in the task pool at the time when the second marking determination time interval ends are in an occupied state;

[0033] The state corresponding to each subtask is re-determined, and the state mark of all subtasks in the task pool at the end of the second marking determination time interval is used as the final state mark.

[0034] Optionally, all subtasks in the task pool are identified to obtain weight feature information of each subtask in its corresponding task, including:

[0035] Obtain the execution time sequence information and the amount of data to be processed of all the idle subtasks in the task pool in their corresponding tasks; based on the execution time sequence information and the amount of data to be processed information, weigh all the idle subtasks in terms of their weights in the corresponding businesses, and obtain the weight ratio of each subtask in its corresponding task, which is used as the weight feature information.

[0036] Optionally, based on the weight feature information, a task execution queue for all subtasks is generated, including:

[0037] Based on the size relationship of the weight proportion values ​​of all the subtasks in the idle state in the task pool in their corresponding tasks, a task execution priority queue for all the subtasks in the idle state is generated.

[0038] Optionally, determining the execution task information of each of the processes under the computing resource pool of the image platform, and scheduling the subtasks under the task execution queue based on the execution task information, thereby allocating the subtasks to the corresponding processes, including:

[0039] Based on the usage status information of the logical computing resources, parallel computing resources and neural network acceleration computing resources in the computing resource pool of the profiling platform, determine the execution task load information of all processes under the computing resource pool;

[0040] Based on the execution task load information, determine the receivable processing task volume information of all processes under the logical computing resources, parallel computing resources and neural network acceleration computing resources; based on the receivable processing task volume information, schedule the subtasks under the task execution priority queue, so as to assign the subtasks to the corresponding processes.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The task scheduling method of the portrait platform under the large model architecture provided in the present application builds a complete picture platform, monitors the task triggering of all business links under the portrait platform, obtains the tasks created by the platform, and obtains the tasks generated by the platform in a timely and accurate manner; the created tasks are finely divided into several sub-tasks and placed in the task pool, and the weight feature information of all sub-tasks in the task pool in their corresponding tasks is identified to realize the identification and measurement of the importance and urgency of the sub-tasks; based on the weight feature information, a task execution queue for all sub-tasks is generated to provide a basis for the subsequent determination of the task scheduling order; the execution task information of all processes under the computing resource pool of the portrait platform is also determined, and the sub-tasks under the task execution queue are scheduled and processed, and the sub-tasks are assigned to the corresponding processes. By introducing fine-grained task splitting and task scheduling in the portrait platform, the computing resources of the portrait platform are fully and efficiently utilized for task processing, avoiding the problems of poor availability and scalability of the portrait platform due to tight computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0044] Figure 1 A flow chart of the task scheduling method of the portrait platform under the large model architecture provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] The terms "include" and "have" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, method, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] See also Figure 1 As shown, an embodiment of the present application provides a task scheduling method for a portrait platform under a large model architecture. The task scheduling method for a portrait platform under a large model architecture includes:

[0049] Based on the big model architecture, a complete portrait platform is built; task trigger monitoring is performed on all business links under the portrait platform to obtain the tasks created by the portrait platform;

[0050] The created task is finely divided into several subtasks and placed in the task pool of the portrait platform; all subtasks in the task pool are identified to obtain the weight feature information of each subtask in its corresponding task;

[0051] Based on the weight feature information, a task execution queue for all subtasks is generated; the execution task information of all processes under the computing resource pool of the portrait platform is determined, and based on the execution task information, the subtasks under the task execution queue are scheduled and processed, so as to assign the subtasks to the corresponding processes.

[0052] The beneficial effects of the above embodiments are as follows: the task scheduling method of the portrait platform under the large model architecture builds a complete picture platform, monitors task triggers for all business links under the portrait platform, obtains tasks created by the platform, and obtains tasks generated by the platform in a timely and accurate manner; the created tasks are finely divided into several subtasks and placed in a task pool, and the weight feature information of all subtasks in the task pool in their corresponding tasks is identified to achieve identification and measurement of the importance and urgency of the subtasks; based on the weight feature information, a task execution queue for all subtasks is generated to provide a basis for subsequent determination of the task scheduling order; the execution task information of all processes under the computing resource pool of the portrait platform is also determined, and the subtasks under the task execution queue are scheduled and processed, and the subtasks are assigned to the corresponding processes. By introducing fine-grained task splitting and task scheduling in the portrait platform, the computing resources of the portrait platform are fully and efficiently utilized for task processing, avoiding the problems of poor availability and scalability of the portrait platform due to tight computing resources.

[0053] In another embodiment, a complete portrait platform is constructed based on the large model architecture, including:

[0054] Obtain the operating mode and operating performance requirements of the portrait platform to be built, and select a matching large model architecture based on the operating mode and the operating performance requirements;

[0055] Based on this matching large model architecture, a complete portrait platform is built.

[0056] The beneficial effect of the above embodiment is that the portrait platform is used to perform analysis operations on different types of portrait data for different objects, so that the analysis models and performance requirements corresponding to different portrait data analysis scenarios of the portrait platform are different. In order to meet the needs of different portrait data analysis scenarios and build a portrait platform that matches the portrait data analysis scenario in the fastest and most efficient way, first obtain the operation mode and operation performance requirements of the portrait platform to be built, where the operation mode may be but not limited to the type of behavior pattern of the portrait platform analyzing the portrait data, and the operation performance requirements may be but not limited to the time and accuracy corresponding to the portrait platform analyzing the portrait data. Based on the operation mode and the operation performance requirements, select the most matching large model architecture from different large model architectures as the basic architecture for building the portrait platform. Then, based on the selected matching large model architecture, adjust the large model architecture to build a complete portrait platform, thereby ensuring that the constructed portrait platform can meet the needs of subsequent actual portrait data analysis.

[0057] In another embodiment, the task triggering monitoring is performed on the portrait platform to obtain the task created by the portrait platform, including:

[0058] Monitor all business links under the portrait platform to obtain the task trigger status information of all business links; wherein, the task trigger status information includes the platform user task trigger request information and the preceding task trigger request information;

[0059] Based on the task trigger status information, the tasks created by the portrait platform in the process of executing all business links are obtained.

[0060] The beneficial effects of the above embodiments are that the portrait platform needs to run a variety of different business links during operation, among which the main business links include but are not limited to resource readiness detection, crowd creation, portrait calculation, portrait analysis report generation, crowd data storage and crowd post-operation. The above business links belong to the conventional business links of the portrait platform, and the specific operation process of the above business links will not be described in detail. In the actual operation of the portrait platform, when an exception occurs in one of the above business links and needs to be re-executed, the process corresponding to all the above business links will be executed again from the beginning as a whole, resulting in increased consumption of computing resources and the output of the portrait data analysis results. In addition, in the corresponding scenario, a single process needs to concentrate resources to calculate all tasks in the process. The probability of abnormality in the calculation process is high, and the execution priority of each task between different processes cannot be controlled. When there are high-priority groups, the output results cannot be guaranteed on time. In addition, the process is used as the computing unit of the portrait platform, and the computing granularity is large. It is difficult to achieve linear improvement of computing power through expansion in engineering, resulting in poor availability and poor scalability of the portrait platform. For this reason, according to the task creation of the portrait platform, the tasks triggered by the portrait platform are accurately identified and the created tasks are obtained in time, providing a basis for subsequent task scheduling. Specifically, the scenarios for triggering the creation of tasks on the portrait platform mainly include business function triggering and pre-task triggering; among them, business function triggering refers to the triggering of the business function of the portrait platform, such as different types of business such as platform users creating a crowd, configuring crowd splitting, and clicking on crowd downloads; pre-task triggering refers to the triggering of the creation of subsequent tasks after the completion of the pre-task, such as the generation of portrait reports depends on the portrait calculation task, and the task of generating portrait reports can be automatically triggered when the portrait calculation task is completed. By monitoring all business links under the portrait platform, the platform user task trigger request information and pre-task trigger request information of all business links are obtained, so as to fully obtain the tasks triggered by the portrait platform in the corresponding trigger creation task scenario, and avoid the situation of missing task identification. In addition, based on the task trigger status information, the tasks created by the portrait platform in the process of executing all business links are obtained to ensure the accuracy and reliability of the execution of subsequent task scheduling.

[0061] In another embodiment, all business links under the portrait platform are monitored to obtain task triggering status information of all business links, including:

[0062] Monitor the business links of resource readiness detection, crowd creation, portrait calculation, portrait analysis report generation, crowd data storage and crowd post-operation on the portrait platform, and obtain the request information from platform users and the execution status information of the preceding tasks during the execution of each business link;

[0063] Based on the request information from the platform user and the execution status information of the preceding task, the task triggering status information of each business link is obtained.

[0064] The beneficial effects of the above embodiments are that the business links executed by the portrait platform, such as resource readiness detection, crowd creation, portrait calculation, portrait analysis report generation, crowd data storage and crowd post-operation, contain different tasks for different objects, and all the above business links are monitored to obtain the request information from platform users and the execution status information of the preceding tasks during the execution of each business link. In this way, task trigger creation monitoring is performed on all business links for the two scenarios of triggering task creation, namely business function triggering and preceding task triggering, to ensure that the task triggering status information of each business link is accurately obtained.

[0065] In another embodiment, the created task is finely divided to obtain a number of subtasks and placed in the task pool of the portrait platform, including:

[0066] Based on the K-means++ algorithm, the execution independence degree of the task content of the created task is identified, and the execution independence degree values ​​of all task contents under the created task are obtained;

[0067] Based on the execution independence value, the created task is finely split into several subtasks; all subtasks are placed in the task pool of the portrait platform, and the status of all subtasks in the task pool is marked to distinguish whether each subtask is in an idle state or an occupied state.

[0068] The beneficial effect of the above embodiment is that in actual operation, when a task is successfully created, its execution mode can adopt the method of multi-process task preemption and concurrent execution, wherein the newly added task can enter different task execution queues according to its priority and concurrency control mechanism, and the process in the computing resource pool of the portrait platform obtains the task information to be executed through task preemption. Each process is only responsible for the tasks it needs to execute and updates the task status in time. The complexity of different tasks created is different. In order to accurately refine the tasks, the created tasks can be fine-grained. Here, the K-means++ algorithm is preferably used to perform fine-grained splitting of the created tasks with respect to the process. Specifically, the K-means++ algorithm mainly includes:

[0069] Step 1: Randomly select a center u among the data points included in the task i ;

[0070] Step 2: For each data point x that has not yet been selected, calculate That is, the distance between x and the closest center that has been selected, where j is the number of centers;

[0071] Step 3: Use the weighted probability distribution to randomly select a new data point as the new center, where the probability of the selected data point x is the same as is proportional to;

[0072] Step 4, repeat the above steps 2 and 3 until K centers are selected, i.e. j = K;

[0073] Step 5: Based on the selected initial centers, continue to use standard K-means clustering.

[0074] Through the above process, the execution independence degree of the created task is identified, and the execution independence degree values ​​of all the task contents under the created task are obtained, and the mutual dependence degree of all the task contents under the created task is quantified and determined; generally speaking, the larger the execution independence degree value, the less dependent the corresponding task content is on other task contents, and the easier it is to separate it from the created task for independent processing. In addition, based on the execution independence degree value, the created task is finely split to obtain several subtasks; all subtasks are placed in the task pool of the portrait platform, and the status of all subtasks in the task pool is marked to distinguish whether each subtask is in an idle state or an occupied state, so as to facilitate the subsequent task scheduling for the idle subtasks.

[0075] In another embodiment, all subtasks in the task pool are marked with status to distinguish whether each subtask is in an idle state or an occupied state, including:

[0076] Scan all subtasks in the task pool to determine whether all subtasks in the current task pool are in an occupied state;

[0077] When the subtask is in an occupied state, marking the occupied state for the subtask;

[0078] When the subtask is in an idle state, mark it as an idle state;

[0079] Counting the occupied state flags and the idle state flags to obtain the corresponding numbers of the occupied state flags and the idle state flags;

[0080] When the number of occupied state marks exceeds the number of idle state marks, a secondary mark determination time interval is set;

[0081] The second marking determination time interval is obtained by the following formula:

[0082]

[0083] Wherein, T represents the time interval for determining the secondary mark; T0 represents the preset initial time interval; M represents the number of idle state marks; N represents the number of occupied state marks; n represents the number of secondary marks that appear in the historical mark record; P gi represents the conversion ratio of occupied state marking to idle state marking at the i-th secondary marking in the historical marking record; P ki represents the conversion ratio of the idle state mark to the occupied state mark at the i-th secondary mark in the historical mark record; f represents the compensation adjustment coefficient, and the compensation adjustment coefficient is obtained by the following formula:

[0084]

[0085] Where, f represents the compensation adjustment coefficient; P gb represents the standard deviation of the conversion ratio from occupied state marking to idle state marking in n secondary markings; P kb represents the standard deviation of the conversion ratio from idle state marking to occupied state marking in n secondary markings; P gmax represents the maximum conversion ratio of occupied state marking to idle state marking in n secondary markings; P kmax represents the maximum conversion ratio of the idle state mark to the occupied state mark in n secondary marks; P gi represents the conversion ratio of occupied state marking to idle state marking at the i-th secondary marking in the historical marking record; P ki represents the conversion ratio of the idle state mark to the occupied state mark during the i-th secondary mark in the historical mark record; M max represents the number of idle state marks corresponding to the maximum value of the conversion ratio of the occupied state mark to the idle state mark in the n secondary marks; N represents the number of occupied state marks corresponding to the maximum value of the conversion ratio of the occupied state mark to the idle state mark in the n secondary marks;

[0086] When the second marking determination time interval ends, all subtasks in the task pool are re-scanned to determine whether all subtasks in the task pool at the time when the second marking determination time interval ends are in an occupied state;

[0087] The state corresponding to each subtask is re-determined, and the state mark of all subtasks in the task pool at the end of the second marking determination time interval is used as the final state mark.

[0088] The beneficial effect of the above embodiment is that by scanning the subtasks in the task pool in real time and marking their states (idle or occupied), the system can quickly identify which resources are available and which are occupied, thereby optimizing resource allocation and improving task processing efficiency. According to the number of subtasks in the idle and occupied states in the current task pool, and the historical conversion ratio, the secondary marking determination time interval (T) is dynamically calculated. This method can automatically adjust the scanning frequency according to the real-time load of the task pool, avoid unnecessary frequent scanning, and reduce system overhead. By introducing the compensation adjustment coefficient (f), the scheme takes into account the standard deviation and maximum value of the historical conversion ratio, so that the system can more flexibly cope with sudden changes in task status. For example, when the task state changes frequently, the system can adjust the scanning interval faster to ensure the real-time nature of the resource state. Through accurate state marking and dynamic adjustment, the system can more effectively utilize idle resources, reduce resource idle time, and improve overall resource utilization. The scheme comprehensively considers the historical conversion ratio and the current state, and introduces statistics such as standard deviation and maximum value, so that the system can maintain more stable performance when facing task state fluctuations, and reduce the system instability caused by inaccurate state judgment. Since the solution can be dynamically adjusted according to historical data and current status, it is highly adaptable. Regardless of the size of the task pool, the type of task, or the execution frequency of the task, the system can maintain efficient operation by adjusting the scanning interval and status determination strategy.

[0089] In summary, this technical solution effectively improves the efficiency, flexibility and stability of task management, optimizes resource utilization, and enhances the adaptability of the system through measures such as real-time status marking, dynamic adjustment of judgment time interval, and introduction of compensation adjustment coefficient.

[0090] In another embodiment, all subtasks in the task pool are identified to obtain weight feature information of each subtask in its corresponding task, including:

[0091] Obtain the execution time sequence information and the amount of data to be processed of all idle subtasks in the task pool in their corresponding tasks. Based on the execution time sequence information and the amount of data to be processed information, weigh the weights of all idle subtasks in their corresponding businesses, and obtain the weight ratio of each subtask in its corresponding task, which is used as the weight feature information.

[0092] The beneficial effect of the above embodiment is to obtain the execution time sequence information and the amount of data to be processed of all the idle subtasks in the task pool in their corresponding tasks, so as to measure the weights of all the idle subtasks in their corresponding businesses, and obtain the weight ratio of each subtask in its corresponding task, wherein the larger the weight ratio of the subtask in the task, the greater the importance of the subtask in the task, and the higher the priority of the subtask in the task execution priority queue, which provides a basis for the subsequent generation of the task execution priority queue.

[0093] In another embodiment, based on the weight feature information, a task execution queue for all subtasks is generated, including:

[0094] Based on the size relationship of the weight proportion values ​​of all the subtasks in the idle state in the task pool in their corresponding tasks, a task execution priority queue for all the subtasks in the idle state is generated.

[0095] The beneficial effect of the above embodiment is that, based on the size relationship of the weight proportion values ​​of all subtasks in the task pool that are in an idle state in their corresponding tasks, a task execution priority queue for all subtasks in an idle state is generated, so that the task execution priority queue can comprehensively and accurately reflect the execution priority levels of all subtasks, providing an accurate basis for subsequent task scheduling.

[0096] In another embodiment, the execution task information of all processes under the computing resource pool of the image platform is determined, and based on the execution task information, the subtasks under the task execution queue are scheduled and processed, thereby allocating the subtasks to the corresponding processes, including:

[0097] Based on the usage status information of the logical computing resources, parallel computing resources and neural network acceleration computing resources in the computing resource pool of the profiling platform, determine the execution task load information of all processes under the computing resource pool;

[0098] Based on the execution task load information, determine the receivable processing task volume information of all processes under the logical computing resources, parallel computing resources and neural network acceleration computing resources; based on the receivable processing task volume information, schedule the subtasks under the task execution priority queue, so as to assign the subtask to the corresponding process.

[0099] The beneficial effect of the above embodiment is that the computing resource pool of the portrait platform includes different types of computing resources such as logical computing resources, parallel computing resources and neural network acceleration computing resources. In order to accurately quantify and allocate all computing resources within the computing resource pool, based on the usage status information corresponding to the logical computing resources, parallel computing resources and neural network acceleration computing resources in the computing resource pool of the portrait platform, the execution task load information of all processes under the computing resource pool is determined, and the execution task load information includes the task processing load that each process has currently received. In addition, based on the execution task load information, the receivable processing task volume information of all processes under the logical computing resources, parallel computing resources and neural network acceleration computing resources is determined, so as to schedule and process the subtasks under the task execution priority queue, thereby allocating the subtasks to the corresponding processes. By introducing fine-grained task splitting and task scheduling in the portrait platform, the computing resources of the portrait platform are fully and efficiently utilized for task processing, avoiding the problem of poor availability and scalability of the portrait platform due to tight computing resources.

[0100] In general, the task scheduling method of the portrait platform under the large model architecture builds a complete picture platform, monitors the task triggering of all business links under the portrait platform, obtains the tasks created by the platform, and obtains the tasks generated by the platform in a timely and accurate manner; the created tasks are fine-grainedly split into several sub-tasks and placed in the task pool, and the weight feature information of all sub-tasks in the task pool in their corresponding tasks is identified to realize the identification and measurement of the importance and urgency of the sub-tasks; based on the weight feature information, a task execution queue for all sub-tasks is generated to provide a basis for the subsequent determination of the task scheduling order; the execution task information of all processes under the computing resource pool of the portrait platform is determined, and the sub-tasks under the task execution queue are scheduled and processed, and the sub-tasks are assigned to the corresponding processes. By introducing fine-grained task splitting and task scheduling in the portrait platform, the computing resources of the portrait platform are fully and efficiently utilized for task processing, avoiding the problems of poor availability and scalability of the portrait platform due to tight computing resources.

[0101] The above is only a specific implementation of the present invention, and any other improvements made based on the concept of the present invention are considered to be within the protection scope of the present invention.

Claims

1. The task scheduling method of the portrait platform under the large model architecture is characterized by: include: Build a complete portrait platform based on the big model architecture; Monitor task triggering of all business links under the portrait platform to obtain tasks created by the portrait platform; Fine-grained splitting of the created task to obtain a number of subtasks and placing them into the task pool of the portrait platform; Identify all subtasks in the task pool to obtain weight feature information of each subtask in its corresponding task; Based on the weight feature information, a task execution queue for all subtasks is generated; the execution task information of all processes under the computing resource pool of the portrait platform is determined, and based on the execution task information, the subtasks under the task execution queue are scheduled and processed, so as to assign the subtasks to the corresponding processes.

2. The task scheduling method of the portrait platform under the large model architecture as claimed in claim 1 is characterized by: Based on the big model architecture, a complete portrait platform is built, including: Obtain the operation mode and operation performance requirements of the portrait platform to be built, and select a matching large model architecture based on the operation mode and the operation performance requirements; Based on the matching large model architecture, a complete portrait platform is constructed.

3. The task scheduling method of the portrait platform under the large model architecture as described in claim 2 is characterized by: The task triggering monitoring is performed on the portrait platform to obtain the task created by the portrait platform, including: Monitor all business links under the portrait platform to obtain task trigger status information of all business links; wherein the task trigger status information includes platform user task trigger request information and pre-task trigger request information; Based on the task triggering status information, the tasks created by the portrait platform during the execution of all business links are obtained.

4. The task scheduling method of the portrait platform under the large model architecture as described in claim 3 is characterized by: Monitor all business links under the portrait platform to obtain task triggering status information of all business links, including: Monitor the business links of resource readiness detection, crowd creation, portrait calculation, portrait analysis report generation, crowd data storage and crowd post-operation on the portrait platform, and obtain the request information from platform users and the execution status information of the pre-task during the execution of each business link; Based on the request information from the platform user and the preceding task execution status information, task triggering status information of each business link is obtained.

5. The task scheduling method of the portrait platform under the large model architecture as claimed in claim 1 is characterized by: The created task is finely divided to obtain several subtasks and placed in the task pool of the portrait platform, including: Based on the K-means++ algorithm, the execution independence degree of the task content of the created task is identified to obtain the execution independence degree values ​​of all task contents under the created task; based on the execution independence degree values, the created task is fine-grainedly split to obtain a number of subtasks; all subtasks are placed in the task pool of the portrait platform, and the status of all subtasks in the task pool is marked to distinguish whether each subtask is in an idle state or an occupied state.

6. The task scheduling method of the portrait platform under the large model architecture as claimed in claim 5 is characterized by: Mark the status of all subtasks in the task pool to distinguish whether each subtask is in an idle state or an occupied state, including: Scan all subtasks in the task pool to determine whether all subtasks in the current task pool are in an occupied state; When the subtask is in an occupied state, marking the occupied state for the subtask; When the subtask is in an idle state, mark it as an idle state; Counting the occupied state flags and the idle state flags to obtain the corresponding numbers of the occupied state flags and the idle state flags; When the number of occupied state marks exceeds the number of idle state marks, a secondary mark determination time interval is set; The second marking determination time interval is obtained by the following formula: Wherein, T represents the time interval for determining the secondary mark; T0 represents the preset initial time interval; M represents the number of idle state marks; N represents the number of occupied state marks; n represents the number of secondary marks that appear in the historical mark record; P gi represents the conversion ratio of occupied state marking to idle state marking at the i-th secondary marking in the historical marking record; P ki represents the conversion ratio of the idle state mark to the occupied state mark at the i-th secondary mark in the historical mark record; f represents the compensation adjustment coefficient, and the compensation adjustment coefficient is obtained by the following formula: Where, f represents the compensation adjustment coefficient; P gb represents the standard deviation of the conversion ratio from occupied state marking to idle state marking in n secondary markings; P kb represents the standard deviation of the conversion ratio from idle state marking to occupied state marking in n secondary markings; P gmax represents the maximum conversion ratio of occupied state marking to idle state marking in n secondary markings; P kmax represents the maximum conversion ratio of the idle state mark to the occupied state mark in n secondary marks; P gi represents the conversion ratio of occupied state marking to idle state marking at the i-th secondary marking in the historical marking record; P ki represents the conversion ratio of the idle state mark to the occupied state mark during the i-th secondary mark in the historical mark record; M max represents the number of idle state marks corresponding to the maximum value of the conversion ratio of the occupied state mark to the idle state mark in the n secondary marks; N represents the number of occupied state marks corresponding to the maximum value of the conversion ratio of the occupied state mark to the idle state mark in the n secondary marks; When the second marking determination time interval ends, all subtasks in the task pool are re-scanned to determine whether all subtasks in the task pool at the time when the second marking determination time interval ends are in an occupied state; The state corresponding to each subtask is re-determined, and the state mark of all subtasks in the task pool at the end of the second marking determination time interval is used as the final state mark.

7. The task scheduling method of the portrait platform under the large model architecture as claimed in claim 5 is characterized by: All subtasks in the task pool are identified to obtain weight feature information of each subtask in its corresponding task, including: Obtain the execution time sequence information and the amount of data to be processed of all the idle subtasks in the task pool in their corresponding tasks; based on the execution time sequence information and the amount of data to be processed information, weigh all the idle subtasks in terms of their weights in the corresponding businesses, and obtain the weight ratio of each subtask in its corresponding task, which is used as the weight feature information.

8. The task scheduling method of the portrait platform under the large model architecture as claimed in claim 6 is characterized by: Based on the weight feature information, a task execution queue for all subtasks is generated, including: Based on the size relationship of the weight proportion values ​​of all the subtasks in the idle state in the task pool in their corresponding tasks, a task execution priority queue for all the subtasks in the idle state is generated.

9. The task scheduling method of the portrait platform under the large model architecture as claimed in claim 7 is characterized by: Determine the execution task information of all processes under the computing resource pool of the image platform, and schedule the subtasks under the task execution queue based on the execution task information, so as to assign the subtasks to the corresponding processes, including: Based on the usage status information of the logical computing resources, parallel computing resources and neural network acceleration computing resources in the computing resource pool of the profiling platform, determine the execution task load information of all processes under the computing resource pool; Based on the execution task load information, determine the receivable processing task volume information of all processes under the logical computing resources, parallel computing resources and neural network acceleration computing resources; based on the receivable processing task volume information, schedule the subtasks under the task execution priority queue, so as to assign the subtasks to the corresponding processes.

Images