Task execution method and device and storage medium

Abstract

The invention discloses a task execution method and device and a storage medium, and the method comprises the steps: determining a plurality of target data processing tasks according to business data of different data types in a business data flow, and obtaining the operation processing time of each target data processing task; determining the resource allocation proportion of each target data processing task according to the operation processing time, determining the task processing priority of each target data processing task according to the operation processing time and the resource allocation proportion, and then determining the task processing priority of each target data processing task according to the task processing priority and the resource allocation proportion of each target data processing task. And performing resource allocation on each target data processing task, and performing task execution on each target data processing task subjected to resource allocation. According to the embodiment of the invention, resource allocation among the data processing tasks can be dynamically adjusted, so that the influence of unbalanced resource allocation on the data processing tasks can be reduced. The embodiment of the invention can be applied to various application scenes such as game data processing.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular, to a task execution method, apparatus, and storage medium. Background Art

[0002] In the related art, when processing service data, generally, ETL (Extract-Transform-Load) processing of data extraction, cleaning transformation, and loading into a data warehouse is performed on the service data, so that the data in the data warehouse can be used to provide an analysis basis for system decision-making during subsequent processing. Among them, when performing ETL processing on service data, the system allocates corresponding resources to each data processing task so that each data processing task can be effectively executed.

[0003] Currently, when the system allocates resources to each data processing task, in order to avoid resource waste, a resource forced isolation method is often used to allocate resources to each data processing task, so that the resources used by each data processing task do not exceed the range of resources it has promised or reserved. However, in some complex business scenarios, there is a correlation between different data processing tasks and it is necessary to dynamically adjust the resource allocation between data processing tasks. However, the resource allocation method of resource forced isolation is relatively rigid and it is difficult to dynamically adjust the resource allocation between data processing tasks, which may cause some data processing tasks to be affected by uneven resource allocation. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail in this document. This overview is not intended to limit the scope of protection of the claims.

[0005] Embodiments of the present application provide a task execution method, apparatus, and storage medium, which can dynamically adjust the resource allocation between data processing tasks, thereby reducing the impact of uneven resource allocation on data processing tasks.

[0006] On the one hand, embodiments of the present application provide a task execution method, including the following steps:

[0007] Determine a plurality of target data processing tasks according to service data of different data types in the service data stream;

[0008] Obtain the running processing time of each of the target data processing tasks;

[0009] Determine the resource allocation ratio of each of the target data processing tasks according to the running processing time of each of the target data processing tasks;

[0010] Determine the task processing priority of each of the target data processing tasks according to the running processing time and the resource allocation ratio of each of the target data processing tasks;

[0011] Allocate resources to each of the target data processing tasks according to the task processing priority and the resource allocation ratio of each of the target data processing tasks;

[0012] Execute the tasks of each of the target data processing tasks for which resources have been allocated.

[0013] On the other hand, an embodiment of the present application further provides a task execution device, including:

[0014] A task determination unit, configured to determine a plurality of target data processing tasks according to service data of different data types in a service data stream;

[0015] A time acquisition unit, configured to acquire the running processing time of each of the target data processing tasks;

[0016] A ratio determination unit, configured to determine the resource allocation ratio of each of the target data processing tasks according to the running processing time of each of the target data processing tasks;

[0017] A priority determination unit, configured to determine the task processing priority of each of the target data processing tasks according to the running processing time and the resource allocation ratio of each of the target data processing tasks;

[0018] A resource allocation unit, configured to allocate resources to each of the target data processing tasks according to the task processing priority and the resource allocation ratio of each of the target data processing tasks;

[0019] A task execution unit, configured to execute the tasks of each of the target data processing tasks for which resources have been allocated.

[0020] Optionally, the ratio determination unit is further configured to:

[0021] Perform matrix division on the running processing time of each of the target data processing tasks to obtain a data matrix and a query matrix of the running processing time of each of the target data processing tasks;

[0022] Determine the resource allocation ratio of each of the target data processing tasks according to the data matrix and the query matrix of the running processing time of each of the target data processing tasks.

[0023] Optionally, the ratio determination unit is further configured to:

[0024] For each of the target data processing tasks, perform a multiplication and summation operation on the data matrix of the running processing time of the current target data processing task, the query matrix of the running processing time of the current target data processing task, and the query matrices of the running processing times of the other target data processing tasks, to obtain the resource allocation ratio for each target data processing task.

[0025] Optionally, the ratio determination unit is further configured to:

[0026] Perform matrix processing on the running processing time of each target data processing task to obtain the running processing time matrix of each target data processing task;

[0027] According to the target segmentation ratio, perform matrix segmentation on the running processing time matrix of each target data processing task to obtain the data matrix and query matrix of the running processing time of each target data processing task.

[0028] Optionally, the priority determination unit is further configured to:

[0029] Multiply the running processing time and the resource allocation ratio of each target data processing task to obtain the task processing priority of each target data processing task.

[0030] Optionally, the time acquisition unit is further configured to:

[0031] Obtain the total running processing time of all data processing tasks corresponding to the service data stream;

[0032] Determine the running processing time ratio of each target data processing task;

[0033] Calculate the running processing time of each target data processing task based on the total running processing time and the running processing time ratio.

[0034] Optionally, the running processing time, the resource allocation ratio, and the task processing priority are all obtained according to a trained multi-modal resource allocation model. The task execution device further includes a model training unit, and the model training unit is used to:

[0035] Obtain a plurality of data processing task samples;

[0036] Call the multi-modal resource allocation model to calculate the processing time of each data processing task sample and assign a task priority coefficient to each data processing task sample;

[0037] Invoke the multi-modal resource allocation model, and calculate the first task priority for each data processing task sample according to the processing time and the task priority coefficient of each data processing task sample.

[0038] Invoke the multi-modal resource allocation model, and determine the resource ratio for each data processing task sample according to the processing time of each data processing task sample.

[0039] Invoke the multi-modal resource allocation model, and determine the second task priority for each data processing task sample according to the processing time and the resource ratio of each data processing task sample.

[0040] Adjust the model parameters of the multi-modal resource allocation model according to the second task priority and the first task priority.

[0041] Optionally, the task execution device further includes a first resource reallocation unit, and the first resource reallocation unit is used for:

[0042] When it is detected that any one of the multiple target data processing tasks has changed, obtain the new running processing time of each target data processing task.

[0043] Determine the new resource allocation ratio for each target data processing task according to the new running processing time of each target data processing task.

[0044] Determine the new task processing priority for each target data processing task according to the new running processing time and the new resource allocation ratio of each target data processing task.

[0045] Re-allocate resources and execute tasks for each target data processing task according to the new task processing priority and the new resource allocation ratio.

[0046] Optionally, that any one of the multiple target data processing tasks has changed includes:

[0047] Any one of the multiple target data processing tasks has changed in task status;

[0048] Or, any one of the multiple target data processing tasks has changed in task content.

[0049] Optionally, the first resource reallocation unit is further used for:

[0050] Obtain the current resource utilization rate;

[0051] Determine the new resource allocation ratio for each of the target data processing tasks according to the resource utilization rate and the new running processing time of each of the target data processing tasks.

[0052] Optionally, the task execution unit is further configured to:

[0053] Determine a first data processing task to be preferentially executed among the multiple target data processing tasks for which resources have been allocated according to the task processing priority;

[0054] Execute the first data processing task, and during the execution of the first data processing task, perform preprocessing on other target data processing tasks among the multiple target data processing tasks;

[0055] After the execution of the first data processing task is completed, perform task execution on the other target data processing tasks that have undergone preprocessing.

[0056] On the other hand, an embodiment of the present application further provides an electronic device, including:

[0057] At least one processor;

[0058] At least one memory for storing at least one program;

[0059] When at least one of the programs is executed by at least one of the processors, the task execution method described above is implemented.

[0060] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program executable by a processor is stored, and when the computer program executable by the processor is executed by the processor, the task execution method described above is implemented.

[0061] On the other hand, an embodiment of the present application further provides a computer program product, including a computer program or computer instructions, the computer program or the computer instructions are stored in a computer-readable storage medium, a processor of an electronic device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the electronic device executes the task execution method described above.

[0062] The embodiments of the present application at least include the following beneficial effects: First, determine multiple target data processing tasks according to the service data of different data types in the service data stream, and obtain the running processing time of each target data processing task; then, determine the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task; for the target data processing task with a long running processing time, it requires more resources, while for the target data processing task with a short running processing time, it requires fewer resources. Therefore, the resource allocation ratio of the target data processing task can be determined more accurately according to the running processing time of the target data processing task, so that more accurate resources can be allocated to the target data processing task according to the resource allocation ratio; then, determine the task processing priority of each target data processing task according to the running processing time and resource allocation ratio of each target data processing task; for the target data processing task with a longer running processing time and a larger resource allocation ratio, its importance is higher, while for the target data processing task with a shorter running processing time and a smaller resource allocation ratio, its importance is lower. Therefore, the task processing priority of the target data processing task can be determined more accurately according to the running processing time and resource allocation ratio of the target data processing task; then, allocate resources to each target data processing task according to the task processing priority and resource allocation ratio, and execute the tasks of each target data processing task that has been allocated resources; since the running processing time of the target data processing task will change dynamically as the task is executed, the resource allocation ratio and task processing priority of the target data processing task will also change dynamically. Therefore, the resource allocation for the target data processing task according to the task processing priority and resource allocation ratio will also change dynamically. That is to say, the technical solution provided by the embodiments of the present application can dynamically adjust the resource allocation between data processing tasks, thereby reducing the impact of uneven resource allocation on data processing tasks.

[0063] Other features and advantages of the present application will be described in the following specification, and some of them will become obvious from the specification, or be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the specification and the drawings. Brief Description of the Drawings

[0064] The drawings are used to provide a further understanding of the technical solution of the present application, and constitute a part of the specification. They are used together with the embodiments of the present application to explain the technical solution of the present application, and do not constitute a limitation to the technical solution of the present application.

[0065] Figure 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application;

[0066] Figure 2 It is a schematic diagram of another implementation environment provided by an embodiment of the present application;

[0067] Figure 3 It is a flowchart of a task execution method provided by an embodiment of the present application;

[0068] Figure 4 It is a schematic flowchart of matrix segmentation for running processing time provided by an embodiment of the present application;

[0069] Figure 5 It is a schematic flowchart of calculating the resource allocation ratio of a target data processing task provided by an embodiment of the present application;

[0070] Figure 6 It is a schematic diagram of the structure of a multimodal resource allocation model provided by an embodiment of the present application;

[0071] Figure 7 It is a schematic system flowchart of a task execution method provided by a specific example of the present application;

[0072] Figure 8 It is a schematic system step diagram of a task execution method provided by a specific example of the present application;

[0073] Figure 9 It is a detailed flowchart of a task execution method provided by a specific example of the present application;

[0074] Figure 10 It is a schematic diagram of a task execution device provided by an embodiment of the present application;

[0075] Figure 11 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0076] The present application will be further described below in conjunction with the accompanying drawings of the specification and specific embodiments. The described embodiments should not be regarded as limitations of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0077] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0079] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are explained, and the nouns and terms involved in the embodiments of this application are applicable to the following explanations.

[0080] 1) Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable machines to have the functions of perception, reasoning, and decision-making. Artificial intelligence technology is an interdisciplinary subject involving a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models, also known as large models or foundation models, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0081] 2) A data warehouse is a structured data environment for decision support systems and online analytical application data sources. A data warehouse is mainly used to solve the problem of efficiently obtaining information from messy business databases.

[0082] 3) ETL (Extract-Transform-Load) is the process of extracting, cleaning, and transforming the data of a business system and then loading it into a data warehouse. Its purpose is to integrate the scattered, messy, and non-uniform data in the business together to provide an analysis basis for the decision-making of the business system.

[0083] 4) A multi-modal large model can process and analyze various different types of data such as text, images, voices, rendering data, physical simulation data, and AI computing data in games, etc., and output the obtained results in different modal forms, so as to achieve the collaborative processing of heterogeneous modal data.

[0084] For the solution in the related art that allocates resources for each data processing task by means of resource mandatory isolation, since this solution will ensure that the resources used by each data processing task do not exceed the scope of the resources it has committed or reserved, there is a problem of insufficient dynamic adaptability in this solution, and it is difficult to flexibly adapt to the dynamic changes of business data load. For example, in a complex game scenario, there are often various structures of business data. The insufficient dynamic adaptability of this solution may lead to its inability to fully utilize system resources, thus affecting the computing efficiency of data. Moreover, due to the relatively rigid resource allocation method of this solution, it is difficult to dynamically adjust the resource allocation between data processing tasks, which may cause some data processing tasks to be affected by unbalanced resource allocation. To solve the problem of insufficient dynamic adaptability, a solution that can use multi-threading for parallel computing has been proposed in the related art. This solution takes advantage of the multi-core processor, allocates some computing tasks to different threads for parallel processing, and adjusts parameters such as the Central Processing Unit (CPU) and memory according to prior knowledge to improve the overall performance. However, this solution can only improve the overall performance by adjusting parameters such as the CPU and memory according to prior knowledge, lacks an intelligent decision-making mechanism, and is difficult to make dynamic adjustments to intelligent resource allocation based on complex business data and task status, resulting in limitations in the optimization effect.

[0085] In order to dynamically adjust the resource allocation among data processing tasks to reduce the impact of uneven resource allocation on data processing tasks, the embodiments of the present application provide a task execution method, a task execution device, an electronic device, a computer-readable storage medium, and a computer program product. First, multiple target data processing tasks are determined according to the service data of different data types in the service data stream, and the running processing time of each target data processing task is obtained; then, according to the running processing time of each target data processing task, the resource allocation ratio of each target data processing task is determined; for a target data processing task with a long running processing time, more resources are required, while for a target data processing task with a short running processing time, fewer resources are required. Therefore, the resource allocation ratio of the target data processing task can be more accurately determined according to the running processing time of the target data processing task, so that more accurate resources can be allocated to the target data processing task according to the resource allocation ratio; then, according to the running processing time and resource allocation ratio of each target data processing task, the task processing priority of each target data processing task is determined; for a target data processing task with a longer running processing time and a larger resource allocation ratio, its importance is higher, while for a target data processing task with a shorter running processing time and a smaller resource allocation ratio, its importance is lower. Therefore, the task processing priority of the target data processing task can be more accurately determined according to the running processing time and resource allocation ratio of the target data processing task; then, according to the task processing priority and resource allocation ratio, resource allocation is performed on each target data processing task, and task execution is performed on each target data processing task that has undergone resource allocation; since the running processing time of the target data processing task will change dynamically as the target data processing task is executed, the resource allocation ratio and task processing priority of the target data processing task will also change dynamically. Therefore, the resource allocation performed on the target data processing task according to the task processing priority and resource allocation ratio will also change dynamically. That is to say, the technical solution provided by the embodiments of the present application can dynamically adjust the resource allocation among data processing tasks, thereby reducing the impact of uneven resource allocation on data processing tasks.

[0086] Figure 1 It is a schematic diagram of an implementation environment provided by the embodiments of the present application. Refer to Figure 1 , this implementation environment includes a first user terminal 101 and a first server 102. The first user terminal 101 and the first server 102 are directly or indirectly connected through a wired or wireless communication method. Among them, the first user terminal 101 and the first server 102 can be nodes in the blockchain, and this embodiment does not make specific limitations on this.

[0087] The first user terminal 101 may include, but is not limited to, intelligent devices such as smart phones, computers, intelligent voice interaction devices, intelligent home appliances, vehicle-mounted terminals, and aircraft. Optionally, the first user terminal 101 may be installed with a service operation client. When the service operation client is in an operating state, the service operation client may generate and send a service data stream to the first server 102. Among them, the service data stream may include multiple service data with different data types. For example, these service data may be in-game text data, image data, voice data, rendering data, physical simulation data, or AI computing data, etc., which are not specifically limited here. In addition, the service operation client may be a game client, a social media client, or a vehicle-mounted client, etc., which are not specifically limited here.

[0088] The first server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0089] In one embodiment, the first server 102 can determine multiple target data processing tasks according to the service data of different data types in the received service data stream, obtain the running processing time of each target data processing task, and then determine the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task. Then, according to the running processing time and resource allocation ratio of each target data processing task, determine the task processing priority of each target data processing task. Next, according to the task processing priority and resource allocation ratio of each target data processing task, allocate resources to each target data processing task, and execute the tasks of each target data processing task for which resources have been allocated.

[0090] Refer to Figure 1As shown, in an application scenario, it is assumed that the first user terminal 101 is a smart phone, and the first user terminal 101 is installed with a service operation client. When the user performs a service operation through the service operation client in the first user terminal 101, in response to the service operation performed by the user in the service operation client, the service operation client generates a corresponding service data stream. Among them, the service data stream may include multiple service data with different data types. After generating the corresponding service data stream, the first user terminal 101 sends the service data stream to the first server 102; in response to receiving the service data stream, the first server 102 determines multiple target data processing tasks according to the service data with different data types in the service data stream, and obtains the running processing time of each target data processing task. Then, according to the running processing time of each target data processing task, the resource allocation ratio of each target data processing task is determined. Then, according to the running processing time and resource allocation ratio of each target data processing task, the task processing priority of each target data processing task is determined. After determining the resource allocation ratio and task processing priority of each target data processing task, the first server 102 allocates resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task, and then executes the tasks of each target data processing task with resources allocated.

[0091] Figure 2 is a schematic diagram of another implementation environment provided by the embodiments of the present application. Refer to Figure 2 , this implementation environment includes a second user terminal 201, a second server 202, and a resource database 203. The second user terminal 201 and the second server 202 are directly or indirectly connected through a wired or wireless communication method. The resource database 203 can be set separately, integrated on the second server 202, or integrated on other devices. Among them, the second user terminal 201, the second server 202, and the resource database 203 can all be nodes in the blockchain. This embodiment does not make specific limitations on this.

[0092] The second user terminal 201 may include, but is not limited to, intelligent devices such as smart phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, and aircraft. Optionally, the second user terminal 201 may be installed with a service operation client. When the service operation client is in an operating state, the service operation client may generate and send a service data stream to the second server 202, where the service data stream may include multiple service data with different data types. For example, these service data may be in-game text data, image data, voice data, rendering data, physical simulation data, or AI computing data, etc., which are not specifically limited herein. Additionally, the service operation client may be a game client, a social media client, or a vehicle-mounted client, etc., which are not specifically limited herein.

[0093] The second server 202 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN networks, and big data and artificial intelligence platforms. Among them, various data computing resources may be stored in the resource database 203, and the resource database 203 can respond to the call of the second server 202 and provide various data computing resources to the second server 202.

[0094] In one embodiment, the second server 202 can determine multiple target data processing tasks according to the service data of different data types in the received service data stream, obtain the running processing time of each target data processing task, then determine the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, and then determine the task processing priority of each target data processing task according to the running processing time and resource allocation ratio of each target data processing task. Then, according to the task processing priority and resource allocation ratio of each target data processing task, resource allocation is performed on each target data processing task, and task execution is performed on each target data processing task that has been allocated resources.

[0095] Refer to Figure 2As shown, in another application scenario, it is assumed that the second user terminal 201 is a computer, and the second user terminal 201 is installed with a business operation client. When the user performs a business operation through the business operation client in the second user terminal 201, the business operation client generates a corresponding business data stream in response to the business operation performed by the user in the business operation client, wherein the business data stream may include multiple business data with different data types. After generating the corresponding business data stream, the second user terminal 201 sends the business data stream to the second server 202; in response to receiving the business data stream, the second server 202 determines multiple target data processing tasks based on the business data of different data types in the business data stream, and obtains the running processing time of each target data processing task, and then determines the resource allocation of each target data processing task based on the running processing time of each target data processing task. proportion, and then determine the task processing priority of each target data processing task according to the running processing time and resource allocation proportion of each target data processing task. After determining the resource allocation proportion and task processing priority of each target data processing task, the second server 202 sends a resource acquisition instruction to the resource database 203 according to the task processing priority and resource allocation proportion of each target data processing task; in response to receiving the resource acquisition instruction, the resource database 203 sends the corresponding resources to the second server 202; in response to receiving the resources sent by the resource database 203, the second server 202 allocates resources to each target data processing task according to these resources, and then performs task execution on each target data processing task to which resources have been allocated.

[0096] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing on data related to the characteristics of the target object such as the attribute information or attribute information set of the target object (such as a user, etc.), the permission or consent of the target object will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, the target object's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the relevant data of the target object necessary for the embodiment of the present application to operate normally will be obtained.

[0097] Figure 3 This is a flowchart of a task execution method provided in an embodiment of the present application. The task execution method can be executed by a server, or by a user terminal, or by both a user terminal and a server. In the embodiment of the present application, the method is described as being executed by a server. Figure 3, the task execution method includes but is not limited to steps 310 to 360.

[0098] Step 310: Determine multiple target data processing tasks according to the business data of different data types in the business data stream.

[0099] Step 320: Obtain the running processing time of each target data processing task.

[0100] Step 330: Determine the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task.

[0101] Step 340: Determine the task processing priority of each target data processing task according to the running processing time and resource allocation ratio of each target data processing task.

[0102] Step 350: Allocate resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task.

[0103] Step 360: Execute the tasks for each target data processing task with resources allocated.

[0104] In one embodiment, the business data stream can be a game data stream, or can be a social media data stream, or can also be an intelligent transportation data stream, etc., which is not specifically limited here.

[0105] In one embodiment, the business data stream may include business data of multiple data types. For example, assuming the business data stream is a game data stream, then the business data stream may include business data of multiple data types such as in-game text data, image data, voice data, rendering data, physical simulation data, and AI computing data. Among them, for each data type of business data, there will be corresponding data processing tasks. In each data processing task, the corresponding business data will be processed. Therefore, when the server receives the business data stream sent by the user terminal, the server can first determine the corresponding multiple target data processing tasks according to the business data of different data types in the business data stream, which is beneficial for subsequent steps to allocate resources to each target data processing task. Moreover, by dividing multiple target data processing tasks according to the business data of different data types in the business data stream, the load and complexity of each target data processing task can be more accurately evaluated, which is not only beneficial for optimizing the performance of each data processing, but also can improve the efficiency of ETL.

[0106] In one embodiment, after the server determines multiple target data processing tasks, the server can obtain the running processing time of each target data processing task, so that in subsequent steps, the resource allocation ratio and task processing priority of each target data processing task can be determined based on the running processing time of each target data processing task. Among them, when the server obtains the running processing time of each target data processing task, it can determine the running processing time of each target data processing task according to the historical running processing time of each target data processing task, or it can first obtain the total running processing time of all data processing tasks corresponding to the service data stream, and then determine the running processing time ratio of each target data processing task, and then calculate the running processing time of each target data processing task according to the total running processing time and the running processing time ratio.

[0107] In one embodiment, when the server determines the running processing time of a target data processing task according to the historical running processing time of the target data processing task, the server can first obtain the average value of the historical running processing time of the target data processing task in the historical time period, and then use this average value as the running processing time of the target data processing task. For example, assume that the historical time period is the time period within the past 5 days. Then, the historical running processing time of the target data processing task within the past 5 days can be obtained, and the historical running processing times of the target data processing task within the past 5 days are T1, T2, T3, T4, and T5 respectively. Then, the average value of T1, T2, T3, T4, and T5 is obtained, and the average value of the historical running processing time of the target data processing task in the historical time period is Tavg. At this time, this average value Tavg can be used as the running processing time of the target data processing task.

[0108] In one embodiment, when the server calculates the running processing time of a target data processing task according to the total running processing time of all data processing tasks and the running processing time ratio of the target data processing task, assume that there are 5 data processing tasks corresponding to the service data stream, and the running processing times of these 5 data processing tasks are T1', T2', T3', T4', and T5' respectively. Then, the sum of T1', T2', T3', T4', and T5' can be obtained, and the total running processing time of all data processing tasks is Ttotal. Then, the running processing time ratio of each target data processing task is determined. Assume that there are 3 target data processing tasks in total, and the running processing time ratios of these 3 target data processing tasks are P1, P2, and P3 respectively. Then, the total running processing time Ttotal is multiplied by P1, P2, and P3 respectively, and the running processing times of these 3 target data processing tasks can be calculated as Tp1, Tp2, and Tp3 respectively.

[0109] In one embodiment, when determining the running processing time ratio of each target data processing task, the server may determine the running processing time ratio of each target data processing task according to the historical running time of each target data processing task. For example, the running processing time ratio of the target data processing task may be determined according to the average value of the ratio of the historical running time of the target data processing task within the past 5 days to the total running processing time of all data processing tasks. For example, assuming that the ratios of the historical running time of the target data processing task within the past 5 days to the total running processing time of all data processing tasks are 20%, 25%, 20%, 20%, and 25% respectively, then the average value of the ratio of the historical running time of the target data processing task within the past 5 days to the total running processing time of all data processing tasks can be calculated to be 22%. Therefore, the running processing time ratio of the target data processing task can be determined to be 22%. In addition, when determining the running processing time ratio of each target data processing task, the server may also call a trained multimodal resource allocation model to determine the running processing time ratio of each target data processing task, which is not specifically limited here. Among them, the multimodal resource allocation model may be a multimodal large model. The training process of the multimodal resource allocation model will be given in detail in the following content and will not be elaborated here for the time being.

[0110] In one embodiment, the total running processing time of all data processing tasks corresponding to the business data stream can be calculated according to the following formula (1):

[0111]

[0112] In formula (1), T total is the total running processing time of all data processing tasks corresponding to the business data stream, and T i is the running processing time of each data processing task in the business data stream. In one embodiment, during the operation of the system, the running processing time of each data processing task will be recorded in the log file. Therefore, T i can be obtained from the log file. When the server obtains the running processing time T i of each data processing task from the log file, after adding up the running processing times T i of all data processing tasks, the total running processing time T total corresponding to the business data stream can be obtained.

[0113] In one embodiment, the server can calculate the running processing time of the target data processing task according to the following formula (2) based on the total running processing time and the running processing time ratio:

[0114] T ta = T total*P ta (2)

[0115] In formula (2), T ta is the running processing time of the target data processing task, and T total is the total running processing time of all data processing tasks corresponding to the service data stream. P ta is the running processing time ratio of the target data processing task. In one embodiment, P ta can be determined according to the historical running time of the target data processing task, or a trained multi-modal resource allocation model can be called to determine it, and appropriate selection can be made according to the actual application situation, which is not specifically limited here. Assume that the service data in the service data stream includes rendering data, physical simulation data, and AI computing data. That is to say, the multiple target data processing tasks determined by the server can include the target data processing task for rendering data, the target data processing task for physical simulation data, and the target data processing task for AI computing data. Then, the running processing time of these target data processing tasks can be calculated by the following formulas (3) to (5):

[0116] T render = T total *P render (3)

[0117] T physics = T total *P physics (4)

[0118] T ai = T total *P ai (5)

[0119] In formulas (3) to (5), T total is the total running processing time of all data processing tasks corresponding to the service data stream, T render is the running processing time of the target data processing task for rendering data, T physics is the running processing time of the target data processing task for physical simulation data, T ai is the running processing time of the target data processing task for AI computing data, P render is the running processing time ratio of the target data processing task for rendering data, P physics is the running processing time ratio of the target data processing task for physical simulation data, P ai is the running processing time ratio of the target data processing task for AI computing data. After the server calculates the total running processing time T totalAfter that, when the server obtains the running processing time ratio P of the target data processing task for the rendering data render 、the running processing time ratio P of the target data processing task for the physical simulation data physics and the running processing time ratio P of the target data processing task for the AI computing data ai , multiply T total by P render to calculate the running processing time T of the target data processing task for the rendering data render ; multiply T total by p physics to calculate the running processing time T of the target data processing task for the physical simulation data physics ; and multiply T total by P ai to calculate the running processing time T of the target data processing task for the AI computing data ai .

[0120] In one embodiment, in the process of the server determining the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, since the longer the running processing time of the target data processing task, it indicates that the more resources the target data processing task requires, that is, the greater the resource allocation ratio of the target data processing task, and the shorter the running processing time of the target data processing task, it indicates that the fewer resources the target data processing task requires, that is, the smaller the resource allocation ratio of the target data processing task. Therefore, the resource allocation ratio of each target data processing task can be determined according to the length of the running processing time of each target data processing task. In addition, in the process of the server determining the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, it can also first perform matrix segmentation on the running processing time of each target data processing task to obtain the data matrix and query matrix of the running processing time of each target data processing task, and then determine the resource allocation ratio of each target data processing task according to the data matrix and query matrix of the running processing time of each target data processing task.

[0121] In one embodiment, when the server performs matrix segmentation on the running processing time of each target data processing task to obtain the data matrix and query matrix of the running processing time of each target data processing task, it can first perform matrix processing on the running processing time of each target data processing task to obtain the running processing time matrix of each target data processing task, and then perform matrix segmentation on the running processing time matrix of each target data processing task according to the target segmentation ratio to obtain the data matrix and query matrix of the running processing time of each target data processing task. The target segmentation ratio can be a preset segmentation ratio determined according to prior knowledge or a segmentation ratio determined by a trained multi-modal resource allocation model, and can be appropriately selected according to the actual application situation, which is not specifically limited herein. Refer to Figure 4 As shown in Figure 4 , when the server performs matrix segmentation on the running processing time of a target data processing task to obtain the data matrix and query matrix of the running processing time of the target data processing task, assuming that the running processing time of the target data processing task is T0, then the running processing time T0 of the target data processing task can be first subjected to matrix processing to obtain the running processing time matrix 410 of the target data processing task, then the target segmentation ratio is determined, and then the running processing time matrix 410 of the target data processing task is subjected to matrix segmentation according to the target segmentation ratio to obtain the data matrix 420 and query matrix 430 of the running processing time of the target data processing task.

[0122] In one embodiment, when the server determines the resource allocation ratio of each target data processing task according to the data matrix and query matrix of the running processing time of each target data processing task, for each target data processing task, the server can multiply and sum the data matrix of the running processing time of the current target data processing task, the query matrix of the running processing time of the current target data processing task, and the query matrices of the running processing times of other target data processing tasks to obtain the resource allocation ratio of each target data processing task. Refer to Figure 5As shown, assume that the target data processing tasks include a first target data processing task 510 and a second target data processing task 520. Among them, the data matrix of the running processing time of the first target data processing task 510 is the first data matrix 511, the query matrix of the running processing time of the first target data processing task 510 is the first query matrix 512, the data matrix of the running processing time of the second target data processing task 520 is the second data matrix 521, and the query matrix of the running processing time of the second target data processing task 520 is the second query matrix 522. Then, for the first target data processing task 510, multiply the first data matrix 511 by the first query matrix 512, and multiply the first data matrix 511 by the second query matrix 522, and then add the two matrix products to obtain the first resource allocation ratio 513 of the first target data processing task 510; for the second target data processing task 520, multiply the second data matrix 521 by the second query matrix 522, and multiply the second data matrix 521 by the first query matrix 512, and then add the two matrix products to obtain the second resource allocation ratio 523 of the second target data processing task 520.

[0123] In one embodiment, the server can perform matrix segmentation on the running processing time of the target data processing task through the following formula (6):

[0124] Value, Query = Seg_Oper(T ta ) (6)

[0125] In formula (6), T ta is the running processing time of the target data processing task, Value is the data matrix of the running processing time T ta , Query is the query matrix of the running processing time T ta , and Seg_Oper() represents performing matrix segmentation on the running processing time T ta . When the target data processing tasks include target data processing tasks for rendering data, target data processing tasks for physical simulation data, and target data processing tasks for AI computing data, the server can perform matrix segmentation on the running processing time of these target data processing tasks through the following formulas (7) to (9):

[0126] Value render , Query render = Seg_Oper(T render ) (7)

[0127] Value physics , Query physics = Seg_Oper(Tphysics ) (8)

[0128] Value ai ,Query ai = Seg_Oper(T ai ) (9)

[0129] In formulas (7) to (9), T render is the running processing time of the target data processing task for rendering data, T physics is the running processing time of the target data processing task for physical simulation data, T ai is the running processing time of the target data processing task for AI computing data, Value render is the data matrix of the running processing time T render , Query render is the query matrix of the running processing time T render , Value physics is the data matrix of the running processing time T physics , Query physics is the query matrix of the running processing time T physics , Value ai is the data matrix of the running processing time T ai , Query ai is the query matrix of the running processing time T ai . After the server calculates the running processing time T render of the target data processing task for rendering data, the running processing time T physics of the target data processing task for physical simulation data, and the running processing time T ai of the target data processing task for AI computing data respectively through the previous formulas (3) to (5), it can call the Seg_Oper() function to perform matrix segmentation on T render , T physics and T ai respectively, so as to obtain the data matrix Value render of T render and the query matrix Query render , the data matrix Value physics of T physics and the query matrix Query physics , the data matrix Value ai of T ai and the query matrix Query ai .

[0130] After completing the operations on T render , T physics and Tai After the matrix is divided, the server can calculate the resource allocation ratio of the target data processing task for rendering data, the resource allocation ratio of the target data processing task for physical simulation data, and the resource allocation ratio of the target data processing task for AI computing data respectively using the following formulas (10) to (12):

[0131] β render = Value render *(Query render , Query physics , Query ai ) (10)

[0132] β physics = Value physics *(Query render , Query physics , Query ai ) (11)

[0133] β ai = Value ai *(Query render , Query physics , Query ai ) (12)

[0134] In formulas (10) to (12), β render is the resource allocation ratio of the target data processing task for rendering data, β physics is the resource allocation ratio of the target data processing task for physical simulation data, and β ai is the resource allocation ratio of the target data processing task for AI computing data.

[0135] Among them, formula (10) means:

[0136] β render = Value render * Query render + Value render * Query physics + Value render * Query ai ;

[0137] Formula (11) means:

[0138] β physics = Value physics * Query render + Value physics * Query physics + Valuephysics *Query ai ;

[0139] Formula (12) indicates that:

[0140] β ai = Value ai *Query render + Value ai *Query physics + Value ai *Query ai 。

[0141] When the server calculates the data matrix Value render and the query matrix Query render for T render respectively through formulas (7) to (9), physics the data matrix Value physics and the query matrix Query physics for T ai and ai the data matrix Value ai and the query matrix Query render for T physics After that, through formulas (10) to (12), the resource allocation ratio β ai for the target data processing task for rendering data,

[0142] In one embodiment, if the running processing time of a target data processing task is longer, it indicates that the target data processing task requires more resources. Also, if the resource allocation ratio of the target data processing task is larger, it also indicates that the target data processing task requires more resources. Conversely, if the running processing time of the target data processing task is shorter, it indicates that the target data processing task requires fewer resources. And if the resource allocation ratio of the target data processing task is smaller, it also indicates that the target data processing task requires fewer resources. Therefore, when the server determines the task processing priority of each target data processing task based on the running processing time and resource allocation ratio of each target data processing task, it can determine the task processing priority of each target data processing task according to the length of the running processing time and the size of the resource allocation ratio of each target data processing task. For example, for a target data processing task, if its running processing time is relatively long and the resource allocation ratio is relatively large, it indicates that the target data processing task requires more resources and needs to execute this target data processing task first. So, it can be determined that its task processing priority is higher. Therefore, based on its running processing time and resource allocation ratio, it can be determined that its task processing priority is relatively high; if its running processing time is relatively long but the resource allocation ratio is relatively small, or its running processing time is relatively short but the resource allocation ratio is relatively large, it indicates that the resources required by this target data processing task are at a medium level and it is not necessary to execute this target data processing task immediately. So, it can be determined that its task processing priority is not too high. Therefore, based on its running processing time and resource allocation ratio, it can be determined that its task processing priority is at a medium level; if its running processing time is relatively short and the resource allocation ratio is relatively small, it indicates that the target data processing task requires fewer resources and there is no need to execute this target data processing task first. So, it can be determined that its task processing priority is lower. Therefore, based on its running processing time and resource allocation ratio, it can be determined that its task processing priority is relatively low.

[0143] In one embodiment, when the server determines the task processing priority of each target data processing task based on the running processing time and resource allocation ratio of each target data processing task, it can also multiply the running processing time and resource allocation ratio of each target data processing task to obtain the task processing priority of each target data processing task. Among them, the server can calculate the task processing priority of the target data processing task through the following formula (13):

[0144] Callback = β ta *T ta (13)

[0145] In formula (13), Callback is the task processing priority of the target data processing task, β tais the resource allocation ratio of the target data processing task, T ta is the running processing time of the target data processing task.

[0146] When the target data processing task includes a target data processing task for rendering data, a target data processing task for physical simulation data, and a target data processing task for AI computing data, the server can calculate the task processing priorities for the target data processing task for rendering data, the task processing priorities for the target data processing task for physical simulation data, and the task processing priorities for the target data processing task for AI computing data respectively through the following formulas (14) to (16):

[0147] Callback render = β render *T render (14)

[0148] Callback physics = β physics *T physics (15)

[0149] Callback ai = β ai *T ai (16)

[0150] In formulas (14) to (16), Callback render is the task processing priority of the target data processing task for rendering data, Callback physics is the task processing priority of the target data processing task for physical simulation data, Callback ai is the task processing priority of the target data processing task for AI computing data, β render is the resource allocation ratio of the target data processing task for rendering data, β physics is the resource allocation ratio of the target data processing task for physical simulation data, β ai is the resource allocation ratio of the target data processing task for AI computing data, T render is the running processing time of the target data processing task for rendering data, T physics is the running processing time of the target data processing task for physical simulation data, T ai is the running processing time of the target data processing task for AI computing data. When the server calculates the running processing time T for the target data processing task for rendering data, the running processing time T for the target data processing task for physical simulation data, and the running processing time T for the target data processing task for AI computing data respectively through the previous formulas (3) to (4), render the running processing time T for the target data processing task for physical simulation data,physics and the running processing time T of the target data processing task for AI computing data ai and respectively calculate the resource allocation ratio β of the target data processing task for rendering data through the previous formulas (10) to (12) render the resource allocation ratio β of the target data processing task for physical simulation data physics and the resource allocation ratio β of the target data processing task for AI computing data ai After that, the task processing priority Callback of the target data processing task for rendering data can be respectively calculated according to formulas (14) to (16) render the task processing priority Callback of the target data processing task for physical simulation data physics and the task processing priority Callback of the target data processing task for AI computing data ai .

[0151] In one embodiment, when the server executes each target data processing task with resource allocation, it can first determine the first data processing task to be preferentially executed among the multiple target data processing tasks with resource allocation according to the task processing priority, then execute the first data processing task, and during the execution of the first data processing task, preprocess the other target data processing tasks among the multiple target data processing tasks. After the first data processing task is executed, then execute the other target data processing tasks that have been preprocessed. By preprocessing the other target data processing tasks in advance when executing the first data processing task, the change of the system state can be made smoother. Therefore, the unnatural or abrupt phenomenon that occurs when executing data processing tasks can be avoided, and thus the efficiency of ETL can be improved. Among them, the process of the server executing each target data processing task with resource allocation can be represented by the following formula (17):

[0152] Smooth=s_f(Task pre , Task cur ) (17)

[0153] In formula (17), Task pre represents the previous executed target data processing task, Task curIndicates the current execution of the target data processing task. s_f() represents the transition function, and Smooth represents the result after smooth transition between two target data processing tasks. Formula (17) can be used to coordinate the states between different target data processing tasks. Through the smooth transition mechanism brought by formula (17), an intelligent and adaptive state adjustment method can be introduced for the switching between different target data processing tasks, which can effectively slow down the speed of system state change, thereby improving the efficiency of ETL.

[0154] In this embodiment, through the task execution method including the previous steps 310 to 360, multiple target data processing tasks are first determined according to the business data of different data types in the business data stream, and the running processing time of each target data processing task is obtained; then, according to the running processing time of each target data processing task, the resource allocation ratio of each target data processing task is determined; for the target data processing task with a long running processing time, more resources are required, while for the target data processing task with a short running processing time, fewer resources are required. Therefore, the resource allocation ratio of the target data processing task can be determined more accurately according to the running processing time of the target data processing task, so that more accurate resources can be allocated to the target data processing task according to the resource allocation ratio; then, according to the running processing time and resource allocation ratio of each target data processing task, the task processing priority of each target data processing task is determined; for the target data processing task with a longer running processing time and a larger resource allocation ratio, its importance is higher, while for the target data processing task with a shorter running processing time and a smaller resource allocation ratio, its importance is lower. Therefore, the task processing priority of the target data processing task can be determined more accurately according to the running processing time and resource allocation ratio of the target data processing task; then, according to the task processing priority and resource allocation ratio, resources are allocated to each target data processing task, and each target data processing task with allocated resources is executed; since the running processing time will change dynamically as the target data processing task is executed, the resource allocation ratio and task processing priority of the target data processing task will also change dynamically. Therefore, the resource allocation for the target data processing task according to the task processing priority and resource allocation ratio will also change dynamically. That is to say, the technical solution provided in the embodiment of the present application can dynamically adjust the resource allocation between data processing tasks, thereby reducing the impact of uneven resource allocation on data processing tasks.

[0155] In one embodiment, the running processing time, resource allocation ratio, and task processing priority of the target data processing task can all be obtained according to a trained multi-modal resource allocation model. Among them, the structure of the multi-modal resource allocation model can refer toFigure 6 As shown in Figure 6 , the multimodal resource allocation model may include multiple parallel data processing branches 610 and a parameter adjustment module 620. In each data processing branch 610, there are included a task priority coefficient calculation module 611, a filtering module 612, a vectorization module 613, and a matrix splitting module 614. In the task priority coefficient calculation module 611, an initial task priority coefficient may be preset, and the initial task priority coefficient may be updated according to the parameters output by the parameter adjustment module 620; in the filtering module 612, the noise in the service data can be filtered; in the vectorization module 613, the running processing time of the target data processing task can be vectorized; in the matrix splitting module 614, the running processing time after vectorization can be matrix-split to obtain a data matrix and a query matrix of the running processing time of the target data processing task. Among them, the vectorization module 613 may be implemented by using a Bidirectional Encoder Representations from Transformers (BERT) model or other neural network models, which is not specifically limited here.

[0156] In one embodiment, after the server determines multiple target data processing tasks according to the service data of different data types in the service data stream, the server can call the multimodal resource allocation model to allocate resources for each target data processing task. In each data processing branch 610, the server can first call the task priority coefficient calculation module 611 in the multimodal resource allocation model to calculate the running processing time of the target data processing task and allocate a corresponding task priority coefficient for the data processing task. And in this process, the server can also call the filtering module 612 in the multimodal resource allocation model to filter the noise in the service data. Then, the server calls the vectorization module 613 in the multimodal resource allocation model to vectorize the running processing time of the target data processing task to obtain a vector representation of the running processing time. At this time, the server then calls the matrix splitting module 614 in the multimodal resource allocation model to split the vector representation of the running processing time to obtain a data matrix and a query matrix of the running processing time. At this time, the multimodal resource allocation model will calculate the resource allocation ratio of the target data processing task according to the data matrix and the query matrix of the running processing time, and calculate the task processing priority of the target data processing task according to the resource allocation ratio and the running processing time of the target data processing task. After the multimodal resource allocation model obtains the resource allocation ratio and the task processing priority of each target data processing task, the server can allocate resources for each target data processing task according to the task processing priority and the resource allocation ratio of each target data processing task, and then execute the tasks for each target data processing task with resources allocated.

[0157] In one embodiment, referring to Figure 6As shown in the figure, the training process of the multi-modal resource allocation model includes the following steps: The server first obtains multiple data processing task samples, and then calls the multi-modal resource allocation model to calculate the processing time of each data processing task sample and assign a task priority coefficient to each data processing task sample. Specifically, the server can call the task priority coefficient calculation module 611 in the multi-modal resource allocation model to calculate the processing time of each data processing task sample and assign a task priority coefficient to each data processing task sample; Then, the server calls the multi-modal resource allocation model and calculates the first task priority of each data processing task sample according to the processing time and task priority coefficient of each data processing task sample. Specifically, the server can call the task priority coefficient calculation module 611 in the multi-modal resource allocation model to calculate the first task priority of each data processing task sample according to the processing time and task priority coefficient of each data processing task sample; Next, the server calls the multi-modal resource allocation model and determines the resource ratio of each data processing task sample according to the processing time of each data processing task sample. Specifically, the server can first call the vectorization module 613 in the multi-modal resource allocation model to vectorize the processing time of each data processing task sample to obtain a vector representation of the processing time of each data processing task sample, and then call the matrix splitting module 614 in the multi-modal resource allocation model to split the vector representation of the processing time of each data processing task sample to obtain a data matrix and a query matrix of the processing time of each data processing task sample. Then, the server calls the multi-modal resource allocation model to calculate the resource allocation ratio of each data processing task sample according to the data matrix and query matrix of the processing time of each data processing task sample; After obtaining the resource allocation ratio of each data processing task sample, the server can call the multi-modal resource allocation model to determine the second task priority of each data processing task sample according to the processing time and resource ratio of each data processing task sample, and then adjust the model parameters of the multi-modal resource allocation model according to the second task priority and the first task priority. Specifically, the server can call the parameter adjustment module 620 in the multi-modal resource allocation model to determine the second task priority of each data processing task sample according to the processing time and resource ratio of each data processing task sample, and calculate the error loss according to the second task priority and the first task priority. After the parameter adjustment module 620 calculates the error loss, the parameter adjustment module 620 will feedback the error loss to the task priority coefficient calculation module 611, so as to adjust the task priority coefficient initially assigned to each data processing task sample.Since the task priority coefficients initially assigned to each data processing task sample will be adjusted, the first task priority calculated based on the task priority coefficients will also be adjusted. Thus, after the training of the multi-modal resource allocation model is completed, the first task priority that meets the requirements can be obtained.

[0158] In one embodiment, when the server calls the multi-modal resource allocation model to calculate the first task priority of each data processing task sample according to the processing time and task priority coefficient of each data processing task sample, it can be implemented through the following formula (18):

[0159] Init = ω * T ta ′ + α (18)

[0160] In formula (18), Init is the first task priority of the data processing task sample, T ta ′ is the processing time of the data processing task sample, ω is the task priority coefficient of the data processing task sample, and α is a hyperparameter matrix used to coordinate dimensional changes. During the processing of the multi-modal resource allocation model, after the multi-modal resource allocation model calculates the processing time of the data processing task sample and assigns the task priority coefficient to the data processing task sample, the processing time and task priority coefficient of the data processing task sample are input into formula (18), and the first task priority of the data processing task sample can be calculated.

[0161] In one embodiment, after the server executes each target data processing task for which resource allocation has been performed, when the server detects a change in any one of the multiple target data processing tasks, the server can obtain the new running processing time of each target data processing task, and then determine the new resource allocation ratio of each target data processing task based on the new running processing time of each target data processing task. Moreover, based on the new running processing time and the new resource allocation ratio of each target data processing task, the server can determine the new task processing priority of each target data processing task. Then, based on the new task processing priority and the new resource allocation ratio, the server can re-allocate resources and execute tasks for each target data processing task. Among them, the change that occurs in any one of the multiple target data processing tasks detected by the server can be a change in the task state of any one of these target data processing tasks (such as the completion or cancellation of a data processing task, etc.), or a change in the task content of any one of these target data processing tasks (such as an adjustment in the task content of a target data processing task after a game update, etc.). No specific limitation is made here. By detecting the status and performance of each target data processing task, potential performance bottleneck problems and resource shortage problems in the system can be identified in a timely manner, so that feedback and optimization of resource adjustment can be made in a timely manner, which helps to better guide the intelligent allocation of resources.

[0162] In one embodiment, in the process of the server determining the new resource allocation ratio of each target data processing task based on the new running processing time of each target data processing task, the server can first obtain the current resource utilization rate, and then determine the new resource allocation ratio of each target data processing task based on the resource utilization rate and the new running processing time of each target data processing task. Among them, the current resource utilization rate can include the usage conditions of key resources such as CPU utilization rate, memory utilization rate, and graphics card utilization rate. By evaluating the current resource usage status and re-determining the resource allocation ratio of each target data processing task based on the current resource utilization rate and the new running processing time of each target data processing task, the server can reasonably re-allocate resources for each target data processing task, which not only helps to ensure that each target data processing task can obtain sufficient resource support, but also can effectively improve the overall system processing efficiency.

[0163] The following uses a specific example to elaborate in detail on the task execution method provided by the embodiments of the present application.

[0164] Refer to Figure 7 As shown, Figure 7 is a system flow diagram of the task execution method provided by a specific example. In Figure 7In this case, it is assumed that a game client is running on the user terminal 710. During its operation, the game client will generate game data of various data types, such as physical simulation data, rendering data, and AI calculation data, etc., and form a game data stream with these game data and send it to the server 720 for data processing. After the server 720 receives the game data stream, the server 720 will execute corresponding data processing tasks for the game data of various data types in the game data stream. Specifically, after the server 720 receives the game data stream, the server 720 will determine multiple target data processing tasks (such as the target data processing task for physical simulation data, the target data processing task for rendering data, and the target data processing task for AI calculation data, etc.) according to the game data of different data types in the game data stream, and then obtain the running processing time of each target data processing task. Among them, when the server 720 obtains the running processing time of each target data processing task, it will first obtain the total running processing time of all data processing tasks corresponding to the game data stream, then determine the running processing time ratio of each target data processing task, and then calculate the running processing time of each target data processing task according to the total running processing time and the running processing time ratio. After obtaining the running processing time of each target data processing task, the server 720 will determine the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, and then determine the task processing priority of each target data processing task according to the running processing time and the resource allocation ratio of each target data processing task. Among them, when the server 720 determines the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, it will first perform matrix segmentation on the running processing time of each target data processing task to obtain the data matrix and query matrix of the running processing time of each target data processing task, and then determine the resource allocation ratio of each target data processing task according to the data matrix and query matrix of the running processing time of each target data processing task; in addition, when the server 720 determines the task processing priority of each target data processing task according to the running processing time and the resource allocation ratio of each target data processing task, it will multiply the running processing time and the resource allocation ratio of each target data processing task to obtain the task processing priority of each target data processing task.After determining the resource allocation ratio and task processing priority for each target data processing task, the server 720 allocates resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task, and then executes the tasks for each target data processing task with resources allocated. Among them, when the server 720 executes the tasks for each target data processing task with resources allocated, it first determines the first data processing task to be preferentially executed among the multiple target data processing tasks with resources allocated according to the task processing priority, then executes the first data processing task, and during the execution of the first data processing task, preprocesses the other target data processing tasks among the multiple target data processing tasks. After the execution of the first data processing task is completed, the server 720 executes the tasks for the other target data processing tasks that have been preprocessed. After the server 720 completes the execution of the target data processing tasks and obtains the corresponding task calculation results, the server 720 can return the obtained task calculation results to the user terminal 710, so that the user terminal 710 can run the game in the game client according to the task calculation results.

[0165] In addition, during the process of the user terminal 710 sending the game data stream to the server 720 for data processing and the server 720 returning the task calculation results to the user terminal 710, the detection device 730 can detect the data processing tasks for the entire process. For example, when the detection device 730 detects a change in the task status or task content of any one of these target data processing tasks, the detection device 730 can trigger the server 720 to obtain the new running processing time of each target data processing task, so that the server 720 can determine the new resource allocation ratio of each target data processing task according to the new running processing time of each target data processing task, and determine the new task processing priority of each target data processing task according to the new running processing time and new resource allocation ratio of each target data processing task, and then re-allocate resources and execute tasks for each target data processing task according to the new task processing priority and new resource allocation ratio.

[0166] According to Figure 8 the system process of the task execution method shown, the system step schematic diagram of the task execution method as shown in Figure 8 can be obtained. As shown in Figure 8 the system steps of the task execution method can include but are not limited to steps 810 to 860.

[0167] Step 810: The server classifies the complex game data in the game data stream sent by the user terminal into multiple target data processing tasks according to the data type, and allocates resources to these target data processing tasks respectively.

[0168] In this step, after the server classifies the complex game data in the game data stream into multiple target data processing tasks according to the data type, it can obtain the running processing time of each target data processing task, and then determine the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task. And according to the running processing time and resource allocation ratio of each target data processing task, determine the task processing priority of each target data processing task. Then, according to the task processing priority and resource allocation ratio of each target data processing task, allocate resources to each target data processing task.

[0169] Step 820: The server executes each target data processing task with resource allocation respectively.

[0170] In this step, when the server executes each target data processing task with resource allocation respectively, it can first determine the first data processing task to be preferentially executed among the multiple target data processing tasks with resource allocation according to the task processing priority, and then execute the first data processing task. And during the execution of the first data processing task, preprocess other target data processing tasks among the multiple target data processing tasks. After the first data processing task is executed, the server executes the other target data processing tasks that have been preprocessed. By preprocessing other target data processing tasks in advance when executing the first data processing task, the change of the system state can be made smoother. Therefore, the unnatural or abrupt phenomenon that occurs during the execution of data processing tasks can be avoided, and thus the efficiency of ETL can be improved.

[0171] Step 830: The server regularly detects the status and performance of each target data processing task.

[0172] In this step, the server can regularly detect the status and performance of each target data processing task. For example, when the server detects a change in the task status or task content of any one of the multiple target data processing tasks, the server can obtain the new running processing time of each target data processing task, and then determine the new resource allocation ratio of each target data processing task according to the new running processing time of each target data processing task. And according to the new running processing time and new resource allocation ratio of each target data processing task, determine the new task processing priority of each target data processing task. Then, according to the new task processing priority and new resource allocation ratio, re-allocate resources and execute tasks for each target data processing task.

[0173] Step 840: The server dynamically responds to the task processing priority and resource allocation of each target data processing task.

[0174] In this step, since the running processing time of the target data processing task will change dynamically as the task is executed, the resource allocation ratio and task processing priority of the target data processing task can be made to change dynamically. Therefore, the resource allocation performed by the server on the target data processing task according to the task processing priority and resource allocation ratio will also change dynamically. That is to say, the server can dynamically adjust the task processing priorities and resource allocations among data processing tasks, thus reducing the impact of uneven resource allocation on data processing tasks.

[0175] Step 850: The server re-obtains multiple target data processing tasks according to the user's adjustment of game parameters, and re-performs resource allocation and task execution on these new target data processing tasks.

[0176] In this step, when the user adjusts game parameters such as the lighting effect and anti-aliasing effect of the game screen during the game, the game data such as physical simulation data, rendering data, and AI calculation data generated by the game client will change. As a result, the data processing content corresponding to these game data will change. In this case, the server can re-obtain multiple target data processing tasks according to the user's adjustment of game parameters, and then re-perform resource allocation and task execution on these new target data processing tasks, so as to effectively respond to the dynamic changes of game data and ensure the normal operation of the game.

[0177] Step 860: The server periodically performs performance testing and optimization.

[0178] In this step, when the game is updated or optimized and maintained, the game data such as physical simulation data, rendering data, or AI calculation data in the game may be adjusted. As a result, the data processing content corresponding to these game data may change. In this case, the server can re-obtain multiple target data processing tasks according to the game data after the game update or after the game optimization and maintenance, and then re-perform resource allocation and task execution on these new target data processing tasks, so as to ensure the normal operation of the game after the update or after the optimization and maintenance.

[0179] Refer to Figure 9 as shown Figure 9 is a detailed flowchart of a task execution method provided by a specific example. This task execution method can be executed by the server, or by the user terminal, or jointly by the user terminal and the server. In this specific example, it is described by taking the method being executed by the server as an example. In Figure 9 it, this task execution method may include but is not limited to steps 901 to 920.

[0180] Step 901: Obtain multiple data processing task samples, call the multi-modal resource allocation model, calculate the processing time of each data processing task sample, and assign a task priority coefficient to each data processing task sample.

[0181] Step 902: Call the multi-modal resource allocation model, and calculate the first task priority of each data processing task sample according to the processing time and task priority coefficient of each data processing task sample.

[0182] Step 903: Call the multi-modal resource allocation model, and determine the resource ratio of each data processing task sample according to the processing time of each data processing task sample.

[0183] Step 904: Call the multi-modal resource allocation model, and determine the second task priority of each data processing task sample according to the processing time and resource ratio of each data processing task sample.

[0184] Step 905: Adjust the model parameters of the multi-modal resource allocation model according to the second task priority and the first task priority.

[0185] Through the processing of steps 901 to 905, the training of the multi-modal resource allocation model can be realized, so that the trained multi-modal resource allocation model can obtain the running processing time, resource allocation ratio, and task processing priority of each target data processing task.

[0186] Step 906: Detect whether a business data stream is received. If so, execute step 907; if not, execute step 906.

[0187] Step 907: Determine multiple target data processing tasks according to the business data of different data types in the business data stream.

[0188] Step 908: Obtain the total running processing time of all data processing tasks corresponding to the business data stream, determine the running processing time ratio of each target data processing task, and then calculate the running processing time of each target data processing task according to the total running processing time and the running processing time ratio.

[0189] Step 909: Perform matrix processing on the running processing time of each target data processing task to obtain the running processing time matrix of each target data processing task.

[0190] Step 910: According to the target segmentation ratio, perform matrix segmentation on the running processing time matrix of each target data processing task to obtain the data matrix and query matrix of the running processing time of each target data processing task.

[0191] Step 911: For each target data processing task, perform a multiplication and summation operation on the data matrix of the running processing time of the current target data processing task, the query matrix of the running processing time of the current target data processing task, and the query matrices of the running processing times of other target data processing tasks, to obtain the resource allocation ratio for each target data processing task.

[0192] Step 912: Multiply the running processing time of each target data processing task by its resource allocation ratio to obtain the task processing priority for each target data processing task.

[0193] Step 913: Allocate resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task.

[0194] Step 914: Determine the first data processing task to be preferentially executed among the multiple target data processing tasks for which resources have been allocated, according to the task processing priority.

[0195] Step 915: Execute the first data processing task, and during the execution of the first data processing task, perform preprocessing on the other target data processing tasks among the multiple target data processing tasks. After the execution of the first data processing task, perform task execution on the other target data processing tasks that have undergone preprocessing.

[0196] Step 916: Detect whether there is any change in the task status or task content of any one of the multiple target data processing tasks. If so, execute Step 917; if not, execute Step 916.

[0197] Step 917: Obtain the new running processing time of each target data processing task.

[0198] Step 918: Obtain the current resource utilization rate, and determine the new resource allocation ratio for each target data processing task according to the resource utilization rate and the new running processing time of each target data processing task.

[0199] Step 919: Determine the new task processing priority for each target data processing task according to the new running processing time and new resource allocation ratio of each target data processing task.

[0200] Step 920: Re-allocate resources and perform task execution on each target data processing task according to the new task processing priority and new resource allocation ratio.

[0201] In this embodiment, through the task execution method of steps 901 to 920 above, first determine multiple target data processing tasks according to the service data of different data types in the service data stream, and obtain the running processing time of each target data processing task; then, according to the running processing time of each target data processing task, determine the resource allocation ratio of each target data processing task; for the target data processing task with a long running processing time, it requires more resources, while for the target data processing task with a short running processing time, it requires fewer resources. Therefore, the resource allocation ratio of the target data processing task can be determined more accurately according to the running processing time of the target data processing task, so that more precise resources can be allocated to the target data processing task according to the resource allocation ratio; then, according to the running processing time and resource allocation ratio of each target data processing task, determine the task processing priority of each target data processing task; for the target data processing task with a longer running processing time and a larger resource allocation ratio, its importance level is higher, while for the target data processing task with a shorter running processing time and a smaller resource allocation ratio, its importance level is lower. Therefore, the task processing priority of the target data processing task can be determined more accurately according to the running processing time and resource allocation ratio of the target data processing task; then, according to the task processing priority and resource allocation ratio, allocate resources to each target data processing task, and execute the task for each target data processing task that has been allocated resources; since the running processing time of the target data processing task will change dynamically as the task is executed, this will cause the resource allocation ratio and task processing priority of the target data processing task to change dynamically. Therefore, the resource allocation for the target data processing task according to the task processing priority and resource allocation ratio will also change dynamically. That is to say, the technical solution provided by the embodiment of the present application can dynamically adjust the resource allocation between data processing tasks, thereby reducing the impact of uneven resource allocation on data processing tasks.

[0202] The following uses some actual examples to illustrate the application scenarios of the embodiments of the present application.

[0203] It should be noted that the task execution method provided by the embodiments of the present application can be applied to different application scenarios such as the execution of data processing tasks for game data, the execution of data processing tasks for social media data, and the execution of data processing tasks for intelligent transportation data. The following uses the execution scenarios of data processing tasks for game data, the execution scenarios of data processing tasks for social media data, and the execution scenarios of data processing tasks for intelligent transportation data as examples for illustration.

[0204] Scenario 1

[0205] The task execution method provided by the embodiments of the present application can be applied to the execution scenario of data processing tasks for game data. For example, when a user plays a game using a game client installed on a user terminal, the game client will generate game data of various data types, such as physical simulation data, rendering data, and AI calculation data, and form a game data stream with this game data through the user terminal and send it to the server for data processing. After the server receives the game data stream, the server can determine multiple target data processing tasks according to the game data of different data types in the game data stream, such as target data processing tasks for physical simulation data, target data processing tasks for rendering data, and target data processing tasks for AI calculation data. Then, the server obtains the running processing time of each target data processing task, and determines the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, and determines the task processing priority of each target data processing task according to the running processing time and resource allocation ratio of each target data processing task. Next, the server allocates resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task, and executes the tasks for each target data processing task with resources allocated. After the server completes the execution of the target data processing task and obtains the corresponding task calculation result, the server returns the obtained task calculation result to the user terminal, so that the user terminal can run the game in the game client according to the task calculation result.

[0206] Scenario 2

[0207] The task execution method provided by the embodiments of the present application can also be applied to the execution scenario of data processing tasks for social media data. For example, when a user uses a social media client installed on the user terminal for social interaction, the social media client will generate social media data of various data types such as comment data, picture data, and video data, and will form a social media data stream with these social media data through the user terminal and send it to the server for data processing. After the server receives the social media data stream, the server can determine multiple target data processing tasks according to the social media data of different data types in the social media data stream, such as target data processing tasks for comment data, target data processing tasks for picture data, and target data processing tasks for video data, etc. Then, the server obtains the running processing time of each target data processing task, and determines the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, and determines the task processing priority of each target data processing task according to the running processing time and resource allocation ratio of each target data processing task. Next, the server allocates resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task, and executes the tasks for each target data processing task with resources allocated. After the server completes the execution of the target data processing task and obtains the corresponding task calculation result, the server returns the obtained task calculation result to the user terminal, so that the user terminal can push information to the user through the social media client according to the task calculation result.

[0208] Scenario Three

[0209] The task execution method provided by the embodiments of the present application can also be applied to the execution scenario of data processing tasks for intelligent transportation data. For example, when the user is in a driving state, the in-vehicle client installed on the in-vehicle terminal can obtain in real time various types of intelligent transportation data such as the driving data of the vehicle, the image data and video data of the road, and form an intelligent transportation data stream from these intelligent transportation data through the in-vehicle terminal and send it to the server for data processing. After receiving the intelligent transportation data stream, the server can determine multiple target data processing tasks according to the intelligent transportation data of different data types in the intelligent transportation data stream, such as the target data processing task for the driving data of the vehicle, the target data processing task for the image data of the road, and the target data processing task for the video data of the road. Then, the server obtains the running processing time of each target data processing task, and determines the resource allocation ratio of each target data processing task according to the running processing time of each target data processing task, and determines the task processing priority of each target data processing task according to the running processing time and resource allocation ratio of each target data processing task. Next, the server allocates resources to each target data processing task according to the task processing priority and resource allocation ratio of each target data processing task, and executes the task for each target data processing task with resource allocation. After the server completes the execution of the target data processing task and obtains the corresponding task calculation result, the server can perform different processes such as traffic flow analysis, road condition analysis, and traffic safety warning according to the obtained task calculation result.

[0210] It can be understood that although each step in the above-mentioned various flowcharts is displayed sequentially according to the indication of the arrow, these steps do not necessarily need to be executed sequentially according to the order indicated by the arrow. Unless there is a clear indication in this embodiment, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowchart may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0211] Referring to Figure 10 , the embodiments of the present application also disclose a task execution device. The task execution device 1000 can implement the task execution method in the previous embodiments. The task execution device 1000 includes:

[0212] A task determination unit 1010, configured to determine multiple target data processing tasks according to the service data of different data types in the service data stream;

[0213] A time acquisition unit 1020, configured to acquire the running processing time of each target data processing task;

[0214] A proportion determination unit 1030, configured to determine the resource allocation proportion of each target data processing task according to the running processing time of each target data processing task;

[0215] A priority determination unit 1040, configured to determine the task processing priority of each target data processing task according to the running processing time and resource allocation proportion of each target data processing task;

[0216] A resource allocation unit 1050, configured to allocate resources to each target data processing task according to the task processing priority and resource allocation proportion of each target data processing task;

[0217] A task execution unit 1060, configured to execute the tasks of each target data processing task for which resources have been allocated.

[0218] In one embodiment, the proportion determination unit 1030 is further configured to:

[0219] Perform matrix segmentation on the running processing time of each target data processing task to obtain a data matrix and a query matrix of the running processing time of each target data processing task;

[0220] Determine the resource allocation proportion of each target data processing task according to the data matrix and the query matrix of the running processing time of each target data processing task.

[0221] In one embodiment, the proportion determination unit 1030 is further configured to:

[0222] For each target data processing task, multiply and sum the data matrix of the running processing time of the current target data processing task, the query matrix of the running processing time of the current target data processing task, and the query matrices of the running processing times of other target data processing tasks to obtain the resource allocation proportion of each target data processing task.

[0223] In one embodiment, the proportion determination unit 1030 is further configured to:

[0224] Perform matrix processing on the running processing time of each target data processing task to obtain a running processing time matrix of each target data processing task;

[0225] According to the target segmentation ratio, perform matrix segmentation on the running processing time matrix of each target data processing task to obtain a data matrix and a query matrix of the running processing time of each target data processing task.

[0226] In one embodiment, the priority determination unit 1040 is further configured to:

[0227] Multiply the running processing time of each target data processing task by the resource allocation ratio to obtain the task processing priority of each target data processing task.

[0228] In one embodiment, the time acquisition unit 1020 is further configured to:

[0229] Obtain the total running processing time of all data processing tasks corresponding to the service data stream;

[0230] Determine the running processing time ratio of each target data processing task;

[0231] Calculate the running processing time of each target data processing task according to the total running processing time and the running processing time ratio.

[0232] In one embodiment, the running processing time, the resource allocation ratio, and the task processing priority are all obtained according to a trained multi-modal resource allocation model. The task execution device 1000 further includes a model training unit, and the model training unit is configured to:

[0233] Obtain a plurality of data processing task samples;

[0234] Call the multi-modal resource allocation model to calculate the processing time of each data processing task sample and assign a task priority coefficient to each data processing task sample;

[0235] Call the multi-modal resource allocation model to calculate the first task priority of each data processing task sample according to the processing time and the task priority coefficient of each data processing task sample;

[0236] Call the multi-modal resource allocation model to determine the resource ratio of each data processing task sample according to the processing time of each data processing task sample;

[0237] Call the multi-modal resource allocation model to determine the second task priority of each data processing task sample according to the processing time and the resource ratio of each data processing task sample;

[0238] Adjust the model parameters of the multi-modal resource allocation model according to the second task priority and the first task priority.

[0239] In one embodiment, the task execution device 1000 further includes a first resource reallocation unit, and the first resource reallocation unit is configured to:

[0240] When it is detected that any one of the multiple target data processing tasks has changed, obtain the new running processing time of each target data processing task;

[0241] Determine the new resource allocation ratio for each target data processing task according to the new running processing time of each target data processing task;

[0242] Determine the new task processing priority for each target data processing task according to the new running processing time and the new resource allocation ratio of each target data processing task;

[0243] Re - allocate resources and execute tasks for each target data processing task again according to the new task processing priority and the new resource allocation ratio.

[0244] In one embodiment, any one of the multiple target data processing tasks changes, including:

[0245] Any one of the multiple target data processing tasks changes in task status;

[0246] Or, any one of the multiple target data processing tasks changes in task content.

[0247] In one embodiment, the first resource re - allocation unit is further configured to:

[0248] Obtain the current resource utilization rate;

[0249] Determine the new resource allocation ratio for each target data processing task according to the resource utilization rate and the new running processing time of each target data processing task.

[0250] In one embodiment, the task execution unit 1060 is further configured to:

[0251] Determine the first data processing task to be preferentially executed among the multiple target data processing tasks with resources allocated according to the task processing priority;

[0252] Execute the first data processing task, and during the execution of the first data processing task, pre - process the other target data processing tasks among the multiple target data processing tasks;

[0253] After the first data processing task is executed, execute the other target data processing tasks that have been pre - processed.

[0254] It should be noted that since the task execution device 1000 in this embodiment can implement the task execution method in the previous embodiment, the task execution device 1000 in this embodiment and the task execution method in the previous embodiment have the same technical principle and the same beneficial effects. To avoid repetition of content, it will not be elaborated here.

[0255] Refer to Figure 11, embodiments of the present application also disclose an electronic device, and the electronic device 1100 includes:

[0256] At least one processor 1101;

[0257] At least one memory 1102 for storing at least one program;

[0258] When the at least one program is executed by the at least one processor 1101, the foregoing task execution method is implemented.

[0259] Embodiments of the present application also disclose a computer-readable storage medium, in which a computer program executable by a processor is stored. When the computer program executable by the processor is executed by the processor, it is used to implement the foregoing task execution method.

[0260] Embodiments of the present application also disclose a computer program product, including a computer program or computer instructions. The computer program or computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the electronic device executes the foregoing task execution method.

[0261] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0262] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0263] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be an indirect coupling or communication connection through some interfaces, devices, or units, and can be in electrical, mechanical, or other forms.

[0264] In the embodiments of this application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0265] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may or may not be a physical unit, that is, it can be located in one place, or it can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0266] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs.

[0268] Regarding the step numbers in the above method embodiments, they are only set for the convenience of elaboration and explanation, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. A task execution method, characterized in that, Including the following steps: Determine multiple target data processing tasks according to the business data of different data types in the business data stream; Obtain the running processing time of each of the target data processing tasks; Determine the resource allocation ratio of each of the target data processing tasks according to the running processing time of each of the target data processing tasks; Determine the task processing priority of each of the target data processing tasks according to the running processing time and the resource allocation ratio of each of the target data processing tasks; Allocate resources to each of the target data processing tasks according to the task processing priority and the resource allocation ratio of each of the target data processing tasks; Execute the tasks for each of the target data processing tasks to which resources have been allocated.

2. The method according to claim 1, characterized in that The determining the resource allocation ratio of each of the target data processing tasks according to the running processing time of each of the target data processing tasks includes: Perform matrix segmentation on the running processing time of each of the target data processing tasks to obtain the data matrix and the query matrix of the running processing time of each of the target data processing tasks; Determine the resource allocation ratio of each of the target data processing tasks according to the data matrix and the query matrix of the running processing time of each of the target data processing tasks.

3. The method according to claim 2, characterized in that, The determining the resource allocation ratio of each of the target data processing tasks according to the data matrix and the query matrix of the running processing time of each of the target data processing tasks includes: For each of the target data processing tasks, multiply and sum the data matrix of the running processing time of the current target data processing task, the query matrix of the running processing time of the current target data processing task, and the query matrices of the running processing times of the other target data processing tasks to obtain the resource allocation ratio of each of the target data processing tasks.

4. The method according to claim 2, characterized in that, The performing matrix segmentation on the running processing time of each of the target data processing tasks to obtain the data matrix and the query matrix of the running processing time of each of the target data processing tasks includes: Perform matrix processing on the running processing time of each of the target data processing tasks to obtain the running processing time matrix of each of the target data processing tasks; According to the target segmentation ratio, perform matrix segmentation on the running processing time matrix of each of the target data processing tasks to obtain the data matrix and the query matrix of the running processing time of each of the target data processing tasks.

5. The method according to claim 1, wherein The determining the task processing priority of each of the target data processing tasks according to the running processing time and the resource allocation ratio of each of the target data processing tasks includes: Multiply the running processing time and the resource allocation ratio of each of the target data processing tasks to obtain the task processing priority of each of the target data processing tasks.

6. The method according to claim 1, characterized in that The obtaining the running processing time of each of the target data processing tasks includes: Obtain the total running processing time of all data processing tasks corresponding to the business data stream; Determine the running processing time ratio of each of the target data processing tasks; Calculate the running processing time of each of the target data processing tasks based on the total running processing time and the running processing time ratio.

7. The method according to claim 1, characterized in that The running processing time, the resource allocation ratio, and the task processing priority are all obtained according to a trained multi-modal resource allocation model. The training process of the multi-modal resource allocation model includes the following steps: Obtain a plurality of data processing task samples; Call the multi-modal resource allocation model to calculate the processing time of each of the data processing task samples and assign a task priority coefficient to each of the data processing task samples; Call the multi-modal resource allocation model to calculate the first task priority of each of the data processing task samples according to the processing time and the task priority coefficient of each of the data processing task samples; Call the multi-modal resource allocation model to determine the resource ratio of each of the data processing task samples according to the processing time of each of the data processing task samples; Call the multi-modal resource allocation model to determine the second task priority of each of the data processing task samples according to the processing time and the resource ratio of each of the data processing task samples; Adjust the model parameters of the multi-modal resource allocation model according to the second task priority and the first task priority.

8. The method according to claim 1, wherein After performing task execution on each of the target data processing tasks for which resource allocation has been performed, the method further includes: When it is detected that any one of the multiple target data processing tasks has changed, obtain the new running processing time of each of the target data processing tasks; Determine the new resource allocation ratio of each of the target data processing tasks according to the new running processing time of each of the target data processing tasks; Determine the new task processing priority of each of the target data processing tasks according to the new running processing time and the new resource allocation ratio of each of the target data processing tasks; Re-allocate resources and perform tasks on each of the target data processing tasks according to the new task processing priority and the new resource allocation ratio.

9. The method according to claim 8, wherein That any one of the multiple target data processing tasks has changed includes: That any one of the multiple target data processing tasks has changed in task status; Or, that any one of the multiple target data processing tasks has changed in task content.

10. The method according to claim 8, wherein The determining the new resource allocation ratio of each of the target data processing tasks according to the new running processing time of each of the target data processing tasks includes: Obtain the current resource utilization rate; Determine the new resource allocation ratio of each of the target data processing tasks according to the resource utilization rate and the new running processing time of each of the target data processing tasks.

11. The method according to claim 1, wherein The performing task execution on each of the target data processing tasks for which resource allocation has been performed includes: Determine a first data processing task with the highest priority among multiple target data processing tasks for which resources have been allocated according to the task processing priority; Execute the first data processing task, and during the execution of the first data processing task, preprocess other target data processing tasks among the multiple target data processing tasks; After the execution of the first data processing task is completed, execute the other target data processing tasks that have been preprocessed.

12. A task execution device, characterized in that, It includes: A task determination unit for determining multiple target data processing tasks according to service data of different data types in the service data stream; A time acquisition unit for acquiring the running processing time of each target data processing task; A proportion determination unit for determining the resource allocation proportion of each target data processing task according to the running processing time of each target data processing task; A priority determination unit for determining the task processing priority of each target data processing task according to the running processing time and the resource allocation proportion of each target data processing task; A resource allocation unit for allocating resources to each target data processing task according to the task processing priority and the resource allocation proportion of each target data processing task; A task execution unit for executing each target data processing task for which resources have been allocated.

13. An electronic device, characterized in that, It includes: At least one processor; At least one memory for storing at least one program; When at least one of the at least one program is executed by at least one of the at least one processor, the task execution method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that, Wherein there is a computer program executable by a processor, and when the computer program executable by the processor is executed by the processor, it is used to implement the task execution method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instruction is stored in a computer-readable storage medium, and the processor of the electronic device reads the computer program or the computer instruction from the computer-readable storage medium, and the processor executes the computer program or the computer instruction, so that the electronic device executes the task execution method according to any one of claims 1 to 11.

Images