Method and device for scheduling execution of multi-modal tasks, storage medium and program product

Abstract

The application discloses a multi-modal task scheduling execution method and device, a storage medium and a program product. The method comprises the following steps: receiving a plurality of multi-modal data reported by a terminal device, and routing a plurality of multi-modal tasks belonging to a target user to the same gateway node according to the user identifier of the plurality of multi-modal data; deploying the plurality of multi-modal tasks to a plurality of computing processes of a GPU server for concurrent execution through the gateway node, wherein the plurality of computing processes comprise a plurality of task processes, and the plurality of task processes are used for executing multi-modal tasks of different task types respectively; receiving an execution state of the plurality of computing processes reported by the GPU server; and dynamically adjusting the resource allocation proportion of different task processes in a process pool according to the execution state of the plurality of computing processes, so as to shorten the time difference between the task completion times of the plurality of multi-modal tasks.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for scheduling and executing multimodal tasks, a storage medium, and a program product. Background Technology

[0002] Currently, multimodal and multi-model technologies are becoming increasingly mature and their application scenarios are becoming more and more diverse. For example, they are widely used in intelligent driving, cloud intelligence of IoT devices, etc. However, multimodal tasks have different processing speeds for data of different dimensions. Decision nodes or service data post-processing nodes are waiting for time-consuming data, which is particularly evident in distributed systems.

[0003] In related technologies, in distributed multimodal large model parallel processing systems, the difference in computation time of different modal tasks leads to inconsistent arrival times of results at decision nodes, resulting in decision delays and low resource utilization. No effective solution has yet been proposed. Summary of the Invention

[0004] This application provides a method and apparatus for scheduling and executing multimodal tasks, a storage medium, and a program product to at least solve the problem in related technologies where, in distributed multimodal large-scale parallel processing systems, the time it takes for results to arrive at decision nodes is inconsistent due to differences in the computation time of different modal tasks, leading to decision delays and low resource utilization.

[0005] According to one embodiment of this application, a method for scheduling and executing multimodal tasks is provided, comprising: receiving multiple multimodal data reported by a terminal device; routing multiple multimodal tasks belonging to a target user to the same gateway node based on the user identifier of the multiple multimodal data, wherein the multiple multimodal tasks are all used to process multimodal data, and the multiple multimodal tasks have different task types; deploying the multiple multimodal tasks to multiple computing processes of a GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, and the multiple types of task processes are used to execute multimodal tasks of different task types respectively; receiving the execution status of the multiple computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information; dynamically adjusting the resource allocation ratio of different types of task processes in a process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes multiple computing processes of the GPU server.

[0006] In one exemplary embodiment, routing multiple multimodal tasks belonging to a target user to the same gateway node based on the user identifiers of the multiple multimodal data includes: generating a fixed-length hash value from the user identifiers of the target user using a hash function; mapping the hash value to a target virtual node among multiple virtual nodes on a virtual ring, wherein each of the multiple gateway nodes has K virtual nodes evenly distributed on the virtual ring, where K is a positive integer, and each of the multiple virtual nodes is bound to the IP address and port number of a corresponding GPU server; and routing the multiple multimodal tasks to the gateway node based on the IP address and port number bound to the target virtual node.

[0007] In an exemplary embodiment, dynamically adjusting the resource allocation ratio of different types of task processes in the process pool according to the execution status of the plurality of computing processes includes: when the resource pressure information indicates that the computing resource pressure of the process pool is less than a first preset threshold, predicting the time interval in which the task completion time of the plurality of multimodal tasks is located based on the expected processing time, and predicting a first time difference in the task completion time of the plurality of multimodal tasks based on the length of the time interval; when the first time difference is greater than a second preset threshold, determining a first task type of the first multimodal task and a second task type of the second multimodal task, and determining a first process type corresponding to the first task type and a second process type corresponding to the second task type, wherein the first multimodal task is a task whose task completion time is located at the beginning of the time interval, the second multimodal task is a task whose task completion time is located at the end of the time interval, and the plurality of task processes include the first process type and the second process type; switching M computing processes of the first process type in the process pool to the second process type, wherein M is a positive integer.

[0008] In an exemplary embodiment, after switching M computing processes of the first process type in the process pool to the second process type, the method further includes: receiving the execution status of the plurality of computing processes reported by the GPU server again, and re-predicting the second time difference of the task completion time of the plurality of multimodal tasks; determining whether the second time difference is greater than the second preset threshold; and if the second time difference is greater than the second preset threshold, adjusting the resource allocation ratio of different types of task processes in the process pool again.

[0009] In an exemplary embodiment, dynamically adjusting the resource allocation ratio of different types of task processes in the process pool according to the execution status of the plurality of computing processes includes: when the resource pressure information indicates that the computing resource pressure of the process pool is greater than a first preset threshold, determining the first allocation ratio of different types of task processes in the process pool, and determining the task pressure of different types of task processes; and adjusting the first allocation ratio to a second allocation ratio according to the task pressure.

[0010] In an exemplary embodiment, after adjusting the first allocation ratio to the second allocation ratio according to the task pressure, the method further includes: classifying the multiple GPU servers into high-pressure servers and low-pressure servers according to the computing resource pressure of the multiple GPU servers, wherein the multiple GPU servers correspond one-to-one with the multiple gateway nodes; predicting the time interval in which the task completion time of the multiple multimodal tasks is located based on the expected processing time, and predicting a third time difference in the task completion time of the multiple multimodal tasks based on the length of the time interval; and performing task migration on the multiple multimodal tasks based on the third time difference.

[0011] In an exemplary embodiment, migrating the plurality of multimodal tasks according to the third time difference includes: determining the server category of the GPU server where the plurality of multimodal tasks are currently located; if the third time difference is greater than a third preset threshold and the server category is a low-pressure server, migrating the plurality of multimodal tasks to a first GPU server for execution, wherein the first GPU server is a high-pressure server; if the third time difference is less than or equal to the third preset threshold and the server category is a high-pressure server, migrating the plurality of multimodal tasks to a second GPU server for execution, wherein the second GPU server is a low-pressure server.

[0012] According to another embodiment of the present application, a multimodal task scheduling and execution apparatus is also provided, comprising: a routing module, configured to receive multiple multimodal data reported by a terminal device, and route multiple multimodal tasks belonging to a target user to the same gateway node according to the user identifier of the multiple multimodal data, wherein the multiple multimodal tasks are all used to process multimodal data, and the multiple multimodal tasks have different task types; a deployment module, configured to deploy the multiple multimodal tasks to multiple computing processes of a GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, and the multiple types of task processes are used to execute multimodal tasks of different task types respectively; a receiving module, configured to receive the execution status of the multiple computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information; and an adjustment module, configured to dynamically adjust the resource allocation ratio of different types of task processes in a process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes multiple computing processes of the GPU server.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the scheduling execution method of the above-described multimodal task at runtime.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described multimodal task scheduling and execution method through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0016] This application provides a method for scheduling and executing multimodal tasks. First, it receives multimodal data, including video and audio sensor signals, reported from terminal devices (such as cameras, wristbands, door sensors, etc.). Based on user identifiers, it uses a consistent hashing algorithm to route all multimodal tasks belonging to the same target user to the same gateway node, ensuring physical consistency in task processing. Then, the gateway node deploys these different task types (such as task detection, audio recognition, and behavior analysis) to corresponding multi-process computing tasks on a GPU server for parallel execution. Simultaneously, it continuously receives real-time execution status reports from each computing process, including single-task expected performance. The system processes data on processing time and resource pressure. Based on this, it dynamically analyzes the time difference between task completions and adjusts the resource allocation ratio of different task types in the process pool according to the time difference and execution status to shorten processing time. This achieves fine-grained computing power sharing and task completion time convergence across server process pools without increasing hardware, ultimately achieving consistency in multimodal task results and synergistic optimization of system resource utilization. Furthermore, it solves the technical problem in distributed multimodal large-scale parallel processing systems where the difference in computation time between different modal tasks leads to inconsistent arrival times of results at decision nodes, resulting in decision delays and low resource utilization. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of the scheduling engine of an optional multimodal task scheduling and execution method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of an optional multimodal task scheduling and execution method according to an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating an optional multimodal task serial detection method according to an embodiment of this application.

[0021] Figure 4 This is a flowchart illustrating an optional multimodal task parallel detection method according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an optional consensus algorithm according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of an optional multi-GPU and intra-GPU multi-process scheduling according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of real-time feedback of an optional decision-making model according to an embodiment of this application;

[0025] Figure 8 This is a structural block diagram of an optional multimodal task scheduling and execution device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] The method embodiments provided in this application can be executed in a scheduling engine or similar computing system. Taking running on a scheduling engine as an example, Figure 1 This is a hardware structure block diagram of the scheduling engine for a multimodal task scheduling and execution method according to an embodiment of this application. For example... Figure 1 As shown, the scheduling engine may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. In one exemplary embodiment, the scheduling engine may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the scheduling engine described above. For example, the scheduling engine may also include... Figure 1 The more or fewer components shown, or having the same Figure 1Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the missing parameter repair method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a secure text network via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission system 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the scheduling engine. In one example, the transmission system 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet.

[0031] This embodiment provides a method for scheduling and executing multimodal tasks. Figure 2 This is a flowchart of an optional multimodal task scheduling and execution method according to an embodiment of this application. The method includes the following steps:

[0032] Step S202: Receive multiple multimodal data reported by the terminal device, and route multiple multimodal tasks belonging to the target user to the same gateway node according to the user identifier of the multiple multimodal data. The multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different.

[0033] Step S204: Deploy the multiple multimodal tasks to multiple computing processes of the GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, which are used to execute multimodal tasks of different task types respectively.

[0034] Step S206: Receive the execution status of the plurality of computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information;

[0035] Step S208: Dynamically adjust the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes the computing processes of the multiple GPU servers.

[0036] The above scheme first receives multimodal data, including video and audio sensor signals, reported from terminal devices (such as cameras, wristbands, door sensors, etc.). Based on user identifiers, a consistent hashing algorithm is used to route all multimodal tasks belonging to the same target user to the same gateway node, ensuring physical consistency of task processing. Then, the gateway node deploys these different task types (such as task detection, audio recognition, and behavior analysis) to corresponding computing processes on the GPU server for parallel execution. Simultaneously, it continuously receives real-time execution status reports from each computing process, including expected processing time for a single task and resource pressure information. Based on this, the system dynamically analyzes the time difference between task completions and adjusts the resource allocation ratio of different types of task processes in the process pool according to the time difference and execution status to shorten processing time. This achieves fine-grained computing power sharing and task completion time convergence across server process pools without increasing hardware, ultimately achieving consistency of multimodal task results and coordinated optimization of system resource utilization. Furthermore, it solves the technical problem in distributed multimodal large-model parallel processing systems where the difference in computation time between different modal tasks leads to inconsistent arrival times of results at decision nodes, resulting in decision delays and low resource utilization.

[0037] Optionally, routing multiple multimodal tasks belonging to a target user to the same gateway node based on the user identifiers of the multiple multimodal data includes: generating a fixed-length hash value from the user identifiers of the target user using a hash function; mapping the hash value to a target virtual node among multiple virtual nodes on a virtual ring, wherein each of the multiple gateway nodes has K virtual nodes evenly distributed on the virtual ring, where K is a positive integer, and each of the multiple virtual nodes is bound to the IP address and port number of a corresponding GPU server; and routing the multiple multimodal tasks to the gateway node based on the IP address and port number bound to the target virtual node.

[0038] Based on the unique identifier of the target user, a fixed-length hash value is generated using a consistent hashing algorithm. This hash value locates a target virtual node on the virtual ring. Each physical gateway node on the virtual ring has K virtual nodes evenly distributed, where K is a pre-set number of replicas to ensure load balancing and smooth migration in case of node failure. Each virtual node is bound to the IP address and port of a gateway server, directly corresponding to the actual running GPU server. When the hash value falls within a certain range of the virtual ring, it locks the real gateway it is bound to. All multimodal tasks belonging to this user, such as video, audio, and sensor tasks, are thus uniformly routed to the same edge gateway, avoiding network latency and state fragmentation caused by cross-node scheduling, and ensuring localized closed-loop processing of the task chain.

[0039] This embodiment achieves high consistency and high availability of task routing through a virtual node mapping mechanism. Even if some gateway nodes fail, the system can still automatically and smoothly migrate user tasks to their replica nodes, which not only maintains the timing consistency of multimodal task processing, but also improves the overall disaster recovery capability and scheduling stability of the cluster.

[0040] Optionally, dynamically adjusting the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes includes: when the resource pressure information indicates that the computing resource pressure of the process pool is less than a first preset threshold, predicting the time interval in which the task completion time of the multiple multimodal tasks is located based on the expected processing time, and predicting a first time difference in the task completion time of the multiple multimodal tasks based on the length of the time interval; when the first time difference is greater than a second preset threshold, determining a first task type of the first multimodal task and a second task type of the second multimodal task, and determining a first process type corresponding to the first task type and a second process type corresponding to the second task type, wherein the first multimodal task is a task whose task completion time is located at the beginning of the time interval, the second multimodal task is a task whose task completion time is located at the end of the time interval, and the multiple types of task processes include the first process type and the second process type; switching M computing processes of the first process type in the process pool to the second process type, wherein M is a positive integer.

[0041] When the overall resource pressure of the process pool is below the first threshold, the system calculates the completion time distribution range of all multimodal tasks based on the expected processing time of each task. The range span is the task completion time difference. If the time difference exceeds the second threshold, the system identifies the first task type (e.g., audio detection) to which the earliest completed task belongs, and the second task type (e.g., video detection) to which the latest completed task belongs. These correspond to the first process type with low load and the second process type with high load, respectively. The system randomly selects M idle or low-load audio detection processes from the process pool, reconfigures their model context and computing resources as video detection processes, and uses GPU virtualization technology to reuse video memory and CUDA streams. This allows for dynamic switching of task types without restarting the service, enabling real-time migration of computing power from surplus mode to bottleneck mode.

[0042] This embodiment intelligently identifies task timing differences and dynamically redefines process functions, proactively compressing the time difference in multimodal task completion without adding new hardware, thus significantly improving system response consistency and resource utilization efficiency.

[0043] Optionally, after switching the M computing processes of the first process type in the process pool to the second process type, the method further includes: receiving the execution status of the plurality of computing processes reported by the GPU server again, and re-predicting the second time difference of the task completion time of the plurality of multimodal tasks; determining whether the second time difference is greater than the second preset threshold; and if the second time difference is greater than the second preset threshold, adjusting the resource allocation ratio of different types of task processes in the process pool again.

[0044] After the resource switch is completed, the system continuously receives the process execution status reported by the GPU server, updates the expected time and resource load of each task in real time, and recalculates the second time difference of the completion time of the multimodal task. If the difference is still higher than the second threshold, it means that the initial resource reallocation has not completely eliminated the timing deviation. The system automatically triggers a second adjustment to continue to migrate computing power from low-load process types to high-load processes, forming multiple rounds of closed-loop optimization. Each adjustment is based on the latest real-time data, without relying on preset models or manual intervention, and the computing power rebalancing is completed entirely autonomously by the system.

[0045] This embodiment achieves gradual convergence of task completion time difference through a multi-round dynamic feedback mechanism, ensuring that the system can maintain high consistency processing capability under complex load fluctuations, and improving the robustness and long-term stability of scheduling.

[0046] In an exemplary embodiment, dynamically adjusting the resource allocation ratio of different types of task processes in the process pool according to the execution status of the plurality of computing processes includes: when the resource pressure information indicates that the computing resource pressure of the process pool is greater than a first preset threshold, determining the first allocation ratio of different types of task processes in the process pool, and determining the task pressure of different types of task processes; and adjusting the first allocation ratio to a second allocation ratio according to the task pressure.

[0047] When the overall resource pressure of the process pool exceeds the first threshold, the system immediately assesses the current resource usage ratio and actual task backlog intensity of each task type process. For example, video detection suffers from a sustained high load on video memory and computing due to image encoding / decoding and high-resolution inference, while audio recognition is in a low-load state for a long time due to the lightweight model. Based on this, the system shifts the originally equal or statically allocated resource ratio to high-pressure task processes, compresses the CUDA cores and video memory quotas of low-load processes, and injects the released computing power into bottleneck processes such as video or sensor fusion in real time, realizing dynamic weight reconstruction based on the real load. The entire process does not require service interruption and relies on GPU virtualization and process-level context isolation to complete resource reallocation.

[0048] This embodiment prioritizes the computing power of critical tasks when resources are scarce, avoiding overall response delays caused by long-tail tasks blocking the system, and significantly improving task throughput and service availability under high load scenarios.

[0049] In an exemplary embodiment, after adjusting the first allocation ratio to the second allocation ratio according to the task pressure, the method further includes: classifying the multiple GPU servers into high-pressure servers and low-pressure servers according to the computing resource pressure of the multiple GPU servers, wherein the multiple GPU servers correspond one-to-one with the multiple gateway nodes; predicting the time interval in which the task completion time of the multiple multimodal tasks is located based on the expected processing time, and predicting a third time difference in the task completion time of the multiple multimodal tasks based on the length of the time interval; and performing task migration on the multiple multimodal tasks based on the third time difference.

[0050] After optimizing the process ratio within a single server, the system will further evaluate the load status of each GPU server horizontally. The GPU servers bound to the gateway node will be divided into high-load and low-load categories according to their computing pressure. The system will calculate the completion time span of the current multimodal task chain based on the expected processing time of each task. If the third time difference still exceeds the tolerance threshold, the system will allocate all multimodal tasks of the same user to processes with the same pressure level, either all slow or all fast, based on the user and task type of the task.

[0051] This embodiment eliminates the imbalance of processing capabilities between nodes at the cluster level by cross-server task load balancing and time-difference driven intelligent migration, thereby achieving global convergence of end-to-end completion time for multimodal tasks and optimization of overall system throughput.

[0052] In an optional embodiment, the task migration of the plurality of multimodal tasks based on the third time difference includes: determining the server category of the GPU server where the plurality of multimodal tasks are currently located; if the third time difference is greater than a third preset threshold and the server category is a low-pressure server, migrating the plurality of multimodal tasks to a first GPU server for execution, wherein the first GPU server is a high-pressure server; if the third time difference is less than or equal to the third preset threshold and the server category is a high-pressure server, migrating the plurality of multimodal tasks to a second GPU server for execution, wherein the second GPU server is a low-pressure server.

[0053] When computing resources are under pressure, multimodal computing tasks from the same user are assigned to processes with similar pressure levels, either all running at a slower or faster pace. For example, if there are multimodal tasks from user 1 and user 2, user 1's tasks can be assigned to nodes with lower pressure levels, and user 2's tasks to nodes with higher pressure levels, ensuring consistent computing speed and maximizing system performance. If user 1 and user 2's multimodal tasks are interleaved between processes with lower and higher pressure levels, both users' tasks will have to wait for slower tasks in the decision engine, resulting in a decrease in overall system real-time performance.

[0054] Traditional multimodal detection tasks involve sequential multimodal detection after data reporting, such as... Figure 3 As shown, this method has a slow detection speed and a pipelined node-based processing approach, making it difficult to meet the requirements of high real-time tasks.

[0055] To address this issue, related technologies propose using multimodal parallel computing for detection to accelerate computation. A schematic diagram of the parallel computing is shown below. Figure 4 As shown, however, under parallel computing and distributed systems, cloud-based multimodal task detection struggles to guarantee the consistency of multimodal task processing time and arrival at the decision model. For example, audio detection takes 1 second, video detection takes 3 seconds, and other sensor data takes 0.5 seconds (the processing time of different processes varies). Figure 6 Service discovery reporting nodes and Figure 7 (The decision model feedback in the process) The decision model needs to wait until all multimodal data detection is completed before performing post-processing based on the results of the multimodal task, similar to the short plank effect of a barrel.

[0056] This application embodiment achieves result consistency assurance for multimodal tasks through service registration and discovery, gateway scheduling engine design, and GPU virtualization, and fully utilizes computing resources such as GPUs to accelerate tasks and reduce resource costs. Specifically, it includes the following steps:

[0057] Step 1: After an event is triggered by a terminal data collection device, such as a camera, door sensor, access control system, or wristband device, this embodiment takes camera motion detection as an example to realize data reporting and data storage, including but not limited to storage on local disk, memory, or cloud storage.

[0058] Step 2: Based on the decision model dimension, such as using consistent hashing or other consistency algorithms with usernames, assign the user's current multimodal task to the same gateway. Execute the distributed task within the gateway to ensure the results reach the decision model as consistently as possible. A diagram illustrating the allocation is shown below. Figure 5 As shown.

[0059] Step 3: Multimodal detection nodes (e.g., video detection, audio detection, other door magnetic data) report processing time and load bit. For example, audio detection takes 1 second, video detection takes 3 seconds, and other sensor data takes 0.5 seconds. During detection, resources for audio and other sensor speeds can be appropriately reduced, or some of their processes can be loaded and scheduled as algorithm nodes with longer processing times. This improves the time it takes for multiple results to reach the decision model during multimodal parallel computation, thus achieving result consistency. See steps 4-7 for details on process loading and scheduling.

[0060] Step 4: In multimodal task processing, the resource consumption of various algorithms varies significantly. For example, there are different computational resource focuses such as GPU resources, CPU resources, video memory resources, and system memory resources. Deploying a single algorithm on the same computational resource cannot fully utilize the resource. Therefore, computational resources are virtualized, i.e., GPU virtualization. This allows N algorithm processes to run on the same GPU server, supporting multiple GPU scheduling. A diagram is shown below. Figure 6 As shown.

[0061] Step 5: Utilize GPU resource virtualization. Through script and process information services for discovery and registration, the gateway scheduling service understands the computational performance and pressure of each GPU and the algorithms within it. For example, in a multi-model service, there are 4 video algorithm processes, 3 audio algorithm processes, and 3 processes for other sensor data types. If video detection takes longer than audio algorithm detection, some audio algorithm processes can be switched to video detection processes. The same principle applies to scheduling between GPUs. Specifically, process scheduling aims to maximize computational resources by combining and deploying different processes—GPU-intensive, CPU-intensive, and memory-intensive—within a single-GPU server. Furthermore, the process resources across multiple GPUs are pooled to maximize resource utilization.

[0062] Step 6: When computing resources are under pressure, distribute multimodal computing tasks of the same user to processes with similar pressure levels, either allocating them to slower or faster processes. For example, if there are multimodal tasks for user 1 and user 2, distribute user 1's tasks to nodes with lower pressure levels, and user 2's tasks to nodes with higher pressure levels, ensuring consistent computing speed and maximizing system performance. If user 1 and user 2's multimodal tasks are interleaved between processes with lower and higher pressure levels, both users' tasks will have to wait for slower tasks in the decision engine, causing a decrease in overall system real-time performance.

[0063] Step 7: In the decision model or engine, perform statistics based on the multimodal task dimension, including the average time spent on a single task category, the multimodal tasks of a single user, and the time difference between the arrival of the first task and the arrival of the last task (e.g., ...). Figure 7 Data such as (as shown) is used to provide feedback to the gateway scheduling service, achieving closed-loop control of task scheduling and computing resource scheduling. This implements a parallel detection and computing scheduling method based on GPU virtualization and a multimodal, multi-model result consistency memory model, which ensures resource utilization, task real-time performance, and real-time feedback closed-loop control.

[0064] Based on this embodiment, the following core improvements are proposed:

[0065] 1. By parallel computing of distributed multimodal large model tasks, the computation of multimodal tasks is accelerated. Through the consensus algorithm of multimodal tasks, tasks are allocated to the same gateway to achieve consistency of results reaching the decision engine, thereby further ensuring the real-time performance of multimodal large model tasks.

[0066] 2. By virtualizing GPU resources, different processes that consume GPU, CPU, and video memory are combined and deployed in a single GPU server. By utilizing service discovery and scheduling mechanisms and pooling process resources across multiple GPUs, resource utilization can be maximized.

[0067] 3. By aggregating the computational resources and time consumption of multimodal parallel tasks in the decision engine, and the difference in the arrival time of the results of multiple parallel tasks of a single multimodal task, real-time feedback is provided to achieve closed-loop scheduling control.

[0068] Based on the above core improvements, this application achieves the following optimization effects:

[0069] 1. Improved Resource Utilization: To ensure service performance and effectiveness, 80% server performance utilization is optimal. In standalone deployment, video algorithms require image encoding and decoding, resulting in high CPU usage. Processing 10,000 tasks per minute requires 40% GPU and 80% CPU, essentially reaching performance stress levels. Audio algorithms primarily utilize GPUs; therefore, processing 20,000 tasks per minute requires 80% GPU and 10% CPU resources. Based on a standalone deployment architecture, processing 40,000 modal tasks requires 4 GPU computing servers for video tasks and 2 for audio detection, totaling 6 GPU computing servers. In a hybrid deployment, GPU computing servers can handle 7,529 audio tasks and 7,529 video tasks, requiring only 5.32 GPU computing servers to handle the same workload, thus improving server resource utilization.

[0070] 2. Improved Processing Time: For example, processing three tasks (video, audio, and other sensors) on GPU1 takes 3 seconds, 2 seconds, and 5.5 seconds respectively, while on GPU2, the same amounts take 3 seconds, 4 seconds, and 2.5 seconds. In traditional allocation algorithms, allocating all tasks for User 1 to GPU1 and all tasks for User 2 to GPU2 results in a processing time of 5.5 seconds for User 1 and 4 seconds for Task 2. With scheduling optimization, optimal allocation of tasks for User 1 is ensured, reducing the processing time for User 1 to 3 seconds and Task 2 to 5.5 seconds. This reduces the average processing time of the system and further improves real-time performance by allocating some resources to algorithm processes with significant long-tail tasks.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0072] Figure 8 This is a structural block diagram of a multimodal task scheduling and execution apparatus according to an embodiment of this application. The apparatus includes:

[0073] The routing module 82 is used to receive multiple multimodal data reported by the terminal device, and to route multiple multimodal tasks belonging to the target user to the same gateway node according to the user identifier of the multiple multimodal data. The multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different.

[0074] The deployment module 84 is used to deploy the multiple multimodal tasks to multiple computing processes of the GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, and the multiple types of task processes are used to execute multimodal tasks of different task types respectively.

[0075] The receiving module 86 is used to receive the execution status of the plurality of computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information;

[0076] The adjustment module 88 is used to dynamically adjust the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes the computing processes of the multiple GPU servers.

[0077] The above scheme first receives multimodal data, including video and audio sensor signals, reported from terminal devices (such as cameras, wristbands, door sensors, etc.). Based on user identifiers, a consistent hashing algorithm is used to route all multimodal tasks belonging to the same target user to the same gateway node, ensuring physical consistency of task processing. Then, the gateway node deploys these different task types (such as task detection, audio recognition, and behavior analysis) to corresponding computing processes on the GPU server for parallel execution. Simultaneously, it continuously receives real-time execution status reports from each computing process, including expected processing time for a single task and resource pressure information. Based on this, the system dynamically analyzes the time difference between task completions and adjusts the resource allocation ratio of different types of task processes in the process pool according to the time difference and execution status to shorten processing time. This achieves fine-grained computing power sharing and task completion time convergence across server process pools without increasing hardware, ultimately achieving consistency of multimodal task results and coordinated optimization of system resource utilization. Furthermore, it solves the technical problem in distributed multimodal large-scale parallel processing systems where the difference in computation time between different modal tasks leads to inconsistent arrival times of results at decision nodes, resulting in decision delays and low resource utilization.

[0078] In an exemplary embodiment, the routing module 82 is further configured to: generate a fixed-length hash value from the user identifier of the target user using a hash function; map the hash value to a target virtual node among multiple virtual nodes on a virtual ring, wherein each of the multiple gateway nodes has K virtual nodes evenly distributed on the virtual ring, where K is a positive integer, and each of the multiple virtual nodes is bound to the IP address and port number of a corresponding GPU server; and route the multiple multimodal tasks to the gateway node according to the IP address and port number bound to the target virtual node.

[0079] In an exemplary embodiment, the adjustment module 88 is further configured to: predict the time interval in which the task completion time of the plurality of multimodal tasks is located based on the expected processing time when the resource pressure information indicates that the computing resource pressure of the process pool is less than a first preset threshold; predict a first time difference in the task completion time of the plurality of multimodal tasks based on the length of the time interval when the resource pressure information indicates that the computing resource pressure of the process pool is less than a first preset threshold; determine a first task type of the first multimodal task and a second task type of the second multimodal task when the first time difference is greater than a second preset threshold; determine a first process type corresponding to the first task type and a second process type corresponding to the second task type, wherein the first multimodal task is a task whose task completion time is located at the beginning of the time interval, the second multimodal task is a task whose task completion time is located at the end of the time interval, and the multi-type task process includes the first process type and the second process type; and switch M computing processes of the first process type in the process pool to the second process type, wherein M is a positive integer.

[0080] In an exemplary embodiment, the adjustment module 88 is further configured to receive the execution status of the plurality of computing processes reported by the GPU server again, and re-predict the second time difference of the task completion time of the plurality of multimodal tasks; determine whether the second time difference is greater than the second preset threshold; and if the second time difference is greater than the second preset threshold, adjust the resource allocation ratio of different types of task processes in the process pool again.

[0081] In an exemplary embodiment, the adjustment module 88 is further configured to, when the resource pressure information indicates that the computing resource pressure of the process pool is greater than a first preset threshold, determine the first allocation ratio of different types of task processes in the process pool, and determine the task pressure of different types of task processes; and adjust the first allocation ratio to a second allocation ratio according to the task pressure.

[0082] In an exemplary embodiment, the adjustment module 88 is further configured to classify the multiple GPU servers into high-pressure servers and low-pressure servers based on the computing resource pressure of the multiple GPU servers, wherein the multiple GPU servers correspond one-to-one with the multiple gateway nodes; predict the time interval in which the task completion time of the multiple multimodal tasks is located based on the expected processing time, predict a third time difference of the task completion time of the multiple multimodal tasks based on the length of the time interval; and perform task migration of the multiple multimodal tasks based on the third time difference.

[0083] In an exemplary embodiment, the adjustment module 88 is further configured to determine the server category of the GPU server where the plurality of multimodal tasks are currently located; if the third time difference is greater than a third preset threshold and the server category is a low-pressure server, the plurality of multimodal tasks are migrated to a first GPU server for execution, wherein the first GPU server is a high-pressure server; if the third time difference is less than or equal to the third preset threshold and the server category is a high-pressure server, the plurality of multimodal tasks are migrated to a second GPU server for execution, wherein the second GPU server is a low-pressure server.

[0084] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0085] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0086] S1, Receive multiple multimodal data reported by the terminal device, and route multiple multimodal tasks belonging to the target user to the same gateway node according to the user identifier of the multiple multimodal data, wherein the multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different;

[0087] S2, the multiple multimodal tasks are deployed to multiple computing processes of the GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, which are used to execute multimodal tasks of different task types respectively;

[0088] S3, receive the execution status of the multiple computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information;

[0089] S4. Dynamically adjust the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes the computing processes of the multiple GPU servers.

[0090] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0091] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0092] Embodiments of this application also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0093] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0094] S1, Receive multiple multimodal data reported by the terminal device, and route multiple multimodal tasks belonging to the target user to the same gateway node according to the user identifier of the multiple multimodal data, wherein the multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different;

[0095] S2, the multiple multimodal tasks are deployed to multiple computing processes of the GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, which are used to execute multimodal tasks of different task types respectively;

[0096] S3, receive the execution status of the multiple computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information;

[0097] S4. Dynamically adjust the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes the computing processes of the multiple GPU servers.

[0098] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0099] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0100] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps when executed by the processor:

[0101] S1, Receive multiple multimodal data reported by the terminal device, and route multiple multimodal tasks belonging to the target user to the same gateway node according to the user identifier of the multiple multimodal data, wherein the multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different;

[0102] S2, the multiple multimodal tasks are deployed to multiple computing processes of the GPU server for concurrent execution through the gateway node, wherein the multiple computing processes include multiple types of task processes, which are used to execute multimodal tasks of different task types respectively;

[0103] S3, receive the execution status of the multiple computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information;

[0104] S4. Dynamically adjust the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes the computing processes of the multiple GPU servers.

[0105] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0106] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0107] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for scheduling and executing multimodal tasks, characterized in that, include: The system receives multiple multimodal data reports from terminal devices and routes multiple multimodal tasks belonging to the target user to the same gateway node based on the user identifiers of the multiple multimodal data. The multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different. The gateway node deploys the multiple multimodal tasks to multiple computing processes on the GPU server for concurrent execution. The multiple computing processes include multiple types of task processes, which are used to execute multimodal tasks of different task types respectively. The system receives the execution status of the plurality of computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information; The resource allocation ratio of different types of task processes in the process pool is dynamically adjusted according to the execution status of the multiple computing processes in order to shorten the time difference between the task completion times of the multiple multimodal tasks. The process pool includes the computing processes of the multiple GPU servers.

2. The method according to claim 1, characterized in that, Based on the user identifiers of the multiple multimodal data, multiple multimodal tasks belonging to the target user are routed to the same gateway node, including: The user identifier of the target user is used to generate a fixed-length hash value through a hash function; The hash value is mapped to a target virtual node among multiple virtual nodes on the virtual ring. Each of the multiple gateway nodes has K virtual nodes evenly distributed on the virtual ring, where K is a positive integer. Each of the multiple virtual nodes is bound to the IP address and port number of the corresponding GPU server. The multiple multimodal tasks are routed to the gateway node based on the IP address and port number bound to the target virtual node.

3. The method according to claim 1, characterized in that, Dynamically adjust the resource allocation ratio of different types of task processes in the process pool based on the execution status of the multiple computing processes, including: When the resource pressure information indicates that the computational resource pressure of the process pool is less than a first preset threshold, the time interval in which the task completion time of the multiple multimodal tasks is located is predicted based on the expected processing time, and the first time difference of the task completion time of the multiple multimodal tasks is predicted based on the length of the time interval. When the first time difference is greater than the second preset threshold, the first task type of the first multimodal task and the second task type of the second multimodal task are determined, as well as the first process type corresponding to the first task type and the second process type corresponding to the second task type are determined. The first multimodal task is a task whose task completion time is located at the beginning of the time interval, and the second multimodal task is a task whose task completion time is located at the end of the time interval. The multi-type task process includes the first process type and the second process type. Switch M computing processes of the first process type in the process pool to the second process type, where M is a positive integer.

4. The method according to claim 3, characterized in that, After switching M computing processes of the first process type in the process pool to the second process type, the method further includes: The execution status of the multiple computing processes reported by the GPU server is received again, and the second time difference of the task completion time of the multiple multimodal tasks is re-predicted. Determine whether the second time difference is greater than the second preset threshold; If the second time difference is greater than the second preset threshold, the resource allocation ratio of different types of task processes in the process pool is adjusted again.

5. The method according to claim 1, characterized in that, Dynamically adjust the resource allocation ratio of different types of task processes in the process pool based on the execution status of the multiple computing processes, including: When the resource pressure information indicates that the computational resource pressure of the process pool is greater than a first preset threshold, the first allocation ratio of different types of task processes in the process pool is determined, and the task pressure of different types of task processes is determined. The first allocation ratio is adjusted to the second allocation ratio based on the task pressure.

6. The method according to claim 5, characterized in that, After adjusting the first allocation ratio to the second allocation ratio based on the task pressure, the method further includes: The GPU servers are classified into high-pressure servers and low-pressure servers based on their computing resource pressure, wherein each GPU server corresponds one-to-one with a gateway node. Based on the expected processing time, predict the time interval in which the task completion time of the multiple multimodal tasks falls, and based on the length of the time interval, predict the third time difference of the task completion time of the multiple multimodal tasks. The multiple multimodal tasks are migrated based on the third time difference.

7. The method according to claim 6, characterized in that, Based on the third time difference, task migration is performed on the multiple multimodal tasks, including: Determine the server category of the GPU server where the multiple multimodal tasks are currently located; If the third time difference is greater than the third preset threshold and the server type is a low-pressure server, the multiple multimodal tasks are migrated to the first GPU server for execution, wherein the first GPU server is a high-pressure server. If the third time difference is less than or equal to the third preset threshold and the server category is a high-pressure server, the multiple multimodal tasks are migrated to the second GPU server for execution, wherein the second GPU server is a low-pressure server.

8. A scheduling and execution device for multimodal tasks, characterized in that, include: The routing module is used to receive multiple multimodal data reported by the terminal device, and to route multiple multimodal tasks belonging to the target user to the same gateway node according to the user identifier of the multiple multimodal data. The multiple multimodal tasks are all used to process multimodal data, and the task types of the multiple multimodal tasks are different. The deployment module is used to deploy the multiple multimodal tasks to multiple computing processes of the GPU server for concurrent execution through the gateway node. The multiple computing processes include multiple types of task processes, which are used to execute multimodal tasks of different task types respectively. The receiving module is used to receive the execution status of the multiple computing processes reported by the GPU server, wherein the execution status includes: expected processing time and resource pressure information; An adjustment module is used to dynamically adjust the resource allocation ratio of different types of task processes in the process pool according to the execution status of the multiple computing processes, so as to shorten the time difference between the task completion times of the multiple multimodal tasks, wherein the process pool includes the computing processes of the multiple GPU servers.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images