Method and apparatus for processing multi-modal data, electronic device, and storage medium

Abstract

Embodiments of the application disclose a multi-modal data processing method and device, electronic equipment and storage medium, and belong to the technical field of data processing. The method comprises the following steps: selecting a plurality of intermediate processors from all candidate processors according to the peak resource requirement of each function module and the residual resource capacity of all candidate processors; obtaining the running load characteristics of each function module and the hardware performance index of each intermediate processor; determining the running bottleneck time of each function module based on each intermediate processor according to the running load characteristics of each function module and the hardware performance index of each intermediate processor; selecting the target processor of each function module from the plurality of intermediate processors according to the plurality of running bottleneck times of each function module; and deploying each function module to the corresponding target processor, and obtaining the data processing result by executing the to-be-processed data based on each deployed target processor. The application can improve the processing efficiency of multi-modal data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method, apparatus, electronic device, and storage medium for processing multimodal data. Background Technology

[0002] Because multimodal large models overcome the limitations of traditional models that can only process single text or single images, they have been widely used in various fields such as industrial autonomous driving, intelligent security, and medical image diagnosis. Related technologies often employ a general inference framework to process the input data when handling multimodal data.

[0003] However, the input data for multimodal large models typically includes various heterogeneous formats such as text and images, and these heterogeneous data have drastically different hardware computing power requirements at different inference stages. General-purpose framework inference in related technologies can prevent processors from achieving optimal utilization, leading to low processing efficiency for multimodal data. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for processing multimodal data, which can improve the processing efficiency of multimodal data.

[0005] To achieve the above objectives, one embodiment of this application provides a method for processing multimodal data, including: Obtain the data to be processed and the remaining resource capacity of each candidate processor, and determine the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed. For each functional module, multiple intermediate processors are selected from all candidate processors based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors. Obtain the runtime load characteristics of each functional module and the hardware performance indicators of each intermediate processor; For each functional module, based on the operational load characteristics of the functional module and the hardware performance indicators of each intermediate processor, the bottleneck time of the functional module running on each intermediate processor is determined. For each functional module, the intermediate processor corresponding to the bottleneck time with the smallest value among multiple bottleneck times is selected as the target processor of the functional module. Each functional module is deployed to the corresponding target processor. The data to be processed is executed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0006] In some embodiments, determining the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed includes: Based on the data type to be processed, determine the total number of parameters and the parameter precision of each functional module in the multimodal large model; Based on the total number and precision of parameters, the weight requirements of each functional module are determined. Based on the data type to be processed, determine the acceptable input data resolution and acceptable intermediate feature map size for each functional module in the multimodal large model; Based on the acceptable input data resolution and acceptable intermediate feature map size, determine the maximum activation value requirement for each functional module; Based on the data type of the data to be processed, determine the cache usage requirements of each functional module in the multimodal large model; Based on the weight usage requirements, maximum activation value usage requirements, and cache usage requirements, the peak resource requirements of each functional module in the multimodal large model are determined.

[0007] In some embodiments, multiple intermediate processors are selected from all candidate processors based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors, including: Multiple first processors are determined based on the remaining resource capacity of each candidate processor and the peak resource requirements of the functional modules; Obtain the maximum redundancy ratio and maximum latency upper bound of the functional modules; The required throughput of the functional modules is determined based on the ratio between the operating load characteristics and the upper limit of the maximum latency. The computing power redundancy ratio when the functional module is deployed on each first processor is determined based on the ratio between each hardware performance indicator and the required throughput. For each first processor, when the computing power redundancy ratio corresponding to the first processor is less than or equal to the maximum redundancy ratio, the first processor is used as an intermediate processor to obtain multiple intermediate processors.

[0008] In some embodiments, a plurality of first processors are determined based on the remaining resource capacity of candidate processors and the peak resource requirements of functional modules, including: Obtain the safety margin factor; The peak resource requirements of the update function module are obtained by updating the peak resource requirements based on the safety margin coefficient; When the remaining resource capacity of a candidate processor is equal to or greater than the updated peak resource requirement, the candidate processor is selected as the first processor to obtain multiple first processors.

[0009] In some embodiments, hardware performance metrics include hardware throughput and hardware bandwidth, and runtime load characteristics include total computational load characteristics and total memory access characteristics. Based on the operational load characteristics of the functional modules and the hardware performance indicators of each intermediate processor, the bottleneck time of the functional modules running on each intermediate processor is determined, including: Based on the ratio between the total computational load characteristics and the hardware throughput of each intermediate processor, the estimated computation time when the functional module is deployed on each intermediate processor is obtained; Based on the ratio between the total memory access characteristics and the hardware bandwidth of each intermediate processor, the estimated memory access time when the functional module is deployed on each intermediate processor is obtained. From all the estimated computation time and all the estimated memory access time, the time with the largest value is selected as the bottleneck time of the functional module.

[0010] In some embodiments, selecting the intermediate processor corresponding to the smallest runtime bottleneck time from a plurality of runtime bottleneck times as the target processor of the functional module includes: When there are at least two intermediate processors with the same bottleneck time, obtain the arithmetic strength value of the functional module; For each intermediate processor with the same runtime bottleneck time, obtain the hardware inflection point value for each intermediate processor; Based on the arithmetic strength value and each hardware inflection point value, determine the hardware utilization efficiency of the functional module running on each intermediate processor. The intermediate processor corresponding to the highest hardware utilization efficiency value among multiple hardware utilization efficiency values ​​is selected as the target processor of the functional module.

[0011] In some embodiments, the data to be processed is executed based on each target processor after deployment to obtain the data processing result corresponding to the data to be processed, including: Obtain the module topology information between all functional modules, with the goal of maximizing throughput, and determine the pipeline information of each functional module based on the module topology information; Based on the pipeline information of each functional module and the data to be processed by each deployed target processor, the data processing results corresponding to the data to be processed are obtained.

[0012] To achieve the above objectives, one embodiment of this application provides a multimodal data processing apparatus, including: The first acquisition module is used to acquire the data to be processed and the remaining resource capacity of each candidate processor, and to determine the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed. The intermediate processor determination module is used to select multiple intermediate processors from all candidate processors for each functional module, based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors. The second acquisition module is used to acquire the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor. The bottleneck time determination module is used to determine the bottleneck time of each functional module on each intermediate processor based on the functional module's operating load characteristics and the hardware performance indicators of each intermediate processor. The target processor determination module is used to select the intermediate processor corresponding to the smallest running bottleneck time from multiple running bottleneck times as the target processor of the functional module for each functional module. The data processing result determination module is used to deploy each functional module to the corresponding target processor, execute the data to be processed based on each deployed target processor, and obtain the data processing result corresponding to the data to be processed.

[0013] To achieve the above objectives, one aspect of this application provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the steps in the multimodal data processing method provided in this application.

[0014] To achieve the above objectives, one aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps in the multimodal data processing method provided in this application.

[0015] To achieve the above objectives, one aspect of this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the multimodal data processing method provided in this application.

[0016] The multimodal data processing method, apparatus, electronic device, and storage medium proposed in this application involve: acquiring the data to be processed and the remaining resource capacity of each candidate processor; determining the peak resource requirement of each functional module in the multimodal large model based on the data type of the data to be processed; selecting multiple intermediate processors from all candidate processors based on the peak resource requirement of the functional module and the remaining resource capacity of all candidate processors for each functional module; acquiring the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor; determining the operating bottleneck time of each functional module on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor for each functional module; selecting the intermediate processor corresponding to the minimum operating bottleneck time from multiple operating bottleneck times as the target processor for the functional module; deploying each functional module to the corresponding target processor; and executing the data to be processed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0017] This application embodiment obtains intermediate processors by initially screening the peak resource requirements of each functional module and the remaining capacity of candidate processors. It then quantitatively evaluates the corresponding bottleneck time by combining the operational load characteristics of the functional modules with the hardware performance indicators of the intermediate processors. This allows for the selection and deployment of the most efficient target processor for each functional module, followed by data processing based on the deployed target processors. In this way, this application embodiment achieves precise adaptation between multimodal functional modules and heterogeneous hardware, effectively avoiding high-end hardware mismatch and idle existing heterogeneous resources. While maximizing the computing power utilization of each target processor and the overall hardware resource utilization of the system, it reduces the processing latency of heterogeneous data, thereby significantly improving the end-to-end data processing efficiency of large multimodal models on heterogeneous platforms.

[0018] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a schematic diagram of the system framework corresponding to the multimodal data processing method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the multimodal data processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the module structure of the multimodal data processing device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that in each specific embodiment of this application, when it is necessary to obtain the remaining resource capacity of each candidate processor, permission or consent from relevant personnel in the management scenario will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of relevant personnel, separate permission or consent from the relevant personnel will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the separate permission or consent of the relevant personnel will the necessary remaining resource capacity information for the normal operation of this application embodiment be obtained. Other data obtained in this application embodiment are all authorized and legal data, and will not be described in detail here.

[0023] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer computer devices, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0024] The technical problems existing in the related technologies are as follows: Because multimodal large models overcome the limitations of traditional models that can only process single text or single images, they have been widely used in various fields such as industrial autonomous driving, intelligent security, and medical image diagnosis. Related technologies often employ a general inference framework to process the input data when handling multimodal data.

[0025] However, the input data for multimodal large models typically includes various heterogeneous formats such as text and images, and these heterogeneous data have drastically different hardware computing power requirements at different inference stages. General-purpose framework inference in related technologies can prevent processors from achieving optimal utilization, leading to low processing efficiency for multimodal data.

[0026] For example, in scenarios requiring heterogeneous hardware collaborative computing, such as multimodal text-to-graph and high-density text-to-graph semantic interactions, inference systems need to leverage efficient stage separation and hardware affinity matching architectures to achieve high throughput and low latency. Because related technologies only deploy different modal inference tasks coarsely and uniformly based on a general single hardware scheduling logic, this approach often fails to accurately adapt to the differentiated requirements of various functional modules (such as lightweight text preprocessing and computationally intensive core generation) for computational throughput and memory bandwidth. This results in significant idle computing power or extremely uneven load on the target processor, especially in high-load complex multimodal interaction scenarios, such as high-frequency multimodal task concurrency, iterative computation of large-resolution latent space feature maps, or multimodal result aggregation. The coupled execution mechanisms used in these technologies often fail to effectively isolate cross-stage data flow conflicts, leading to severe hardware resource contention and pipeline scheduling blockages. This can even trigger memory overflow at critical nodes, causing system service interruptions, ultimately resulting in severely low overall deployment and end-to-end inference efficiency for large multimodal models on heterogeneous platforms.

[0027] The multimodal data processing method, apparatus, electronic device, and storage medium proposed in this application involve: acquiring the data to be processed and the remaining resource capacity of each candidate processor; determining the peak resource requirement of each functional module in the multimodal large model based on the data type of the data to be processed; selecting multiple intermediate processors from all candidate processors based on the peak resource requirement of the functional module and the remaining resource capacity of all candidate processors for each functional module; acquiring the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor; determining the operating bottleneck time of each functional module on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor for each functional module; selecting the intermediate processor corresponding to the minimum operating bottleneck time from multiple operating bottleneck times as the target processor for the functional module; deploying each functional module to the corresponding target processor; and executing the data to be processed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0028] This application embodiment obtains intermediate processors by initially screening the peak resource requirements of each functional module and the remaining capacity of candidate processors. It then quantitatively evaluates the corresponding bottleneck time by combining the operational load characteristics of the functional modules with the hardware performance indicators of the intermediate processors. This allows for the selection and deployment of the most efficient target processor for each functional module, followed by data processing based on the deployed target processors. In this way, this application embodiment achieves precise adaptation between multimodal functional modules and heterogeneous hardware, effectively avoiding high-end hardware mismatch and idle existing heterogeneous resources. While maximizing the computing power utilization of each target processor and the overall hardware resource utilization of the system, it reduces the processing latency of heterogeneous data, thereby significantly improving the end-to-end data processing efficiency of large multimodal models on heterogeneous platforms.

[0029] The specific details regarding the multimodal data processing methods, apparatus, electronic devices, and storage media provided in the embodiments of this application will be described in detail below.

[0030] Please see Figure 1 , Figure 1 This is a schematic diagram of the system framework corresponding to the multimodal data processing method provided in the embodiments of this application. The multimodal data processing method provided in the embodiments of this application can be applied to this system framework.

[0031] It includes terminal 140, Internet 130, gateway 120, server 110, etc.

[0032] Terminal 140 or server 110 may be a device that performs a method for processing multimodal data.

[0033] Terminal 140 includes, but is not limited to, mobile phones, tablets, computers, and intelligent computing centers. Terminal 140 can be a single device or a collection of multiple devices. For example, multiple computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, thus forming a terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0034] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0035] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0036] The embodiments of this application can be applied to various scenarios, such as AI-generated content cloud service platforms, multimodal intelligent interaction and virtual digital human systems, and medical image multimodal intelligent assisted diagnostic centers. These are only illustrative examples and do not represent a limitation on the scenarios in which the multimodal data processing methods of this application are applied.

[0037] Next, we will describe it from the perspective of the multimodal data processing device, such as... Figure 2 As shown, Figure 2 This is a flowchart illustrating a multimodal data processing method provided in an embodiment of this application. The multimodal data processing method is applied to a multimodal data processing device. Figure 2 The method may include, but is not limited to, the following steps 210 to 260. When the multimodal data processing device executes the multimodal data processing method, the specific process is as follows. It should be noted that this embodiment... Figure 2The order of steps 210 to 260 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0038] Step 210: Obtain the data to be processed and the remaining resource capacity of each candidate processor, and determine the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed. Step 220: For each functional module, select multiple intermediate processors from all candidate processors based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors. Step 230: Obtain the runtime load characteristics of each functional module and the hardware performance indicators of each intermediate processor; Step 240: For each functional module, determine the bottleneck time of the functional module running on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor. Step 250: For each functional module, select the intermediate processor corresponding to the smallest running bottleneck time from multiple running bottleneck times as the target processor of the functional module. Step 260: Deploy each functional module to the corresponding target processor, and execute the data to be processed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0039] Steps 210 to 260 are described in detail below.

[0040] In step 210, the data to be processed and the remaining resource capacity of each candidate processor are obtained, and the peak resource requirements of each functional module in the multimodal large model are determined according to the data type of the data to be processed.

[0041] The data to be processed refers to the raw information currently received by the device that needs to be input into the multimodal large model for processing. Examples include a text prompt entered by the user, an uploaded image that needs editing, or a video clip requiring image-text alignment analysis. After acquiring the data to be processed, the data type needs to be determined. The data type refers to the classification label used to distinguish the modality to which the input data belongs; for example, it could be plain text, plain image, or a combination of text and images. Different data types often mean different requirements for the computational density and memory bandwidth of the multimodal large model (hereinafter referred to as "large model" for ease of description) during subsequent model inference. For example, text tends to require logical processing, while images tend to require intensive computation.

[0042] In this context, candidate processors refer to all currently available physical chips or computing nodes in a heterogeneous hardware platform (resource pool) that are theoretically capable of handling computational tasks. These typically include general-purpose central processing units (CPUs) and AI-specific acceleration chips of varying computing power levels and models, such as neural processing units (NPUs) and graphics processing units (GPUs). The remaining resource capacity of candidate processors refers to the amount of hardware resources that are not currently occupied and can be allocated to new tasks for each candidate processor.

[0043] Multimodal large-scale models refer to ultra-large-scale artificial intelligence networks capable of simultaneously understanding, processing, and generating multiple modalities of information (such as text, images, and sound), such as the text-to-image model Qwen-Image. These models break the limitations of traditional AI, which can only process single-type data, and enable cross-modal semantic alignment and collaborative reasoning. The functional modules of a multimodal large-scale model refer to the relatively independent execution stages (stages) that are broken down during the inference process based on algorithmic logic and computational characteristics.

[0044] For example, taking the Qwen-Image multimodal large model as an example, for the multimodal text-to-image inference scenario, the multimodal large model inference process is divided into 3 stages with clear boundaries and independent functions to adapt to the multimodal inference computing requirements, as follows: (1) Text encoding stage: responsible for multimodal text preprocessing and deep feature extraction. After the user input text prompts are segmented and embedded, deep semantic features are extracted and encoded through LLM, and a conditional vector with internal model semantics is output. (2) MMDiT Diffusion Stage: Responsible for multimodal text-image bimodal (text-latent space feature map) calculation and core image generation, which is a computationally intensive link in the multimodal reasoning process; (3) VAE-Decoder Stage: Responsible for image reconstruction in multimodal mode, outputting the final multimodal text-to-image result.

[0045] Peak resource requirements refer to the maximum hardware resource requirements of a specific functional module when it is running at full load (processing a batch of data). It usually includes the highest peak memory usage (including weights, activation values, and cache) required by the module at the moment of the most intensive computation, as well as the highest peak computing power throughput required.

[0046] In this embodiment of the application, a set of inference stages to be deployed is provided. and available heterogeneous hardware pools The goal is to solve This is to maximize the throughput of large models.

[0047] In some embodiments, determining the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed includes: (1.1) Based on the data type of the data to be processed, determine the total number of parameters and the parameter precision of each functional module in the multimodal large model; (1.2) Determine the weight requirements of each functional module based on the total number of parameters and the parameter precision; (1.3) Determine the acceptable input data resolution and acceptable intermediate feature map size for each functional module in the multimodal large model based on the data type to be processed; (1.4) Determine the maximum activation value requirement for each functional module based on the acceptable input data resolution and the acceptable intermediate feature map size; (1.5) Determine the cache requirements of each functional module in the multimodal large model based on the data type to be processed; (1.6) Based on the weight usage requirements, maximum activation value usage requirements and cache usage requirements, determine the peak resource requirements of each functional module in the multimodal large model.

[0048] In some embodiments, to select a suitable processor for each functional module from a heterogeneous hardware pool, it is necessary to first determine the peak resource requirements of each functional module in the multimodal large model. First, based on the data type of the data to be processed, the total number of parameters and the parameter precision of the total number of parameters for each functional module in the multimodal large model are determined.

[0049] The total number of parameters refers to the overall order of magnitude of the learnable parameters contained in a certain functional module of a large model, which is usually measured in millions (M) or billions (B). The parameter precision refers to the numerical data type format used to represent the total number of parameters in the computer computing and storage system, such as half-precision floating-point numbers (FP16). The two can be combined to determine the weight occupancy requirements of each functional module of the large model. The weight occupancy requirements represent the actual physical storage space required to fully load all parameters corresponding to each stage into the video memory or memory of the corresponding target hardware during the inference execution phase of the multimodal model.

[0050] Acceptable input data resolution refers to the spatial dimension of the image or visual feature tensor that a multimodal model functional module can support and actually input in a single inference process, such as 1024×1024 pixels. Acceptable input data resolution determines the sequence length and basic computational scale of the underlying matrix multiplications during a single forward propagation. Acceptable intermediate feature map size refers to the spatial dimension and number of channels of the hidden layer tensor output after processing by some network layers (such as upsampling or transposed convolutional layers) during the deep computation of a large model network structure, such as a tensor with a size of 512×512×128. Its size reflects the scale of feature representation during data flow within the model. Acceptable input data resolution and acceptable intermediate feature map size can be combined to determine the maximum activation value requirement. The maximum activation value requirement refers to the upper limit of the peak storage space that must be dynamically allocated in the processor's GPU memory during the forward propagation of a large model inference in order to store the intermediate feature results generated by each network layer.

[0051] Among them, cache usage requirements refer to the video memory or memory space that must be reserved in the pipeline scheduling process of multimodal inference, in addition to the weights of the large model itself and the activation values ​​between layers, in order to maintain the operation of the system scheduler, implement the layered startup cache, and temporarily store intermediate calculation variables and other additional system overhead.

[0052] In some embodiments, the peak resource requirements of each functional module are determined using the following formula: .in, Indicates the weight usage requirement; This indicates the maximum activation value required. This indicates the cache usage requirement.

[0053] It is understood that the embodiments of this application effectively avoid the risk of overflow and crash caused by insufficient video memory when large models are actually running on the processor by predicting the different video memory consumption characteristics of different heterogeneous data during inference, and provide reliable data support for accurately selecting candidate processors that meet the capacity requirements in the heterogeneous hardware pool.

[0054] In step 220, for each functional module, multiple intermediate processors are selected from all candidate processors based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors.

[0055] In some embodiments, the calculated maximum resource overhead of the module is compared with the current remaining available resources of each processor in the hardware pool to filter out hardware whose capacity cannot meet the lower limit of the functional module's operation, thereby obtaining the set of processors retained in the first round of screening (i.e., intermediate processors), realizing the initial screening of hardware feasibility and narrowing the calculation scope of subsequent optimization.

[0056] In some embodiments, multiple intermediate processors are selected from all candidate processors based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors, including: (2.1) Multiple first processors are determined based on the remaining resource capacity of each candidate processor and the peak resource requirements of the functional modules; (2.2) Obtain the maximum redundancy ratio and maximum delay upper bound of the functional modules; (2.3) Determine the required throughput of the functional modules based on the ratio between the operating load characteristics and the upper limit of the maximum delay; (2.4) Determine the computing power redundancy ratio when the functional module is deployed on each first processor based on the ratio between each hardware performance indicator and the required throughput; (2.5) For each first processor, when the computing power redundancy ratio corresponding to the first processor is less than or equal to the maximum redundancy ratio, the first processor is used as an intermediate processor to obtain multiple intermediate processors.

[0057] First, let's introduce the steps (2.1): In some embodiments, for each functional module, multiple first processors are first selected based on the peak resource requirements of the functional module and the remaining resource capacity of each candidate processor in the hardware pool.

[0058] Regarding step (2.1), in some embodiments, multiple first processors are determined based on the remaining resource capacity of the candidate processors and the peak resource requirements of the functional modules, including: (2.1.1) Obtain the safety margin factor; (2.1.2) Update the peak resource requirements of the function module according to the safety margin coefficient to obtain the updated peak resource requirements; (2.1.3) When the remaining resource capacity of a candidate processor is equal to or greater than the updated peak resource requirement, the candidate processor is used as the first processor to obtain multiple first processors.

[0059] In some embodiments, multiple first processors are selected from the hardware pool using the following formula: .in, This represents the safety margin factor, which can take values ​​from 0.1 to 0.15. Indicates the peak resource requirements of the functional module; This indicates the updated peak resource demand; Indicates the corresponding processor The remaining resource capacity.

[0060] It is understood that a candidate processor is selected as the first processor only if its remaining resource capacity is equal to or greater than the updated peak resource requirement, thus obtaining multiple first processors corresponding to the currently selected functional module. This application's embodiments, by setting reasonable fault tolerance boundaries for candidate processors, effectively avoid the risk of system crashes or memory overflows caused by resource overload under extreme operating conditions, significantly improving the robustness of the initial screening of heterogeneous hardware from the underlying architecture.

[0061] Next, we will continue with steps (2.2) to (2.5): In some embodiments, to prevent high-level hardware from being allocated to memory-intensive lightweight stages, thereby causing a serious waste of computing power, a computing power redundancy ratio constraint is introduced using the following formula to screen multiple candidate processors: .in, This represents the maximum redundancy ratio of the i-th functional module, typically ranging from 2.0 to 3.0. This represents the upper bound of the maximum delay for the functional module; The number of floating-point operations represents the runtime load characteristics of a functional module. This represents the required throughput, which is the minimum computing rate needed to meet latency constraints. This represents the hardware performance metrics of the first processor, specifically the first processor h. j Peak computing throughput.

[0062] in, R is the computing power redundancy ratio, representing the ratio of peak hardware computing power to the throughput required by the stage. ij ≤R max When this occurs, it indicates that the hardware computing power has not exceeded the allowable multiple of the stage requirements, and will not cause excessive waste of computing power; therefore, the first processor is used as an intermediate processor. ij >R max If this occurs, it indicates that the hardware computing power far exceeds the stage requirements, and the system should be downgraded to hardware with lower computing power to avoid over-configuration of computing resources.

[0063] Among them, the upper bound of the maximum delay Indicates functional module s i The maximum theoretical execution time allowed when executed on the target processor. The determination includes the following three sources, which are applied in order of priority: (1) User explicit specification: When a user has a clear delay constraint on a specific functional module (such as requiring the end-to-end latency to be lower than a specific threshold in a real-time interaction scenario), the user can directly set the delay for that functional module. In this case, the user-specified value is used directly.

[0064] (2) Determination method based on pipeline steady-state balance (default method): When the user does not explicitly specify the delay upper bound, the system automatically determines the delay upper bound of each functional module through the following steps: Based on the computational characteristics and hardware capabilities of each stage, the bottleneck stage with the longest execution time is identified. Typically, computationally intensive stages (such as MMDit Diffusion) are the bottleneck stages.

[0065] Set bottleneck stage The micro-batch size is The execution time on the target hardware is Then the steady-state throughput of the bottleneck stage is: .

[0066] For each functional module Its upper limit of latency is determined based on the bottleneck throughput and the micro-batch size of the stage: In this regard, the upper bound of the delay for each stage should match its throughput with that of the bottleneck stage to avoid pipeline bubbles.

[0067] (3) Based on end-to-end delay constraint decomposition (alternative method): When the system has an end-to-end delay constraint L total At that time, the upper bound of the delay is obtained by decomposing the computational load of each functional module according to its proportion, as shown in the following formula: =L total ×( / Σ k ), where Σ k FLOPs k The total computational load of all functional modules is summed up. Delay budgets are allocated proportionally to the computational load, with functional modules that have a larger computational load receiving more delay margin.

[0068] It should be noted that among the three sources mentioned above, the one explicitly specified by the user has the highest priority; when the user does not specify, method (2) is used by default; when the system has end-to-end delay constraints and no throughput target is set, method (3) can be used.

[0069] For example, taking the Qwen-Image model as an example, assume that the micro-batch sizes of each functional module are as follows: Text Encoding Stage, 64 batches; MMDit Diffusion Stage, 16 batches; VAE-Decoder Stage, 32 batches.

[0070] Assuming that, based on the analysis of the method proposed in the embodiments of this application, MMDiT Diffusion is the bottleneck stage. Assuming that the single-step execution time of MMDiT on the target hardware (such as a 16×910C cluster) is 247.5ms, and the total execution time for 30 denoising steps is 7.43 seconds, then the bottleneck throughput is: Requests per second.

[0071] According to the pipeline steady-state balance principle, the upper bound of the maximum delay for each functional module is as follows: Text Encoding Stage: Second.

[0072] MMDiT Diffusion Stage: Second.

[0073] VAE-Decoder Stage: Second.

[0074] The allocation result of this example has the following characteristics: Physical rationale: MMDit, as a computationally intensive bottleneck stage, has an upper limit for latency (7.43 seconds) that is equal to the actual execution time, which matches its computational complexity; Pipeline balancing: The throughput of each stage is matched with the bottleneck throughput (2.15 requests / second). Lightweight stages complete quickly and wait for the bottleneck stage to finish, thus avoiding pipeline bubbles. Hardware adaptability: The upper limit of latency is calculated based on the actual hardware capabilities and is automatically adjusted when the hardware configuration changes.

[0075] Specifically, taking the MMDiT Diffusion Stage as an example, its ≈3 TFLOPs, =4 seconds, then the required throughput / ≈750GFLOPs i / s, if the candidate processor =2000GFLOPs i / s, then the redundancy ratio R ij =2000 / 750≈2.67. When R max When = 3.0, R ij ≤R max This processor is selected as a processor based on its redundancy ratio.

[0076] Understandably, when the computing power redundancy ratio of the first processor is less than or equal to the maximum redundancy ratio after screening, it is used as the intermediate processor corresponding to the current functional module, thus obtaining multiple intermediate processors. Especially when... When this occurs, it indicates that the computing power of the first processor far exceeds the stage requirements, and it should be downgraded to a first processor with lower computing power to avoid selection. Excessively high redundancy in the first processor. Based on the initial screening using remaining resource capacity, this application's embodiments construct a secondary screening mechanism to prevent computing power misallocation. By eliminating first processors with excessive redundancy, it effectively prevents high-end, high-computing-power hardware from being incorrectly allocated to functional stages with lighter computational loads or less stringent latency requirements. From a matching logic perspective, this reduces the over-configuration and waste of computing resources in heterogeneous hardware platforms, thereby maximizing overall hardware utilization efficiency and the rationality of cross-hardware collaborative allocation.

[0077] In step 230, the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor are obtained.

[0078] Among them, the runtime load characteristics characterize the underlying computing mode of the functional module during execution. For example, whether the functional module tends to perform computationally intensive tasks with floating-point operations or memory-intensive tasks with frequent data read and write operations. Hardware performance indicators reflect the theoretical physical limits of the selected intermediate processor in terms of peak computing throughput and memory access bandwidth.

[0079] In step 240, for each functional module, the bottleneck time of the functional module running on each intermediate processor is determined based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor.

[0080] Bottleneck time refers to the extended task execution time during the execution of a specific functional module when its computational or communication demands (i.e., workload characteristics) exceed the processor's actual available resource processing capacity (i.e., hardware performance indicators). This results in task backlog, limited execution efficiency, or performance saturation, leading to prolonged task execution time. In other words, bottleneck time refers to the shortest theoretical execution time required to complete the computational task of the functional module, determining whether the module's performance is limited by computational power or memory access bottlenecks on the processor.

[0081] In some embodiments, based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor, the bottleneck time of the functional module running on each intermediate processor is determined, including: (3.1) Based on the ratio between the total computational load characteristics and the hardware throughput of each intermediate processor, the estimated computation time when the functional module is deployed on each intermediate processor is obtained; (3.2) Based on the ratio between the total memory access characteristics and the hardware bandwidth of each intermediate processor, the estimated memory access time when the functional module is deployed on each intermediate processor is obtained; (3.3) Select the time with the largest value from all estimated computation time and all estimated memory access time as the bottleneck time of the functional module.

[0082] In some embodiments, the runtime characteristics include total computational load characteristics and total memory access characteristics, and the hardware performance indicators include hardware throughput and hardware bandwidth. For each functional module, the estimated computation time when the functional module is deployed on different intermediate processors is determined using the following formula. : .in, This indicates a characteristic based on the total computational load; This represents the hardware throughput of each intermediate processor. In the embodiments of this application, the same parameters have the same meaning, and repeated descriptions will not be repeated.

[0083] Furthermore, for each functional module, the estimated memory access time when the functional module is deployed on different intermediate processors is determined using the following formula. : .in, This indicates the total number of memory accesses. This represents the hardware bandwidth of each intermediate processor.

[0084] Furthermore, the bottleneck time for each functional module is determined using the following formula. : That is, from all the estimated computation time and all the estimated memory access time, the time with the largest value is selected as the running bottleneck time for the current functional module.

[0085] It should be noted that the hardware utilization efficiency η ij Used to measure the balanced utilization of hardware resources, it can serve as an auxiliary evaluation metric for deployment effectiveness. When selecting a target processor, the bottleneck time T is considered. ij Minimize as the direct selection criterion. When T ij =max( , When T is at its minimum, this processor is the optimal target processor. ij =max( / ,Bytes i / BW j The selection criterion implicitly implements the following matching logic: computationally intensive stages tend to select... The largest hardware; memory-intensive stages tend to choose the hardware with the largest BW; balanced stages make full use of both types of resources.

[0086] It is understood that the embodiments of this application determine the corresponding runtime bottleneck time from the estimated computation time and estimated memory access time of each functional module, so as to accurately identify the actual performance limitation dimension of each functional module when it is executed on a specific intermediate processor, that is, whether it is computationally intensive or memory-intensive, so as to determine the corresponding computation density value of each functional module when set on different intermediate processors based on the runtime bottleneck time.

[0087] In step 250, for each functional module, the intermediate processor corresponding to the smallest running bottleneck time from multiple running bottleneck times is selected as the target processor of the functional module.

[0088] In some embodiments, the target processor refers to the optimal processor determined by the device to be most suitable for executing the current functional module after initial screening of memory capacity in the first stage and re-screening of performance in the second stage. By comparing the bottleneck times of each intermediate processor, the hardware with the shortest execution time is selected as the target processor. This embodiment of the application can ensure that each functional module in the large model can be matched with dedicated heterogeneous hardware that minimizes execution latency and does not cause resource waste.

[0089] In some embodiments, selecting the intermediate processor corresponding to the smallest runtime bottleneck time from a plurality of runtime bottleneck times as the target processor of the functional module includes: (4.1) When there are at least two intermediate processors with the same runtime bottleneck, obtain the arithmetic strength value of the functional module; (4.2) For each intermediate processor with the same runtime bottleneck time, obtain the hardware inflection point value of each intermediate processor; (4.3) Determine the hardware utilization efficiency of the functional module running on each intermediate processor based on the arithmetic strength value and each hardware inflection point value; (4.4) Select the intermediate processor with the highest hardware utilization efficiency from multiple hardware utilization efficiencies as the target processor of the functional module.

[0090] In some embodiments, for the same functional module, there may be two or more intermediate processors with the same runtime bottleneck time. In order to further improve the processing efficiency of multimodal data, this application embodiment does not arbitrarily select one of the multiple identical intermediate processors to deploy the current functional module. Instead, it calculates the hardware utilization efficiency of these intermediate processors respectively, and selects the target processor most suitable for deployment of the current functional module based on the hardware utilization efficiency. In this way, this application embodiment can maximize the overall utilization efficiency of the heterogeneous hardware pool.

[0091] Hardware utilization efficiency measures the degree to which the computational or memory access requirements of a specific functional module match the underlying architecture of the intermediate processor (such as computing power and bandwidth distribution). Utilization efficiency tends to be maximized when the load characteristics of the functional module are highly matched with the performance bottleneck of the hardware.

[0092] Furthermore, define the following two intermediate variables: (1) Arithmetic strength value C i : Represents functional module s i The actual total number of floating-point operations required, i.e., C i = ; (2) Equivalent memory access computation M i This indicates that the total memory access volume of the functional module is converted into the equivalent computational volume based on the Roofline inflection point of the intermediate processor, i.e., M. i =Bytes i × =Bytes i ×( / BW j )in, = / BW j For intermediate processor h j The Roofline inflection point (hardware inflection point value) represents the arithmetic intensity threshold at which the hardware transitions from a memory-intensive region to a compute-intensive region.

[0093] The functional module s is determined using the following formula. i Based on intermediate processor h j Runtime hardware utilization efficiency η ij : η ij =min(C i M i ) / max(C i M i )=min( Bytes i · ) / max( Bytes i · Among them, due to = / Bytes i Substituting the values, we get: η ij =min( , ) / max( , ), where min is the minimum function and max is the maximum function. ηij ∈(0,1], which is used to measure functional modules s i Arithmetic strength and intermediate processor h j The degree of matching between the inflection points of the Roofline. Specifically: when > Time (computation-intensive), C i >M i η ij =M i / C i / <1 indicates that there is redundancy in hardware bandwidth resources; when < Time-consuming (memory-intensive), C i <M i η ij =C i / M i = / <1 indicates that there is redundancy in hardware computing resources; when = Time (Equilibrium type), C i =M i η ij =1 indicates that both resources are fully utilized, and the hardware utilization efficiency is the highest.

[0094] Furthermore, when the system determines that there are at least two intermediate processors with the same bottleneck time, it calculates the hardware utilization efficiency of each intermediate processor and selects the intermediate processor with the largest value (closest to 1) as the target processor for the current functional module.

[0095] It is understood that by evaluating the runtime bottleneck time of each functional module, the embodiments of this application can accurately restore the correlation between the underlying execution state of each functional module of the multimodal large model and the underlying hardware physical constraint boundary of each processor, thereby providing data support for the subsequent accurate quantification of hardware utilization efficiency and ensuring the maximization of the overall hardware resource utilization of the system in the multimodal heterogeneous deployment scenario.

[0096] In step 260, each functional module is deployed to the corresponding target processor, and the data to be processed is executed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0097] In some embodiments, the model weights of functional modules are loaded into the storage space of the target processor so that the data processing flow can be executed subsequently according to the deployed target processor. The system performs pipelined computation on the input data to be processed through a cross-hardware data flow mechanism, and finally outputs high-precision multimodal inference results.

[0098] It is understood that the embodiments of this application, through multi-stage processor selection, achieve deep integration between the functional modules within the large model and the underlying architecture of heterogeneous hardware, effectively avoiding resource idleness caused by computing power redundancy and the inability to process data due to insufficient video memory. In other words, the embodiments of this application not only significantly improve the overall resource utilization of the heterogeneous hardware platform, but also fundamentally reduce the end-to-end inference latency of multimodal heterogeneous data, ensuring high throughput and high energy efficiency of the multimodal large model in complex concurrent scenarios.

[0099] In some embodiments, the data to be processed is executed based on each target processor after deployment to obtain the data processing result corresponding to the data to be processed, including: (5.1) Obtain the module topology information between all functional modules, and determine the pipeline information of each functional module based on the module topology information with the goal of maximizing throughput; (5.2) Based on the pipeline information of each functional module and the deployed target processor, execute the data to be processed to obtain the data processing result corresponding to the data to be processed.

[0100] Among these, module topology information refers to the execution order and data dependencies between various functional stages within a multimodal large model, typically represented as a Directed Acyclic Graph (DAG) structure. Throughput refers to the number of multimodal requests or data scale successfully processed per unit time. Pipeline information refers to the global scheduling strategy and execution timing for asynchronous parallel execution across hardware. By acquiring the dependencies between these modules, the device can, from a global perspective, scientifically plan the pipeline overlap and parallel timing arrangement of each functional module across heterogeneous hardware, with the ultimate optimization goal of overcoming processing bottlenecks and maximizing the system's steady-state throughput. This provides precise scheduling guidance for subsequent data execution.

[0101] In some embodiments, after determining the optimal hardware selection for each stage, the parallel scheduling and resource allocation between stages are then optimized from a global perspective, specifically with system throughput as the ultimate optimization objective: Suppose that the set of stages forms a DAG based on dependencies, and each stage... Assigned to hardware Execution time is The steady-state throughput is: .in, stage The micro-batch size. The optimization problem is: That is, under the constraint of the video memory capacity of each piece of hardware, the mapping relationship and the micro-batch size of each stage are adjusted. Maximize the throughput of large models.

[0102] Furthermore, during the actual operation phase, based on the predetermined pipeline information, multiple target processors are coordinated to perform calculations on the input data to be processed. Specifically, the output data of the functional modules is streamed to the subsequent functional modules according to the predetermined pipeline rhythm. Under this scheduling mechanism, each target processor achieves highly parallel asynchronous execution, and finally completes the verification and fusion of multimodal data at the end of the processing flow, outputting the final data processing result.

[0103] Understandably, compared to local optimization methods that only focus on the performance of a single node, the embodiments of this application introduce module topology information and generate pipeline information with the goal of maximizing throughput. This effectively overlaps the computation process with the cross-hardware communication process, eliminates the synchronization waiting delay and execution blockage when multimodal functional modules switch between different hardware, and ensures that all target processors can maintain a continuous and full computing state. Thus, without increasing additional hardware costs, the overall throughput and end-to-end inference efficiency of multimodal large models on heterogeneous platforms are significantly improved.

[0104] Furthermore, when processing the next heterogeneous data, the deployment of the large model can be updated again. Based on the scale of the input data (including image resolution, text length, etc.) and the real-time load and memory usage of each hardware, the batch size of each stage can be dynamically adjusted, so that the lightweight stage (text encoding) and the computationally intensive stage (image and text dual-modal generation) form a load-staggered and peak-complementary execution mode, thereby adapting to the processing needs of multimodal heterogeneous data.

[0105] For example, the key optimization strategies are as follows: (1) Cross-hardware pipeline overlap: For adjacent stages in a DAG that are on different hardware (e.g., MMDiT → VAE-Decoder), the currently requested stage With the next requested stage Execution is performed in parallel on different hardware. Let stage... The execution time is stage The execution time is Then, after pipeline overlap, the effective inter-stage waiting time of the system is: Among them, when Time (e.g., VAE-Decoder delay does not exceed MMDiT denoising delay), stage Completely hidden in the stage During the execution time, the system throughput is determined only by the bottleneck stage.

[0106] (2) Parallelism with multiple hardware instances: For computationally intensive bottleneck stages (such as MMDiT), under the condition of satisfying memory constraints, the hardware instances can be parallelized. Divided into There are 1 independent instance (TP=1), and each instance handles different requests. The throughput of this stage is now increased as follows: Among them, the number of instances The upper bound is determined by the video memory constraint: , This is the floor symbol.

[0107] (3) Load balancing scheduling: When multiple homogeneous instances exist in a stage, a round-robin or shortest queue priority strategy is used to allocate requests, avoiding queuing delays caused by single instance overload. The scheduler maintains the busy / idle status of each instance and only sends tasks to idle instances to ensure: .in, For example The current queue depth, This is the maximum allowable backpressure limit (used for backpressure control).

[0108] (4) Communication-Computation Overlap: Data transfer between stages (such as the transfer of latent tensors output by MMDiT to the VAE-Decoder) uses asynchronous point-to-point communication. While the transfer is in progress, the sending hardware can immediately begin processing the next requested stage. This achieves complete concealment of communication delays.

[0109] In this embodiment of the application, taking a multimodal image generation model as an example, the input data processing is completed through the following steps: (a) Stage heterogeneous deployment: The system uses the stage heterogeneous deployment module to deploy the three multimodal stages to the corresponding hardware respectively, and completes hardware matching and initialization; (b) Layered cache loading: The layered startup cache module builds a three-level cache structure, preloading the backbone weights of the multimodal large model (e.g., about 20GB), commonly used multimodal style features (e.g., 500+ types), and inference parameters, maintaining a cache hit rate of over 92%; (c) Pipeline scheduling: The cross-hardware pipeline scheduling module dynamically adjusts the batch size of each stage (64g tasks per batch for text encoding, 16 tasks per batch for MMDit, and 32 tasks per batch for VAE-Decoder) based on the scale of the input multimodal data (text length, etc.) and starts cross-hardware asynchronous parallel execution; (d) Output of results: The result aggregation module verifies the multimodal reasoning results (PSNR≥35dB, semantic similarity≥0.85), and outputs a PNG format text image after the verification is passed.

[0110] Regarding step (a), we also have: (1) Text Encoding (Text Encoder and LLM Feature Encoding) Stage → Mid-range AI processor (e.g., Ascend NPU-310P): Lightweight text encoding in multimodal processing, such as parameter scale 7B, time consumption <3%, adapted to the computing power of mid-range AI processors and 24GB of video memory, freeing up high-end AI processor resources, and adapting to the needs of multimodal lightweight text processing. (2) MMDiT Diffusion Stage → High-end AI processor (e.g., Ascend NPU-910C): Multimodal image-text dual-modal (text-latent space feature map) computation and core generation stage, for example, parameter scale 20B, time consumption accounts for 60%-80%, giving full play to the parallel computing advantages and 64GB large video memory of high-end AI processors to meet the multimodal intensive computing needs; (3) VAE-Decoder Stage → Corresponding AI processor (e.g., Ascend NPU-910B): Image reconstruction in multimodal processing, for example, the parameter scale is about 100M, the time consumption accounts for 10%-20%, it is compatible with 32GB of video memory, and it is executed in parallel with the core generation stage to improve the overall efficiency of multimodal inference; the above hardware models and parameters are all illustrative examples and do not constitute specific limitations.

[0111] Regarding step (b), we also have: (1) Cache preloading: By preloading the core multimodal data of each stage (such as the backbone weights of the bimodal model, commonly used multimodal style features, inference parameters, etc.) into the cache of the AI ​​processor (such as Ascend NPU), repeated reading from the disk is avoided, and the startup time is shortened; the above models and hardware are for illustrative purposes only and are not limited to this specific model.

[0112] (2) Cache update and eviction: Establish a cache update and eviction mechanism based on multimodal task access frequency, data validity and stage execution priority to ensure the timeliness and validity of cached data and maintain a high cache hit rate.

[0113] Regarding step (c), we also have: (1) Task queue management: Maintain a dedicated task queue for each multimodal independent stage to realize the isolation and independent scheduling of multimodal tasks (text / image) in each stage and avoid task interference; at the same time, monitor the task volume of each queue in real time, dynamically adjust the priority of multimodal task allocation, ensure that the lightweight stage and the computationally intensive stage advance in tandem, and adapt to the need for independent execution of multimodal and bimodal tasks.

[0114] (2) Batch processing strategy: Based on the scale of multimodal input data (batch size, text length) and the real-time computing power load and memory usage of each hardware, dynamically adjust the batch processing size of each stage to adapt to the computing power capacity and memory capacity of different hardware, improve collaborative efficiency, avoid memory overflow, and adapt to the batch processing needs of multimodal heterogeneous data.

[0115] (3) Asynchronous scheduling: The multimodal cross-hardware pipeline asynchronous scheduling mechanism is implemented. Through the optimized cross-hardware transmission link, the multimodal output results (text vectors, latent space feature maps) of the preceding stage are transmitted to the subsequent stage in real time, eliminating inter-stage blocking, reducing transmission delay, and adapting to the parallel scheduling characteristics of heterogeneous hardware and the transmission requirements of multimodal heterogeneous data.

[0116] Regarding step (d), we also have: (1) Receive the intermediate and final results of multimodal inference transmitted from each stage and complete the multimodal data aggregation; (2) After the verification is passed, multimodal text and image results in various formats such as JPG, PNG, WebP, and TIFF are output to meet the needs of different application scenarios.

[0117] like Figure 3 As shown, Figure 3 This is a schematic diagram of the module structure of a multimodal data processing device provided in an embodiment of this application. The multimodal data processing device 300 may include the following modules 310 to 360: The first acquisition module 310 is used to acquire the data to be processed and the remaining resource capacity of each candidate processor, and to determine the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed. The intermediate processor determination module 320 is used to select multiple intermediate processors from all candidate processors for each functional module based on the peak resource requirements of the functional module and the remaining resource capacity of all candidate processors. The second acquisition module 330 is used to acquire the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor. The bottleneck time determination module 340 is used to determine the bottleneck time of each functional module running on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor. The target processor determination module 350 is used to select the intermediate processor corresponding to the smallest running bottleneck time from multiple running bottleneck times as the target processor of the functional module for each functional module. The data processing result determination module 360 ​​is used to deploy each functional module to the corresponding target processor, execute the data to be processed based on each deployed target processor, and obtain the data processing result corresponding to the data to be processed.

[0118] In some embodiments, the first acquisition module 310 is used for: Based on the data type to be processed, determine the total number of parameters and the parameter precision of each functional module in the multimodal large model; Based on the total number and precision of parameters, the weight requirements of each functional module are determined. Based on the data type to be processed, determine the acceptable input data resolution and acceptable intermediate feature map size for each functional module in the multimodal large model; Based on the acceptable input data resolution and acceptable intermediate feature map size, determine the maximum activation value requirement for each functional module; Based on the data type of the data to be processed, determine the cache usage requirements of each functional module in the multimodal large model; Based on the weight usage requirements, maximum activation value usage requirements, and cache usage requirements, the peak resource requirements of each functional module in the multimodal large model are determined.

[0119] In some embodiments, the intermediate processor determining module 320 is used for: Multiple first processors are determined based on the remaining resource capacity of each candidate processor and the peak resource requirements of the functional modules; Obtain the maximum redundancy ratio and maximum latency upper bound of the functional modules; The required throughput of the functional modules is determined based on the ratio between the operating load characteristics and the upper limit of the maximum latency. The computing power redundancy ratio when the functional module is deployed on each first processor is determined based on the ratio between each hardware performance indicator and the required throughput. For each first processor, when the computing power redundancy ratio corresponding to the first processor is less than or equal to the maximum redundancy ratio, the first processor is used as an intermediate processor to obtain multiple intermediate processors.

[0120] In some embodiments, the intermediate processor determining module 320 is further configured to: The peak resource requirements of capacity and functional modules identified several primary processors, including: Obtain the safety margin factor; The peak resource requirements of the update function module are obtained by updating the peak resource requirements based on the safety margin coefficient; When the remaining resource capacity of a candidate processor is equal to or greater than the updated peak resource requirement, the candidate processor is selected as the first processor to obtain multiple first processors.

[0121] In some embodiments, the bottleneck time determination module 340 is used for: Based on the operating load characteristics of the functional modules and the hardware performance indicators of each intermediate processor, the operating bottleneck time of the functional modules is determined. When each functional module is set on a different intermediate processor, the corresponding computing density value is determined based on the bottleneck time of the functional module and the throughput of each hardware, and the corresponding computing-memory balance value is determined based on the bottleneck time of the functional module and the bandwidth of each hardware. For each functional module, the minimum and maximum values ​​are determined from all computation density values ​​and all compute-memory balance values. Based on the ratio between the minimum and maximum values, the hardware utilization efficiency of the functional module based on each intermediate processor is determined.

[0122] In some embodiments, hardware performance metrics include hardware throughput and hardware bandwidth, and runtime load characteristics include total computational load characteristics and total memory access characteristics. In some embodiments, the bottleneck time determination module 340 is further configured to: Based on the ratio between the total computational load characteristics and the hardware throughput of each intermediate processor, the estimated computation time when the functional module is deployed on each intermediate processor is obtained; Based on the ratio between the total memory access characteristics and the hardware bandwidth of each intermediate processor, the estimated memory access time when the functional module is deployed on each intermediate processor is obtained. From all the estimated computation time and all the estimated memory access time, the time with the largest value is selected as the bottleneck time of the functional module.

[0123] In some embodiments, the target processor determining module 350 is further configured to: When there are at least two intermediate processors with the same bottleneck time, obtain the arithmetic strength value of the functional module; For each intermediate processor with the same runtime bottleneck time, obtain the hardware inflection point value for each intermediate processor; Based on the arithmetic strength value and each hardware inflection point value, determine the hardware utilization efficiency of the functional module running on each intermediate processor. The intermediate processor corresponding to the highest hardware utilization efficiency value among multiple hardware utilization efficiency values ​​is selected as the target processor of the functional module.

[0124] In some embodiments, the data processing result determination module 360 ​​is used for: Obtain the module topology information between all functional modules, with the goal of maximizing throughput, and determine the pipeline information of each functional module based on the module topology information; Based on the pipeline information of each functional module and the data to be processed by each deployed target processor, the data processing results corresponding to the data to be processed are obtained.

[0125] The multimodal data processing method, apparatus, electronic device, and storage medium proposed in this application involve: acquiring the data to be processed and the remaining resource capacity of each candidate processor; determining the peak resource requirement of each functional module in the multimodal large model based on the data type of the data to be processed; selecting multiple intermediate processors from all candidate processors based on the peak resource requirement of the functional module and the remaining resource capacity of all candidate processors for each functional module; acquiring the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor; determining the operating bottleneck time of each functional module on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor for each functional module; selecting the intermediate processor corresponding to the minimum operating bottleneck time from multiple operating bottleneck times as the target processor for the functional module; deploying each functional module to the corresponding target processor; and executing the data to be processed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0126] This application embodiment obtains intermediate processors by initially screening the peak resource requirements of each functional module and the remaining capacity of candidate processors. It then quantitatively evaluates the corresponding bottleneck time by combining the operational load characteristics of the functional modules with the hardware performance indicators of the intermediate processors. This allows for the selection and deployment of the most efficient target processor for each functional module, followed by data processing based on the deployed target processors. In this way, this application embodiment achieves precise adaptation between multimodal functional modules and heterogeneous hardware, effectively avoiding high-end hardware mismatch and idle existing heterogeneous resources. While maximizing the computing power utilization of each target processor and the overall hardware resource utilization of the system, it reduces the processing latency of heterogeneous data, thereby significantly improving the end-to-end data processing efficiency of large multimodal models on heterogeneous platforms.

[0127] like Figure 4 As shown, Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes: The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401 to execute the multimodal data processing method of the embodiments of this application. Input / output interface 403 is used to implement information input and output; The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0128] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for processing multimodal data.

[0129] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] The multimodal data processing method, apparatus, electronic device, and storage medium proposed in this application involve: acquiring the data to be processed and the remaining resource capacity of each candidate processor; determining the peak resource requirement of each functional module in the multimodal large model based on the data type of the data to be processed; selecting multiple intermediate processors from all candidate processors based on the peak resource requirement of the functional module and the remaining resource capacity of all candidate processors for each functional module; acquiring the operating load characteristics of each functional module and the hardware performance indicators of each intermediate processor; determining the operating bottleneck time of each functional module on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor for each functional module; selecting the intermediate processor corresponding to the minimum operating bottleneck time from multiple operating bottleneck times as the target processor for the functional module; deploying each functional module to the corresponding target processor; and executing the data to be processed based on each deployed target processor to obtain the data processing result corresponding to the data to be processed.

[0131] This application embodiment obtains intermediate processors by initially screening the peak resource requirements of each functional module and the remaining capacity of candidate processors. It then quantitatively evaluates the corresponding bottleneck time by combining the operational load characteristics of the functional modules with the hardware performance indicators of the intermediate processors. This allows for the selection and deployment of the most efficient target processor for each functional module, followed by data processing based on the deployed target processors. In this way, this application embodiment achieves precise adaptation between multimodal functional modules and heterogeneous hardware, effectively avoiding high-end hardware mismatch and idle existing heterogeneous resources. While maximizing the computing power utilization of each target processor and the overall hardware resource utilization of the system, it reduces the processing latency of heterogeneous data, thereby significantly improving the end-to-end data processing efficiency of large multimodal models on heterogeneous platforms.

[0132] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0133] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0136] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0137] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0139] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for processing multimodal data, characterized in that, include: Obtain the data to be processed and the remaining resource capacity of each candidate processor, and determine the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed; For each of the functional modules, multiple intermediate processors are selected from all the candidate processors based on the peak resource requirements of the functional module and the remaining resource capacity of all the candidate processors. Obtain the runtime load characteristics of each functional module and the hardware performance indicators of each intermediate processor; For each functional module, the bottleneck time of the functional module running on each intermediate processor is determined based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor. For each of the functional modules, the intermediate processor corresponding to the smallest running bottleneck time among the multiple running bottleneck times is selected as the target processor of the functional module; Each of the functional modules is deployed to the corresponding target processor, and the data to be processed is executed based on each of the deployed target processors to obtain the data processing result corresponding to the data to be processed.

2. The method for processing multimodal data according to claim 1, characterized in that, Determining the peak resource requirements of each functional module in the multimodal large model based on the data type of the data to be processed includes: Based on the data type of the data to be processed, determine the total number of parameters and the parameter precision of each functional module in the multimodal large model; Based on the total number of parameters and the precision of the parameters, the weight requirement of each functional module is determined; Based on the data type of the data to be processed, determine the acceptable input data resolution and acceptable intermediate feature map size for each functional module in the multimodal large model; Based on the acceptable input data resolution and the acceptable intermediate feature map size, the maximum activation value requirement for each of the functional modules is determined; Based on the data type of the data to be processed, determine the cache usage requirements of each functional module in the multimodal large model; Based on the weight usage requirement, the maximum activation value usage requirement, and the cache usage requirement, the peak resource requirement of each functional module in the multimodal large model is determined.

3. The method for processing multimodal data according to claim 1, characterized in that, The step of selecting multiple intermediate processors from all the candidate processors based on the peak resource requirements of the functional modules and the remaining resource capacity of all the candidate processors includes: Multiple first processors are determined based on the remaining resource capacity of each candidate processor and the peak resource requirements of the functional module; Obtain the maximum redundancy ratio and maximum latency upper bound of the functional module; The required throughput of the functional module is determined based on the ratio between the operating load characteristics and the maximum latency upper bound. The computing power redundancy ratio when the functional module is deployed on each of the first processors is determined based on the ratio between each of the hardware performance indicators and the required throughput. For each of the first processors, when the computing power redundancy ratio corresponding to the first processor is less than or equal to the maximum redundancy ratio, the first processor is used as an intermediate processor to obtain multiple intermediate processors.

4. The method for processing multimodal data according to claim 3, characterized in that, The determination of multiple first processors based on the remaining resource capacity of the candidate processors and the peak resource requirements of the functional modules includes: Obtain the safety margin factor; The peak resource requirements of the functional module are updated based on the safety margin coefficient to obtain the updated peak resource requirements; When the remaining resource capacity of the candidate processor is equal to or greater than the updated peak resource requirement, the candidate processor is selected as the first processor to obtain multiple first processors.

5. The method for processing multimodal data according to claim 1, characterized in that, The hardware performance metrics include hardware throughput and hardware bandwidth, and the operating load characteristics include total computational load characteristics and total memory access characteristics. The step of determining the bottleneck time of the functional module running on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor includes: Based on the ratio between the total computational load characteristics and the hardware throughput of each intermediate processor, the estimated computation time when the functional module is deployed on each intermediate processor is obtained; Based on the ratio between the total memory access characteristics and the hardware bandwidth of each intermediate processor, the estimated memory access time when the functional module is deployed on each intermediate processor is obtained. From all the estimated computation time and all the estimated memory access time, the time with the largest value is selected as the bottleneck time of the functional module.

6. The method for processing multimodal data according to claim 1, characterized in that, The step of selecting the intermediate processor corresponding to the smallest bottleneck time from among multiple bottleneck times as the target processor of the functional module includes: When there are at least two intermediate processors with the same bottleneck time, obtain the arithmetic strength value of the functional module; For each intermediate processor with the same runtime bottleneck time, obtain the hardware inflection point value for each intermediate processor; Based on the arithmetic strength value and each of the hardware inflection point values, the hardware utilization efficiency of the functional module running on each of the intermediate processors is determined; The intermediate processor corresponding to the highest hardware utilization efficiency value among the multiple hardware utilization efficiencies is selected as the target processor of the functional module.

7. The method for processing multimodal data according to claim 1, characterized in that, The process of executing the data to be processed based on each of the deployed target processors to obtain the data processing result corresponding to the data to be processed includes: Obtain the module topology information between all the functional modules, and determine the pipeline information of each functional module based on the module topology information with the goal of maximizing throughput. Based on the pipeline information of each functional module and the deployed target processor, the data to be processed is executed to obtain the data processing result corresponding to the data to be processed.

8. A multimodal data processing device, characterized in that, include: The first acquisition module is used to acquire the data to be processed and the remaining resource capacity of each candidate processor, and to determine the peak resource requirements of each functional module in the multimodal large model according to the data type of the data to be processed. An intermediate processor determination module is used to select multiple intermediate processors from all the candidate processors for each of the functional modules, based on the peak resource requirements of the functional module and the remaining resource capacity of all the candidate processors. The second acquisition module is used to acquire the operating load characteristics of each of the functional modules and the hardware performance indicators of each of the intermediate processors. The bottleneck time determination module is used to determine the bottleneck time of each functional module running on each intermediate processor based on the operating load characteristics of the functional module and the hardware performance indicators of each intermediate processor. The target processor determination module is used to select, for each of the functional modules, the intermediate processor corresponding to the smallest running bottleneck time from a plurality of running bottleneck times as the target processor of the functional module. The data processing result determination module is used to deploy each of the functional modules to the corresponding target processor, and execute the data to be processed based on each of the deployed target processors to obtain the data processing result corresponding to the data to be processed.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal data processing method according to any one of claims 1 to 7.

Images