A method, system and related device for multi-hardware hybrid large model reasoning

Abstract

The present application discloses a method, system and related device for multi-hardware hybrid large model reasoning, which is used to improve the reasoning efficiency and flexibility of complex modal tasks. The method of the present application includes: decomposing the target reasoning task in the task queue according to the structural information of the multi-modal large model and the modal change characteristics of different task stages, obtaining several subtasks and several computing units corresponding to each subtask; formulating a scheduling strategy according to the hardware adaptation report and the task resource requirements, and allocating the computing units to suitable hardware for execution according to the scheduling strategy; using a multi-modal semantic fusion model and a time series prediction model to monitor dynamic modal changes during task execution, analyzing the modal change trend during task execution in real time, and dynamically adjusting the scheduling strategy according to the modal change trend; when all subtasks are completed, merging and verifying the results of the subtasks, and outputting the final reasoning result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, system and related device for multi-hardware hybrid large model reasoning. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large-scale multimodal models in various fields, how to efficiently deploy and run these large models has become a current technical challenge. Traditional computing platforms and hardware architectures cannot meet the performance and resource consumption requirements of large-scale models, especially in terms of computing requirements, storage bandwidth, and data transmission.

[0003] In existing technologies, the use of multiple hardware for model reasoning can integrate the advantages of different hardware. For example, the powerful parallel computing capabilities of GPU can accelerate the core part of model calculation, the programmability and low power consumption of FPGA are suitable for specific data processing tasks, and new smart chips can optimize computing efficiency in specific scenarios. Especially in terms of computing requirements, the collaboration of multiple hardware can break through the upper limit of the computing power of a single hardware and achieve more efficient computing processing. Through multi-hardware model reasoning, the deployment and operation efficiency of large-scale multimodal models can be effectively improved, and resource consumption can be reduced.

[0004] However, traditional multi-hardware reasoning task scheduling solutions often allocate tasks to different hardware resources based on pre-set rules or fixed model evaluation results, and lack the ability to dynamically respond to changes in the actual operation of the task during task execution. For example, once a task is assigned to a specific hardware, it is usually not adjusted according to changes in data characteristics. Even if the subsequent data processing requirements change, resource allocation is difficult to change flexibly, and it is impossible to effectively respond to rapid changes in data characteristics and computing loads, which ultimately leads to reduced efficiency in multimodal data processing and reduced accuracy of reasoning results. Summary of the invention

[0005] The present application provides a method, system and related devices for multi-hardware hybrid large model reasoning, which are used to improve the reasoning efficiency and flexibility of complex modal tasks.

[0006] The first aspect of the present application provides a method for multi-hardware hybrid large model reasoning, comprising:

[0007] Collecting structural information of a multimodal large model, the structural information including several modal processing branches of the multimodal large model and submodules and layer structures within the modal processing branches;

[0008] Benchmarking the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of the multiple hardware;

[0009] Evaluate the compatibility of the multimodal large model with the multiple hardware based on the performance indicators, and generate a hardware adaptation report;

[0010] Receive an inference task request, and generate a corresponding inference task according to the inference task request and store it in a task queue;

[0011] Decomposing the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages to obtain a plurality of subtasks and a plurality of computing units corresponding to each of the subtasks;

[0012] Determining the calculation type and input and output data scale of the calculation unit, and estimating the task resource requirement of the calculation unit according to the calculation type and the input and output data scale;

[0013] Formulate a scheduling strategy according to the hardware adaptation report and the task resource requirements, and allocate the computing unit to appropriate hardware for execution according to the scheduling strategy;

[0014] During the task execution process, a multimodal semantic fusion model and a time series prediction model are used to monitor dynamic modal changes, analyze the modal change trend during the task execution process in real time, and dynamically adjust the scheduling strategy according to the modal change trend;

[0015] When all the subtasks are completed, the results of the subtasks are merged and verified, and the final reasoning result is output.

[0016] Optionally, decomposing the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages to obtain a plurality of subtasks and a plurality of computing units corresponding to each of the subtasks includes:

[0017] Dividing the target reasoning task in the task queue into different task stages according to the task requirements of the target reasoning task, and determining the modal change characteristics of the target reasoning task in different task stages according to the dependency relationship and conversion rules between the different task stages;

[0018] Preliminarily decomposing the target reasoning task according to the modal processing branch and the modal change characteristic to obtain a plurality of subtasks;

[0019] The subtasks are broken down into smaller parts according to the submodules and layer structures within the modal processing branch to obtain a number of computing units.

[0020] Optionally, the multimodal semantic fusion model is constructed based on a multi-head attention mechanism and a multi-layer perceptron network, and the time series prediction model is constructed using a long short-term memory network or a gated recurrent unit;

[0021] The multimodal semantic fusion model and the time series prediction model are used to monitor dynamic modal changes during the task execution process, and the modal change trend during the task execution process is analyzed in real time, and the scheduling strategy is dynamically adjusted according to the modal change trend, including:

[0022] During the task execution, the feature vectors of different modalities are fused through the multimodal semantic fusion model to generate a multimodal semantic feature vector;

[0023] Using the multimodal semantic feature vector as the input of the time series prediction model, and determining the modal change trend during the task execution process according to the output result of the time series prediction model, wherein the time series prediction model is used to convert the multimodal semantic feature vector into time series data for prediction;

[0024] When it is determined according to the modality change trend that the scheduling strategy does not meet the task requirements after the modality change, a task migration operation is performed according to the modality change trend.

[0025] Optionally, determining the modal change trend during the task execution process according to the output result of the time series prediction model includes:

[0026] Calculating a prediction error between an output result of the time series prediction model and an actual multimodal semantic feature vector;

[0027] A comprehensive analysis is performed based on the prediction error and a preset error threshold to determine the modal change trend during the task execution process.

[0028] Optionally, in a simulation environment of multiple hardware, using multimodal data samples to perform a benchmark test on the multimodal large model to obtain performance indicators of the multiple hardware includes:

[0029] Using virtualization technology to create a hardware simulation environment for the multiple hardware, and setting parameters of the hardware simulation environment according to the architectural characteristics and performance parameters of different hardware;

[0030] Using multimodal data samples in each of the hardware simulation environments to independently test the multimodal large model, and recording first performance indicators of the multimodal large model in the hardware simulation environment, wherein the first performance indicators include inference time, peak memory usage, computing resource utilization, and power consumption;

[0031] The multimodal data samples are used in multiple hardware simulation environments to perform collaborative testing of multiple hardware combinations, and second performance indicators of different hardware combinations when processing multimodal fusion tasks are recorded, where the second performance indicators include overall inference time and data transmission volume.

[0032] Optionally, the evaluating the compatibility of the multimodal large model with the multiple hardware based on the performance indicators and generating a hardware adaptation report includes:

[0033] Determine a preset weight of the first performance indicator according to the operation requirements of the multimodal large model, and calculate a hardware adaptation score of each hardware for different modal data by using the first performance indicator and the preset weight;

[0034] Calculating the collaborative efficiency scores of the different hardware combinations by using the second performance indicator;

[0035] A hardware adaptation report is generated according to the hardware adaptation score and the collaborative efficiency score.

[0036] Optionally, the subtasks include an image recognition subtask, an audio recognition subtask, a text extraction subtask, a motion analysis subtask, a scene conversion subtask, a sentiment analysis subtask and a modal fusion subtask.

[0037] The second aspect of the present application provides a system for multi-hardware hybrid large model reasoning, including:

[0038] A collecting unit, used to collect structural information of the multimodal large model, wherein the structural information includes several modal processing branches of the multimodal large model and submodules and layer structures inside the modal processing branches;

[0039] A testing unit, used to perform a benchmark test on the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of the multiple hardware;

[0040] A generating unit, configured to evaluate the compatibility of the multimodal large model with the plurality of hardware based on the performance indicators, and generate a hardware adaptation report;

[0041] A receiving unit, used for receiving an inference task request, and generating a corresponding inference task according to the inference task request and storing it in a task queue;

[0042] A decomposition unit, used to decompose the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages, to obtain a plurality of subtasks and a plurality of computing units corresponding to each of the subtasks;

[0043] a determination unit, used to determine the calculation type and input and output data scale of the calculation unit, and estimate the task resource requirement of the calculation unit according to the calculation type and the input and output data scale;

[0044] A scheduling unit, configured to formulate a scheduling strategy according to the hardware adaptation report and the task resource requirements, and to allocate the computing unit to appropriate hardware for execution according to the scheduling strategy;

[0045] A monitoring unit, used to use a multimodal semantic fusion model and a time series prediction model to perform dynamic modal change monitoring during task execution, analyze the modal change trend during the task execution in real time, and dynamically adjust the scheduling strategy according to the modal change trend;

[0046] The output unit is used to merge and verify the results of the subtasks after all the subtasks are completed, and output the final reasoning result.

[0047] A third aspect of the present application provides a device for multi-hardware hybrid large model reasoning, the device comprising:

[0048] Processor, memory, input-output unit, and bus;

[0049] The processor is connected to the memory, the input and output unit, and the bus;

[0050] The memory stores a program, and the processor calls the program to execute the first aspect and any optional multi-hardware hybrid large model reasoning method in the first aspect.

[0051] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method for multi-hardware hybrid large model reasoning according to the first aspect and any optional item in the first aspect is executed.

[0052] It can be seen from the above technical solutions that this application has the following advantages:

[0053] By collecting and analyzing the structural information of multimodal large models, combining multiple hardware platforms for benchmarking, and evaluating hardware adaptability, the reasoning ability of large models can be dynamically evaluated under different hardware environments, providing a strong basis for hardware selection in the actual reasoning process. Before the reasoning task begins, the task is decomposed into multiple subtasks and computing units by analyzing the modal change characteristics of different task stages, and the scheduling strategy is optimized based on resource requirements and hardware adaptation reports, so that the task can obtain the best resource allocation on different hardware platforms.

[0054] Furthermore, during the task execution process, dynamic modal change monitoring is also used to analyze the modal change trend during the task execution process in real time, and the scheduling strategy is dynamically adjusted based on this, which improves the responsiveness to data changes during the reasoning process and ensures that the allocation of hardware resources during the reasoning task execution process can be adjusted in time according to the actual needs of the task. Through the method provided by this application, multiple hardware can be effectively used to cope with the complexity of multi-modal data, give full play to the value of multiple hardware and each modal data at different task stages, and improve the reasoning efficiency and flexibility of complex modal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A flowchart of an embodiment of the method for multi-hardware hybrid large model reasoning provided by the present application;

[0057] Figure 2 A flowchart of another embodiment of the method for multi-hardware hybrid large model reasoning provided by the present application;

[0058] Figure 3 A schematic diagram of the structure of an embodiment of a system for multi-hardware hybrid large model reasoning provided by the present application;

[0059] Figure 4 A schematic diagram of the structure of an embodiment of the device for multi-hardware hybrid large model reasoning provided in this application. DETAILED DESCRIPTION

[0060] The present application provides a method, system and related devices for multi-hardware hybrid large model reasoning, which are used to improve the reasoning efficiency and flexibility of complex modal tasks.

[0061] It should be noted that the large model in this application refers to a model with powerful multimodal data processing capabilities, which can integrate information of different modalities and realize cross-modal understanding, generation and reasoning. Multi-hardware refers to various hardware that provides computing resource support for multimodal large models and meets the demand for computing power during model training and reasoning. The hardware can cooperate with each other to provide computing resources for multimodal large models in a variety of combinations. The focus is on a variety of domestic hardware platforms, including computing chips such as Ascend, Cambrian, Shenwei, and Sugon, as well as different types of processors such as CPU, GPU, ASIC, etc., and also includes edge chip equipment.

[0062] See also Figure 1 , Figure 1 An embodiment of the method for multi-hardware hybrid large model reasoning provided by the present application includes:

[0063] 101. Collecting structural information of the multimodal large model, where the structural information includes several modal processing branches of the multimodal large model and submodules and layer structures within the modal processing branches;

[0064] With the widespread application of multimodal data (such as images, audio, text and other types of data coexisting) in many fields, including intelligent security, multimedia content analysis, human-computer interaction and other fields, multi-hardware hybrid large model reasoning has become an effective way to process such complex data. Multimodal large models usually have complex structures to process data of different modalities and perform fusion analysis. Collecting the structural information of multimodal large models is crucial for the subsequent efficient execution of reasoning tasks. By collecting detailed structural information of multimodal large models, including modal processing branches and their internal submodules and layer structures, the computing requirements and resource consumption characteristics of each part can be determined, which helps to reasonably allocate resources according to the characteristics of hardware resources, such as the advantages of different hardware in computing power, memory bandwidth, etc., and improve the overall reasoning efficiency.

[0065] Specifically, specialized deep learning model parsing libraries or tools can be used to collect structural information of multimodal large models. For example, in Python-based deep learning frameworks, for PyTorch, relevant methods of its torch.nn.Module class can be used to traverse the model structure. By recursively calling these methods, you can go deep into each level of the model, obtain information about each modal processing branch, and determine several modal processing branches and the submodules and layer structures within the modal processing branches.

[0066] It should be noted that the modal processing branch specifically refers to an independent part of a multimodal large model that is specifically used to process a certain modal data. For example, in a multimodal sentiment analysis model that processes images, audio, and text at the same time, the image modal processing branch is responsible for performing operations such as feature extraction and object recognition on the input image data to extract image information related to emotions (such as character expressions, scene atmosphere, etc.); the audio modal processing branch focuses on the processing of audio data, such as audio feature extraction, speech recognition (if involved), and audio sentiment feature analysis; the text modal processing branch mainly performs tasks such as lexical and syntactic analysis of text, semantic understanding, and judgment of text sentiment tendency. Within the modal processing branch, it is composed of multiple submodules or layers with specific functions. Taking the convolutional neural network in the image modal processing branch as an example, the convolution layer is a submodule that extracts features by sliding the convolution kernel on the image data. It is the core layer structure of image feature extraction; the pooling layer is another submodule that is used to reduce the data dimension, reduce the amount of calculation, and extract the main features; the fully connected layer is usually at the end of the network, which is used to integrate the extracted features and perform the final classification or regression tasks. These sub-modules and layers are combined together in a specific order and connection method to form the internal structure of the modal processing branch, and together complete the processing task of the modal data.

[0067] In practical applications, a tree structure can be used to store the collected structural information. The root node can represent the entire multimodal large model, each child node represents a modal processing branch, and the child nodes of the child nodes correspond to the sub-modules or layer structures within the branch. In the process of traversing the model, the information of each layer (such as layer type, number of parameters, input and output data dimensions, etc.) also needs to be stored in the corresponding node data structure. For example, for the image convolution layer, the convolution kernel size, step size, padding method, and number of input and output channels are stored; for the fully connected layer, the number of input and output neurons and the type of activation function are stored.

[0068] 102. Benchmark the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of multiple hardware;

[0069] Different hardware (such as domestic GPU, FPGA, ASIC, etc.) has its own unique architecture and performance characteristics. And with the continuous development of hardware technology, new hardware products emerge in an endless stream, and their performance differences are not only reflected in computing speed, but also involve memory management, data transmission efficiency, power consumption and other aspects. Therefore, it is necessary to use benchmark tests to evaluate the performance of multimodal large models on different hardware.

[0070] During the benchmarking process, a sample set covering multiple modal data is required, such as a dataset containing images, audio, text and other modalities. Multimodal data samples help evaluate the processing performance of large multimodal models on different hardware for the interaction and fusion of various modalities. In addition, data of different modalities have different structures and processing requirements. The use of multimodal data samples can identify possible bottlenecks in the hardware for each modality. Multimodal data samples can cover a variety of situations, from simple single-modal dominant tasks to complex multimodal fusion tasks. By testing these different situations, we can fully understand the performance of the hardware in various possible multimodal task scenarios, thereby obtaining more accurate hardware performance indicators.

[0071] The performance indicators obtained from the benchmark test are the key basis for evaluating the compatibility of multi-modal large models with multiple hardware. By comparing the performance data of the model on different hardware, such as running time, memory usage, computing resource utilization, power consumption, etc., it is possible to determine which hardware is more suitable for processing specific parts of the model or the entire model. For example, if a piece of hardware has high computational efficiency (low running time and high computing resource utilization) and reasonable memory usage when processing convolution operations in the image modality, then the image feature extraction subtask can be assigned to this hardware first, thereby improving the efficiency of the entire inference task and optimizing resource utilization.

[0072] 103. Evaluate the compatibility of multimodal large models with multiple hardware based on performance indicators and generate a hardware adaptation report;

[0073] By analyzing and calculating the performance indicators of the multimodal large model on multiple hardware, the degree of match between the model and the hardware is determined, and a hardware adaptation report is generated by comprehensively considering multiple factors such as the computing power, storage resources, power consumption, etc. of the hardware when processing different parts of the model. The hardware adaptation report records in detail the compatibility evaluation results of the multimodal large model and multiple hardware.

[0074] Through adaptability evaluation, we can determine which hardware the different parts of the multimodal large model are most suitable to run on, so as to achieve the optimal allocation of resources. For example, if it is found that a certain hardware has high computational efficiency and low power consumption in processing the audio feature extraction part of the model, then the audio-related subtasks can be allocated to the hardware first to avoid wasting resources on mismatched hardware and improve the resource utilization and task processing speed of the entire system. And appropriate hardware adaptation can significantly improve the inference performance of multimodal large models. When the model is well adapted to the hardware, the hardware can better meet the model's requirements in terms of computing, storage, and data transmission, reducing waiting time and resource competition.

[0075] Furthermore, for a long-term or evolving multi-modal large model, the hardware adaptation report can provide an important reference for hardware selection and upgrade. If the evaluation finds that the existing hardware has performance bottlenecks when processing a specific modality or model structure, then a more suitable hardware type or model can be selected in a targeted manner during the hardware upgrade.

[0076] 104. Receive a reasoning task request, and generate a corresponding reasoning task according to the reasoning task request and store it in a task queue;

[0077] In actual application scenarios, there will be various inference requirements from different clients or application modules. For example, in intelligent security systems, multimodal analysis of real-time surveillance videos may be required to detect abnormal behavior; in multimedia content recommendation systems, it is necessary to analyze user preferences based on video, audio and text information to recommend appropriate content. These different requirements are sent to the system in the form of task requests. The system manages and processes these requests in an orderly manner in the form of task queues, and queues and schedules tasks according to certain rules to ensure that tasks are processed in a specific order. Especially in multi-user and multi-application scenarios, different tasks may have different priorities, resource requirements and time limits. The task queue can sort tasks according to pre-set rules (such as priority, first come first served, etc.) to achieve orderly management. And by storing inference tasks in the task queue, the system can plan resource allocation in advance based on the situation of tasks in the queue. By knowing the tasks to be executed in the task queue and their characteristics, the scheduler can evaluate the hardware resources required for each task in advance, and then allocate resources to the corresponding tasks at the right time, improve the overall utilization efficiency of resources, avoid idle resources or excessive competition, and ensure the smooth operation of the system.

[0078] 105. Decompose the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages to obtain several subtasks and several computing units corresponding to each subtask;

[0079] Complex multimodal large models often have sophisticated and diverse internal structures, and different reasoning tasks will dynamically change their reliance on and utilization of different modal data at different stages during execution. For example, in a multimedia video analysis task, the initial stage may focus more on the image modality to identify the scene type, the middle stage needs to combine audio and images to analyze the specific content, and the later stage may also need to combine text, audio and images to judge the overall emotional tendency. In order to give full play to the advantages of multiple hardware resources and improve the efficiency of task execution, it is necessary to reasonably decompose the overall target reasoning task into smaller subtasks and computing units based on the model structure and the modal change characteristics of the task stage, so that resources can be allocated and scheduled more accurately in the future.

[0080] In addition to obtaining the structural information of the multimodal large model, it is also necessary to divide the target reasoning task into different stages according to the nature of the task and past experience, and analyze the importance, dominance and mutual relationship between the modal data in each stage. For example, in the video content analysis task, it can be divided into the initial scene recognition stage (the image modality is dominant, used to determine whether the video scene is indoors, outdoors, etc.), the main content analysis stage (image, audio, and text work together to identify the main characters and events in the video), and the emotional atmosphere judgment stage (comprehensive information from each modality to judge the overall emotional tendency). For each stage, it is clear how different modalities interact with each other at this stage. For example, in the main content analysis stage, the voice content in the audio may assist the character actions in the image to jointly determine the specific event, and the text description further supplements the characters and events.

[0081] According to the aforementioned model structure and the modal change characteristics of different task stages, the target reasoning task is decomposed to obtain several subtasks and several computing units corresponding to each subtask. Among them, the subtask is a relatively independent task unit with clear functions and goals obtained by decomposing the overall target reasoning task. Each subtask focuses on processing one or several aspects of the task at a certain stage. For example, the image scene classification subtask focuses on using image modal information to determine the scene type at a specific task stage. Multiple subtasks cooperate with each other to complete the entire reasoning task. The subtask can be further decomposed into computing units. The computing unit is a smaller task execution unit formed by further refining each subtask. It corresponds to a specific operation or a group of operations in the multi-modal large model structure, and has relatively clear input and output data and specific computing functions. For example, the image preprocessing unit is a computing unit in the subtask "image object recognition", which is mainly responsible for basic data processing operations such as cropping, scaling, and normalization of the input image, and provides a suitable data format and quality assurance for subsequent feature extraction and other computing units.

[0082] 106. Determine the calculation type and input and output data scale of the calculation unit, and estimate the task resource requirements of the calculation unit according to the calculation type and input and output data scale;

[0083] Different hardware has its own advantages and disadvantages in terms of computing power, memory capacity and bandwidth, data transmission speed, etc. For different computing units, due to the different nature of the operations they perform, the requirements for hardware resources vary greatly. For example, some computing units focus on computationally intensive operations such as matrix multiplication, such as the convolution computing unit in image modality processing, which requires powerful computing power to quickly process a large amount of data operations; while some computing units may pay more attention to memory read and write operations, which are memory access intensive, such as the part of text processing that involves frequent reading of word vector tables. Therefore, by determining the computing type and input and output data scale of the computing unit, and then estimating its task resource requirements, it can provide an accurate basis for allocating the computing unit to the most suitable hardware. Among them, the computing type refers to the main computing characteristics presented by the computing unit during the execution of the operation, which can usually be divided into computing intensive, memory access intensive, and computing and memory access mixed intensive categories. The input data scale refers to the amount of data when the computing unit receives data, which is determined by factors such as the dimension, format, and batch size of the data. For example, the input data scale can be composed of the size of the image data, the number of channels, and the number of images processed in batches.

[0084] It should be noted that when judging the type of calculation, some computing units may not belong to a single type, but have multiple characteristics, and it is necessary to carefully analyze the proportion of each operation and the degree of impact on the overall performance. For example, some computing units that integrate multimodal data contain a large number of matrix multiplication operations and frequent memory data reading and writing to update the modal weights. At this time, it is necessary to comprehensively judge whether it is more computationally intensive or a combination of computation and memory access intensive based on the time proportion of floating-point operations and memory access operations in actual operations, resource consumption proportion, etc., so as to more accurately estimate resource requirements.

[0085] 107. Formulate a scheduling strategy based on the hardware adaptation report and task resource requirements, and allocate computing units to appropriate hardware for execution according to the scheduling strategy;

[0086] The hardware adaptation report contains the advantages and disadvantages of each hardware in dealing with different task links of multimodal large models. This report is based on the previous benchmark test results of multimodal large models in various hardware simulation environments. By comprehensively considering multi-dimensional performance indicators such as running time, memory usage, computing resource utilization, and power consumption, it accurately evaluates the degree of adaptation of each hardware to different parts of the multimodal large model (such as different modal processing branches and sub-modules). At the same time, the task resource requirements accurately define the specific requirements of each computing unit for computing resources, storage resources, and data transmission resources during the execution process. Therefore, based on the hardware adaptation report and task resource requirements, a scheduling strategy can be formulated to allocate computing units to appropriate hardware for execution.

[0087] 108. Use multimodal semantic fusion model and time series prediction model to monitor dynamic modal changes during task execution, analyze modal change trends during task execution in real time, and dynamically adjust scheduling strategies based on modal change trends;

[0088] In complex scenarios of multimodal data processing, the information processed by the multimodal large model will not follow the modal change characteristics determined in step 105 during the task execution. The actual situation will change dynamically due to the particularity of the data, user behavior, or changes in the external environment. For example, in a video analysis task, the image modality may be crucial for scene recognition in the initial stage, but when there is a dialogue between people in the scene, the importance of the audio modality will become prominent. If there are related subtitles or text information involved, the text modality will also participate in the core processing flow, which leads to a modal change. As the modality changes, the importance and resource requirements of different modalities in the task will also change accordingly.

[0089] In this embodiment, after the task execution process is started, the feature data generated by each modal processing branch during the task execution is received through the multimodal semantic fusion model, and these scattered modal features are organically integrated into a unified multimodal semantic feature vector, and then the time series prediction model is used to analyze the continuously generated multimodal semantic feature vector to determine the modal change trend. The modal change trend specifically refers to the change trend of different modalities in importance, data characteristics, or correlation with other modalities over time during the execution of multimodal tasks. By monitoring the modal change trend, it can be understood which modality is gradually becoming dominant in the current task stage and which modality is weakening, thereby providing key information for dynamically adjusting resource allocation and task scheduling strategies.

[0090] It should be noted that the initial task decomposition and scheduling strategy formulation in steps 105 to 107 are based on the static modal change characteristics. In the absence of dynamic modal changes, the tasks can be executed in an orderly manner according to this initial plan. The scheduling strategy adjustment made by the dynamic modal change monitoring in step 108 is actually an optimization and supplement to the initial static plan, ensuring that the tasks can be efficiently processed according to the actual modal changes.

[0091] 109. When all subtasks are completed, the results of the subtasks are merged and verified, and the final reasoning results are output.

[0092] Each subtask only partially processes the overall task from a different angle or for a different modality. Their respective results cannot directly represent the final complete task conclusion. Therefore, it is necessary to integrate these scattered results and verify their accuracy and rationality to ultimately form an accurate reasoning result that meets the needs of practical applications.

[0093] In this embodiment, by collecting and analyzing the structural information of the multimodal large model, combining multiple hardware platforms for benchmarking, and evaluating the hardware adaptability, the reasoning ability of the large model can be dynamically evaluated under different hardware environments, providing a strong basis for hardware selection in the actual reasoning process. Before the reasoning task begins, by analyzing the modal change characteristics of different task stages, the task is decomposed into multiple subtasks and computing units, and the scheduling strategy is optimized according to resource requirements and hardware adaptation reports, so that the task can obtain the best resource allocation on different hardware platforms.

[0094] Furthermore, during the task execution process, dynamic modal change monitoring is also used to analyze the modal change trend during the task execution process in real time, and the scheduling strategy is dynamically adjusted based on this, which improves the responsiveness to data changes during the reasoning process and ensures that the allocation of hardware resources during the reasoning task execution process can be adjusted in time according to the actual needs of the task. Through the method provided by this application, multiple hardware can be effectively used to cope with the complexity of multi-modal data, give full play to the value of multiple hardware and each modal data at different task stages, and improve the reasoning efficiency and flexibility of complex modal tasks.

[0095] The following is a detailed description of the multi-hardware hybrid large model reasoning method provided by this application. Figure 2 , Figure 2 Another embodiment of the method for multi-hardware hybrid large model reasoning provided by the present application includes:

[0096] 201. Collecting structural information of the multimodal large model, the structural information including several modal processing branches of the multimodal large model and submodules and layer structures within the modal processing branches;

[0097] In this embodiment, step 201 is similar to step 101 in the aforementioned embodiment and will not be described again here.

[0098] 202. Use virtualization technology to create a hardware simulation environment for multiple hardware, and set parameters of the hardware simulation environment according to the architectural characteristics and performance parameters of different hardware;

[0099] By using virtualization technology to create a hardware simulation environment, multiple different hardware environments can be simulated on the same physical device at the same time, without the need to prepare a large number of real hardware devices, which greatly saves hardware costs and physical space. The architectural characteristics and performance parameters of different hardware have a decisive impact on the operation effect of multimodal large models. By setting the simulation environment parameters according to the architectural characteristics and performance parameters of the real hardware, the operation mode of the multimodal large model in the simulation environment can be as similar as possible to that on the real hardware. For example, the GPU has a multi-core parallel processing architecture, and its core number, thread scheduling mechanism and other factors are directly related to the processing efficiency of parallel computing tasks. When simulating the GPU environment, accurately setting the corresponding parameters (such as the number of virtual cores, thread block size, shared memory configuration, etc.) can make the multimodal large model perform similar to the real GPU when performing computationally intensive tasks such as image convolution in the simulation environment, providing a reliable basis for accurately evaluating the compatibility of the model with the GPU. Similarly, for reconfigurable hardware such as FPGA, by setting parameters such as the number of logic units, wiring resources, input and output pin configurations corresponding to the real FPGA in the simulation environment, it is possible to simulate its flexibility in processing different multimodal data processing logics, ensuring that the behavior of the model in the simulation environment conforms to the actual working mode of the FPGA, thereby more accurately judging the applicability of the model on the hardware.

[0100] 203. Use multimodal data samples in each hardware simulation environment to independently test the multimodal large model, and record the first performance indicators of the multimodal large model in the hardware simulation environment. The first performance indicators include inference time, peak memory usage, computing resource utilization, and power consumption;

[0101] Ensure that each hardware simulation environment has been correctly configured according to the architecture characteristics and performance parameters of the corresponding hardware (as in the previous steps), including the installation of the appropriate operating system, driver, and dependent libraries required by the multimodal large model. Import the multimodal data samples into each hardware simulation environment, run the multimodal large model separately in each hardware simulation environment, input the multimodal data samples for multiple iterative tests, and obtain stable and representative performance data, i.e., the first performance indicator.

[0102] The first performance indicator includes the inference time, which refers to the time taken by the multimodal large model from receiving the multimodal data sample as input to completing the inference task and outputting the corresponding result. It directly reflects the computing speed and efficiency of the hardware when processing the model. For data of different modalities, the hardware processing process and time consumption are very different, so it is necessary to test the inputs of different modalities separately. For example, first input a sample containing only the image modality, and record its inference time. This time reflects the efficiency of the hardware in processing image data, including the entire process from image data loading, preprocessing to feature extraction and final inference output. Similarly, the audio and text modalities are tested separately. By comparing the inference time of different modal data, the efficiency of the hardware in processing different modal data can be intuitively seen.

[0103] The first performance indicator also includes peak memory usage, computing resource utilization and power consumption. The peak memory usage reflects whether the hardware's memory resources can meet the needs of model operation. If the peak memory usage is too high and exceeds the hardware's memory capacity limit, it may cause the model to run stuck or even crash due to insufficient memory. Computing resource utilization refers to the degree to which the hardware's computing resources (such as CPU cores, GPU CUDA cores, etc.) are actually utilized when the hardware is running a large multimodal model. It is usually expressed as a percentage. A higher computing resource utilization indicates that the hardware can give full play to its computing power when processing the model. Power consumption refers to the electrical energy consumed by the hardware when running a large multimodal model, measured in units such as watts (W). In some application scenarios that have strict requirements on energy consumption (such as mobile devices, edge computing devices, etc.), power consumption is one of the key factors in selecting hardware.

[0104] 204. Use multimodal data samples to conduct collaborative testing of multiple hardware combinations in multiple hardware simulation environments, and record the second performance indicators of different hardware combinations when processing multimodal fusion tasks, where the second performance indicators include overall inference time and data transmission volume;

[0105] In a multi-hardware hybrid large model inference system, in many cases, it is not a single hardware that completes the task independently, but multiple hardware work together to handle complex multimodal fusion tasks. Different hardware can play their respective advantages through reasonable combination and collaboration, so in addition to independent testing of a single hardware, collaborative testing of multiple hardware combinations is also required.

[0106] First, according to the type and function of the hardware and the previous understanding of its individual performance, determine the different hardware combinations to be tested collaboratively. In the simulation environment of each hardware combination, install and configure the corresponding multimodal large model and the middleware or interface program required for collaborative work. The middleware can be responsible for coordinating the task allocation, data transfer and other tasks between different hardware to ensure that they can collaboratively process multimodal fusion tasks in a predetermined manner. In the simulation environment of each set hardware combination, use the same multimodal data sample to perform collaborative testing of multimodal fusion tasks to obtain the second performance indicator.

[0107] The second performance indicator includes the overall reasoning time and data transmission volume. The overall reasoning time refers to the total time spent from inputting multimodal data to outputting the final fused reasoning results when multiple hardware combinations collaboratively process multimodal fusion tasks. This time covers the transmission of data between different hardware, the processing of corresponding modal data by each hardware, and the time overhead of the final fusion stage, reflecting the speed and efficiency of the hardware combination in completing the entire multimodal task. The data transmission volume refers to the total amount of data transmitted between different hardware in the process of multiple hardware combinations collaboratively processing multimodal fusion tasks, which can reflect the degree of data interaction between hardware and the occupation of system resources by data transmission.

[0108] 205. Determine a preset weight of the first performance indicator according to the operation requirements of the multimodal large model, and calculate a hardware adaptation score of each hardware for different modal data by using the first performance indicator and the preset weight;

[0109] The multimodal large model pays different attention to various performance in different application scenarios. For example, in a multimodal video surveillance scenario with extremely high real-time requirements, since the system needs to quickly analyze and respond to modal information such as monitoring images and audio, the performance indicator of inference time is particularly important. When running a multimodal large model on some resource-constrained edge computing devices, in order to ensure that the device can run stably for a long time and will not fail due to problems such as insufficient memory, power consumption and peak memory usage may become key considerations. Based on this, in this embodiment, it is necessary to analyze the operating requirements of the multimodal large model, determine the application scenarios and task objectives, and determine the preset weight of the first performance indicator accordingly.

[0110] After that, the hardware adaptation score is calculated for each hardware and different modal data. The hardware adaptation score is a quantitative indicator, which is obtained by multiplying the first performance indicator value of each hardware when processing different modal data with the corresponding preset weight and then accumulating it. The hardware adaptation score comprehensively reflects the degree of adaptation of the hardware to the specific modal data in terms of various performance aspects. The higher the score, the more adapted the hardware is to process the modal data, which facilitates comparison and selection among multiple hardware, thereby providing a basis for hardware allocation.

[0111] Specifically, taking the image feature extraction part as an example, assuming that the AHP analysis determines that the running time weight is 0.5, the memory usage weight is 0.3, and the power consumption weight is 0.2, the process of calculating the hardware adaptation score when processing image data is as follows:

[0112] First, calculate the normalized values ​​of each performance indicator. Assume that the running time of a certain hardware for image feature extraction is t seconds, the minimum running time of this part of all hardware is t_min, and the maximum running time is t_max. Then the normalized value of the running time is n_t=(t_max-t) / (t_max-t_min). Here, reverse normalization is used, that is, the shorter the running time, the higher the score. Similarly, calculate the normalized values ​​n_m and n_p of memory usage and power consumption. Then calculate the adaptation score S=0.5×n_t+0.3×n_m+0.2×n_p based on the weight. Similar adaptation score calculations are performed for other parts of the model (such as text processing part, multimodal fusion part, etc.).

[0113] 206. Calculate the collaborative efficiency scores of different hardware combinations through the second performance indicator;

[0114] Different hardware combinations have different performances in overall reasoning time and data transmission volume when performing multimodal fusion tasks. Some hardware combinations may have a shorter overall reasoning time, but a large data transmission volume, which means that more resources may be consumed in data interaction between hardware; while some combinations have relatively small data transmission volumes, but longer overall reasoning times, reflecting deficiencies in computing efficiency. Therefore, in this embodiment, these two key second performance indicators are considered together through the collaborative efficiency score, and a specific numerical value is used to quantify the collaborative effect of the hardware combination, which is convenient for intuitively comparing the quality of collaborative work between different hardware combinations.

[0115] Specifically, the collaborative efficiency score can be directly measured by calculating the ratio of the overall reasoning time to the data transmission volume. In this calculation method, the lower the collaborative efficiency score, the faster the task can be completed with relatively less data transmission volume, that is, the higher the collaborative efficiency of the hardware combination. However, this method is relatively simple and intuitive, and does not fully consider the impact of factors such as the complexity of different tasks on the collaborative efficiency. Therefore, a benchmark value comparison calculation method can be introduced, that is, first determine the benchmark overall reasoning time (T_benchmark) and benchmark data transmission volume (D_benchmark) corresponding to each multimodal fusion task under ideal conditions (such as on a single hardware with optimal performance or under the theoretical optimal collaborative state). Then, the collaborative efficiency score is calculated by the following formula: Score=(T_benchmark×D_benchmark) / (T×D). This formula compares the performance indicators of the actual hardware combination with the benchmark value, and more comprehensively considers the characteristics of the task itself and the collaborative efficiency of the hardware combination relative to the ideal state. A collaborative efficiency score greater than 1 indicates that the collaborative efficiency of the hardware combination is better than the benchmark. The higher the score, the better the collaborative efficiency; a score less than 1 indicates that the collaborative efficiency is not as good as the benchmark. In addition, the overall reasoning time and data transmission volume can be calculated by assigning them certain weights according to the different emphases of the multimodal large model, which are not limited here.

[0116] 207. Generate a hardware adaptation report according to the hardware adaptation score and the collaborative efficiency score;

[0117] The hardware adaptation score can reflect the degree of adaptation of a single hardware to different modal data, and show the compatibility of its various performance aspects with a specific modality from the perspective of individual hardware; the collaborative efficiency score focuses on reflecting the overall efficiency of different hardware combinations in collaboratively processing multimodal fusion tasks. By integrating these two scores and generating a hardware adaptation report, it is possible to evaluate the degree of adaptation of a single hardware to different modal data, and reflect the efficiency of different hardware combinations in collaboratively processing multimodal tasks. In this way, hardware resources can be accurately allocated according to the needs of different modal tasks in actual applications, providing a comprehensive hardware performance reference.

[0118] 208. Receive a reasoning task request, and generate a corresponding reasoning task according to the reasoning task request and store it in a task queue;

[0119] In this embodiment, step 208 is similar to step 104 in the above embodiment and will not be described again here.

[0120] 209. Divide the target reasoning task in the task queue into different task stages according to the task requirements of the target reasoning task, and determine the modal change characteristics of the target reasoning task in different task stages according to the dependency relationship and conversion rules between different task stages;

[0121] The reasoning tasks processed by multimodal large models are often more complex, involving data of multiple modalities and multiple processing links. In order to process these tasks more orderly and efficiently, it is necessary to divide them into reasonable stages according to the characteristics and requirements of the tasks themselves, so as to facilitate subsequent resource scheduling. Different task stages usually have different goals and operational priorities. By dividing the stages, the execution process of the entire reasoning task can be clearly sorted out. For example, in a multimodal video content analysis task, it can be divided into a video frame image preprocessing stage, an image object recognition stage, an audio feature extraction stage, and a content understanding stage based on image and audio information. The requirements for hardware resources at each task stage are often different. After dividing the stages, resource allocation can be planned in advance according to the characteristics of different stages.

[0122] Specifically, the task stages can be divided according to the goals set in the reasoning task request and the logical order of achieving the goals. For common multimodal task types, the stages can be divided directly based on the experience of handling similar tasks in the past. Dividing the stages based on existing mature experience helps to ensure the rationality and effectiveness of the division.

[0123] Since data of different modalities often have a sequence and inherent logical association when being processed, after dividing the task stages, it is necessary to further check the input-output dependency and conversion rules between different modal data in each task stage. For example, in the multimodal intelligent security monitoring task, in the early stage of monitoring scene construction, the image modality plays a leading role, and the basic environment of the monitoring area, such as indoor and outdoor scenes, the approximate distribution of people and objects, is determined by the images captured by the camera. At this time, the audio modality may be in an auxiliary or inactive state. However, when suspicious personnel activities appear in the picture and trigger the need for sound monitoring, the audio modality begins to play a key role, such as by analyzing the sound to determine the content of the personnel's conversation, whether there is abnormal sound, etc., and at this time, the effective processing of the audio modality depends on the information such as the personnel position that has been determined by the image modality. Therefore, by sorting out the dependency relationship between these modalities, it can be clearly seen that as the task progresses, each modality changes from relative independence to mutual dependence, and then grasps the modal change characteristics of the target reasoning task at different task stages.

[0124] 210. Preliminarily decompose the target reasoning task according to the modal processing branches and modal change characteristics to obtain several subtasks;

[0125] In order to perform the reasoning task more efficiently, after the task is divided into stages and the modal change characteristics are mastered, the overall reasoning task can be initially decomposed into relatively independent and interrelated subtasks. Specifically, according to the existing modal processing branches in the multimodal large model and the modal change characteristics of the target reasoning task determined in the previous steps, the main processing content corresponding to each branch can be divided into subtasks. For example, in a multimodal large model that processes images, audio, and text at the same time, the basic subtasks corresponding to the image modal processing branch can include image preprocessing subtasks (responsible for image cropping, normalization, etc.), image feature extraction subtasks (using convolution and other operations to extract image features), image classification or recognition subtasks (judging the object category in the image based on the extracted features, etc.); the audio modal branch can be divided into audio preprocessing subtasks, audio feature extraction subtasks, audio event recognition subtasks, etc.; the text modal branch can similarly determine subtasks such as text preprocessing, text feature extraction, and text semantic understanding. In addition, it also involves action analysis subtasks, scene conversion subtasks, sentiment analysis subtasks, and modal fusion subtasks.

[0126] 211. According to the sub-modules and layer structures within the modal processing branch, the sub-tasks are refined and decomposed to obtain several computing units;

[0127] Subtasks are only a preliminary decomposition of the target reasoning task. Each subtask is still relatively complex and contains multiple specific operations and calculation links. In order to allocate resources more accurately, schedule tasks more meticulously, and better match the underlying computing capabilities of the hardware, it is necessary to continue to go deep into the subtasks and further refine the subtasks into smaller computing units based on the specific sub-modules and layer structures of the modal processing branches. These computing units are the basic units for the multimodal large model to actually perform operations on hardware.

[0128] Specifically, for each subtask, the submodules and layer structures within the corresponding modal processing branch are deeply analyzed. Taking the image feature extraction subtask as an example, it contains submodules and layer structures such as convolution layer, pooling layer, and activation function layer. Each layer performs specific calculation operations, such as the convolution layer performs convolution operations to extract local features of the image, the pooling layer performs dimensionality reduction operations to reduce the amount of data and retain key features, and the activation function layer performs nonlinear transformations on the features. According to the internal structure analyzed above, the subtasks are further decomposed into specific calculation units. For example, the image feature extraction subtask is refined into a convolution calculation unit (responsible for performing convolution operations), a pooling calculation unit (performing pooling operations), an activation calculation unit (performing activation function related calculations), etc. Each calculation unit has clear input and output data and specific calculation functions, which facilitates subsequent operations such as resource demand estimation and hardware allocation.

[0129] 212. Determine the computing type and input and output data scale of the computing unit, and estimate the task resource requirements of the computing unit based on the computing type and input and output data scale;

[0130] 213. Formulate a scheduling strategy based on the hardware adaptation report and task resource requirements, and allocate computing units to appropriate hardware for execution according to the scheduling strategy;

[0131] In this embodiment, steps 212-213 are similar to steps 106-107 of the aforementioned embodiment and are not described again here.

[0132] 214. During the task execution process, the feature vectors of different modalities are fused through the multimodal semantic fusion model to generate a multimodal semantic feature vector;

[0133] During the task execution, feature vectors of different modalities can be obtained by performing corresponding feature extraction operations on data of different modalities. At this time, the multimodal semantic fusion model is used to effectively integrate and fuse these feature vectors from different modalities, dig out the semantic associations between them, and generate a multimodal semantic feature vector that can comprehensively reflect multimodal information.

[0134] For this multimodal semantic fusion model, the multi-head attention mechanism is used as the core component. The feature vectors of each modality are used as the input query, key and value of the multi-head attention mechanism. The multi-head attention mechanism can simultaneously focus on the semantic associations between different modalities in different subspaces. For example, one head can focus on the association between the image and the name of the object in the text, and the other head can focus on the matching of the audio emotion and the color emotion of the image. In this way, the model can comprehensively fuse the multimodal semantic information and generate a fused multimodal semantic feature vector. After the multi-head attention mechanism, a multi-layer perceptron (MLP) network is connected to further process and analyze the fused multimodal semantic feature vector. The MLP network can learn the complex nonlinear relationship between the multimodal semantic feature vectors, so that the semantic change trend can be more accurately predicted.

[0135] 215. Using the multimodal semantic feature vector as the input of the time series prediction model, and determining the modal change trend during the task execution process according to the output result of the time series prediction model. The time series prediction model is used to convert the multimodal semantic feature vector into time series data for prediction;

[0136] The time series prediction model is designed to process data arranged in chronological order and predict future trends or values ​​based on the rules of historical data. By using the multimodal semantic feature vector generated by the multimodal semantic fusion model as the input of the time series prediction model, the time series of the multimodal semantic feature vector is modeled using a time series prediction algorithm, such as the long short-term memory network (LSTM) or the gated recurrent unit (GRU). Since this vector integrates the semantic information of different modalities and contains the status and relationship of each modality during the task execution process, the time series prediction model converts these input vectors into time series data by performing a series of mathematical transformations and analyzing the patterns that change over time, thereby predicting the trend of modal changes in the future.

[0137] Specifically, when determining the modal change trend, it is necessary to calculate the prediction error between the output result of the time series prediction model and the actual multimodal semantic feature vector; based on the prediction error and the preset error threshold, a comprehensive analysis is performed to determine the modal change trend during task execution.

[0138] The time series prediction model is used to predict the change of multimodal semantic feature vectors over time. The prediction error can be calculated with specific numerical values ​​to measure the degree of deviation between the model output and the actual situation. For example, indicators such as mean square error (MSE) and mean absolute error (MAE) can objectively reflect the accuracy of model prediction by calculating the difference between the predicted value and the actual value in each dimension. The multimodal semantic feature vector contains information from different modalities and is a multi-dimensional data representation. The prediction error can be calculated to evaluate the difference between the model output and the actual vector for each dimension or as a whole. The preset error threshold is a reference standard used to distinguish whether the model prediction can effectively track the modal change. If the prediction error is within the threshold range, it means that the model can better capture the modal change law, and the current modal change is relatively stable, which is in line with the model's expectations. When the prediction error exceeds the threshold, it indicates that the actual modal change has a large deviation from the model's expectations, which may be due to changes in the characteristics of the modality itself. For example, in a multimodal sentiment analysis task, if the error between the predicted sentiment semantic feature vector and the actual one exceeds a threshold, it may be because a new emotional factor (such as a sudden strong emotional expression) appears in a certain modality (such as a change in facial expression in an image, a sudden change in intonation in audio, etc.), causing the balance and relationship between the modalities to change, thereby determining that the modality has a change trend that needs attention.

[0139] As the task progresses, the importance of the modality and data distribution may change. By constantly comparing the prediction error and threshold, the system can dynamically adapt to modal changes. This comparison mechanism can timely discover the trend of modal changes, and thus adjust the resource allocation and task scheduling strategies based on this.

[0140] 216. When it is determined according to the modal change trend that the scheduling strategy does not meet the task requirements after the modal change, a task migration operation is performed according to the modal change trend;

[0141] As the task progresses, the actual change trend of the modality may deviate greatly from the expected trend, resulting in the originally set scheduling strategy being unable to adapt well to the new task requirements. The data characteristics, importance, and mutual relationship of each modality in the reasoning task will change dynamically over time. This change may be caused by various reasons such as external environmental factors and the uncertainty of the data itself. If timely adjustments are not made based on the trend of modal changes, the efficiency of task execution and the accuracy of results will be seriously affected.

[0142] Therefore, by monitoring the modal change trend in real time, and judging whether the current scheduling strategy meets the task requirements after the modal change according to the modal change trend, if not, analyze the adaptability of each hardware to the changed modality, and select the most suitable hardware as the target of task migration. Through the task migration operation, tasks can be reallocated to hardware resources that are more suitable for handling the changed modal situation, or the execution order of subtasks can be changed, so that the system can flexibly adapt to these dynamic changes and ensure that tasks are always performed under the optimal configuration.

[0143] 217. When all subtasks are completed, the results of the subtasks are merged and verified, and the final reasoning results are output.

[0144] In this embodiment, step 217 is similar to step 109 in the aforementioned embodiment and will not be described again here.

[0145] In this embodiment, by using virtualization technology to create a hardware simulation environment, independent tests and collaborative tests based on the simulation environment are conducted to comprehensively and deeply obtain the first performance index and the second performance index of the hardware and hardware combination, and then determine the weights according to the multimodal large model operation requirements and calculate the hardware adaptation score and the collaborative efficiency score, thereby generating a hardware adaptation report to achieve accurate evaluation and matching of hardware resources. In addition, a multimodal semantic fusion model and a time series prediction model are introduced during the task execution process to dynamically monitor the modal change trend. This dynamic monitoring mechanism enables the system to no longer be limited to fixed task scheduling strategies, but can be flexibly adjusted according to real-time changes in modal data. When the modal change causes the original scheduling strategy to fail to meet the task requirements, a task migration operation is performed, which further ensures the continuity and efficiency of task execution, comprehensively improves the operating efficiency of the multimodal large model in multiple hardware environments, and enhances the adaptability and responsiveness to complex multimodal reasoning tasks.

[0146] The following is a detailed description of the multi-hardware hybrid large model reasoning system provided by this application. Figure 3 , Figure 3Another embodiment of the system for multi-hardware hybrid large model reasoning provided by the present application includes:

[0147] A collecting unit 301 is used to collect structural information of the multimodal large model, where the structural information includes several modal processing branches of the multimodal large model and submodules and layer structures within the modal processing branches;

[0148] A testing unit 302 is used to perform a benchmark test on the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of multiple hardware;

[0149] A generating unit 303 is used to evaluate the compatibility of the multimodal large model with multiple hardware based on the performance index and generate a hardware adaptation report;

[0150] The receiving unit 304 is used to receive the reasoning task request, and generate a corresponding reasoning task according to the reasoning task request and store it in the task queue;

[0151] A decomposition unit 305 is used to decompose the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages to obtain a plurality of subtasks and a plurality of computing units corresponding to each subtask;

[0152] A determination unit 306, used to determine the calculation type and input and output data scale of the calculation unit, and estimate the task resource requirements of the calculation unit according to the calculation type and input and output data scale;

[0153] The scheduling unit 307 is used to formulate a scheduling strategy according to the hardware adaptation report and the task resource requirements, and allocate the computing unit to the appropriate hardware for execution according to the scheduling strategy;

[0154] The monitoring unit 308 is used to use the multimodal semantic fusion model and the time series prediction model to perform dynamic modality change monitoring during the task execution process, analyze the modality change trend during the task execution process in real time, and dynamically adjust the scheduling strategy according to the modality change trend;

[0155] The output unit 309 is used to merge and verify the results of the subtasks after all subtasks are completed, and output the final reasoning result.

[0156] Optionally, the decomposition unit 305 is specifically used for:

[0157] The target reasoning tasks in the task queue are divided into different task stages according to the task requirements of the target reasoning tasks, and the modal change characteristics of the target reasoning tasks in different task stages are determined according to the dependency relationship and conversion rules between the different task stages;

[0158] The target reasoning task is preliminarily decomposed according to the modal processing branches and modal change characteristics to obtain several subtasks;

[0159] The subtasks are broken down into smaller parts according to the submodules and layer structures within the modal processing branch to obtain several computing units.

[0160] Optionally, the multimodal semantic fusion model is built based on the multi-head attention mechanism and the multi-layer perceptron network, and the time series prediction model is built using the long short-term memory network or the gated recurrent unit;

[0161] The monitoring unit 308 is specifically used for:

[0162] During the task execution, the feature vectors of different modalities are fused through the multimodal semantic fusion model to generate a multimodal semantic feature vector;

[0163] The multimodal semantic feature vector is used as the input of the time series prediction model, and the modal change trend during the task execution is determined according to the output result of the time series prediction model. The time series prediction model is used to convert the multimodal semantic feature vector into time series data for prediction;

[0164] When it is determined according to the modal change trend that the scheduling strategy does not meet the task requirements after the modal change, a task migration operation is performed according to the modal change trend.

[0165] Optionally, the monitoring unit 308 is further configured to:

[0166] Calculate the prediction error between the output of the time series prediction model and the actual multimodal semantic feature vector;

[0167] A comprehensive analysis is performed based on the prediction error and the preset error threshold to determine the modal change trend during task execution.

[0168] Optionally, the testing unit 302 is specifically used for:

[0169] Use virtualization technology to create a hardware simulation environment for multiple hardware, and set the parameters of the hardware simulation environment according to the architectural characteristics and performance parameters of different hardware;

[0170] Use multimodal data samples in each hardware simulation environment to independently test the multimodal large model, and record the first performance indicators of the multimodal large model in the hardware simulation environment. The first performance indicators include inference time, peak memory usage, computing resource utilization, and power consumption;

[0171] Use multimodal data samples in multiple hardware simulation environments to conduct collaborative testing of multiple hardware combinations, and record the second performance indicators of different hardware combinations when processing multimodal fusion tasks. The second performance indicators include overall inference time and data transmission volume.

[0172] Optionally, the generating unit 303 is specifically configured to:

[0173] Determine a preset weight of the first performance indicator according to the operation requirements of the multimodal large model, and calculate the hardware adaptation score of each hardware for different modal data through the first performance indicator and the preset weight;

[0174] Calculate the collaborative efficiency scores of different hardware combinations through the second performance indicator;

[0175] Generate a hardware adaptation report based on the hardware adaptation score and the collaborative efficiency score.

[0176] Optionally, the subtasks include an image recognition subtask, an audio recognition subtask, a text extraction subtask, a motion analysis subtask, a scene conversion subtask, a sentiment analysis subtask, and a modal fusion subtask.

[0177] In this embodiment, the functions of each unit are the same as those described above. Figure 1 or Figure 2 The steps in the method embodiment shown correspond to each other and will not be repeated here.

[0178] This application also provides a device for multi-hardware hybrid large model reasoning, please refer to Figure 4 , Figure 4 An embodiment of a device for multi-hardware hybrid large model reasoning provided by the present application includes:

[0179] Processor 401, memory 402, input and output unit 403, bus 404;

[0180] The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404;

[0181] The memory 402 stores a program, and the processor 401 calls the program to execute any of the above multi-hardware hybrid large model reasoning methods.

[0182] The present application also relates to a computer-readable storage medium on which a program is stored. When the program is run on a computer, the computer executes any of the above multi-hardware hybrid large model reasoning methods.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A method for multi-hardware hybrid large model reasoning, characterized in that: The method comprises: Collecting structural information of a multimodal large model, the structural information including several modal processing branches of the multimodal large model and submodules and layer structures within the modal processing branches; Benchmarking the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of the multiple hardware; Evaluate the compatibility of the multimodal large model with the multiple hardware based on the performance indicators, and generate a hardware adaptation report; Receive an inference task request, and generate a corresponding inference task according to the inference task request and store it in a task queue; Decomposing the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages to obtain a plurality of subtasks and a plurality of computing units corresponding to each of the subtasks; Determining the calculation type and input and output data scale of the calculation unit, and estimating the task resource requirement of the calculation unit according to the calculation type and the input and output data scale; Formulate a scheduling strategy according to the hardware adaptation report and the task resource requirements, and allocate the computing unit to appropriate hardware for execution according to the scheduling strategy; During the task execution process, a multimodal semantic fusion model and a time series prediction model are used to monitor dynamic modal changes, analyze the modal change trend during the task execution process in real time, and dynamically adjust the scheduling strategy according to the modal change trend; When all the subtasks are completed, the results of the subtasks are merged and verified, and the final reasoning result is output; The multimodal semantic fusion model is constructed based on a multi-head attention mechanism and a multi-layer perceptron network, and the time series prediction model is constructed using a long short-term memory network or a gated recurrent unit; The multimodal semantic fusion model and the time series prediction model are used to monitor dynamic modal changes during the task execution process, and the modal change trend during the task execution process is analyzed in real time, and the scheduling strategy is dynamically adjusted according to the modal change trend, including: During the task execution, the feature vectors of different modalities are fused through the multimodal semantic fusion model to generate a multimodal semantic feature vector; Using the multimodal semantic feature vector as the input of the time series prediction model, and determining the modal change trend during the task execution process according to the output result of the time series prediction model, wherein the time series prediction model is used to convert the multimodal semantic feature vector into time series data for prediction; When it is determined according to the modality change trend that the scheduling strategy does not meet the task requirements after the modality change, a task migration operation is performed according to the modality change trend.

2. The method according to claim 1, characterized in that The target reasoning task in the task queue is decomposed according to the structural information of the multimodal large model and the modal change characteristics of different task stages to obtain a plurality of subtasks and a plurality of computing units corresponding to each of the subtasks, including: Dividing the target reasoning task in the task queue into different task stages according to the task requirements of the target reasoning task, and determining the modal change characteristics of the target reasoning task in different task stages according to the dependency relationship and conversion rules between the different task stages; Preliminarily decomposing the target reasoning task according to the modal processing branch and the modal change characteristic to obtain a plurality of subtasks; The subtasks are broken down into smaller parts according to the submodules and layer structures within the modal processing branch to obtain a number of computing units.

3. The method according to claim 1, characterized in that Determining the modal change trend during the task execution process according to the output result of the time series prediction model includes: Calculating a prediction error between an output result of the time series prediction model and an actual multimodal semantic feature vector; A comprehensive analysis is performed based on the prediction error and a preset error threshold to determine the modal change trend during the task execution process.

4. The method according to claim 1, characterized in that: Benchmarking the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of the multiple hardware includes: Using virtualization technology to create a hardware simulation environment for the multiple hardware, and setting parameters of the hardware simulation environment according to the architectural characteristics and performance parameters of different hardware; Using multimodal data samples in each of the hardware simulation environments to independently test the multimodal large model, and recording first performance indicators of the multimodal large model in the hardware simulation environment, wherein the first performance indicators include inference time, peak memory usage, computing resource utilization, and power consumption; The multimodal data samples are used in multiple hardware simulation environments to perform collaborative testing of multiple hardware combinations, and second performance indicators of different hardware combinations when processing multimodal fusion tasks are recorded, where the second performance indicators include overall inference time and data transmission volume.

5. The method according to claim 4, characterized in that The step of evaluating the compatibility of the multimodal large model with the plurality of hardware based on the performance indicators and generating a hardware adaptation report includes: Determine a preset weight of the first performance indicator according to the operation requirements of the multimodal large model, and calculate a hardware adaptation score of each hardware for different modal data by using the first performance indicator and the preset weight; Calculating the collaborative efficiency scores of the different hardware combinations by using the second performance indicator; A hardware adaptation report is generated according to the hardware adaptation score and the collaborative efficiency score.

6. The method according to any one of claims 1 to 5, characterized in that The subtasks include an image recognition subtask, an audio recognition subtask, a text extraction subtask, a motion analysis subtask, a scene conversion subtask, a sentiment analysis subtask and a modal fusion subtask.

7. A multi-hardware hybrid large model reasoning system, characterized in that: The system comprises: A collecting unit, used to collect structural information of the multimodal large model, wherein the structural information includes several modal processing branches of the multimodal large model and submodules and layer structures inside the modal processing branches; A testing unit, used to perform a benchmark test on the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain performance indicators of the multiple hardware; A generating unit, configured to evaluate the compatibility of the multimodal large model with the plurality of hardware based on the performance indicators, and generate a hardware adaptation report; A receiving unit, used for receiving an inference task request, and generating a corresponding inference task according to the inference task request and storing it in a task queue; A decomposition unit, used to decompose the target reasoning task in the task queue according to the structural information of the multimodal large model and the modal change characteristics of different task stages, to obtain a plurality of subtasks and a plurality of computing units corresponding to each of the subtasks; a determination unit, used to determine the calculation type and input and output data scale of the calculation unit, and estimate the task resource requirement of the calculation unit according to the calculation type and the input and output data scale; A scheduling unit, configured to formulate a scheduling strategy according to the hardware adaptation report and the task resource requirements, and to allocate the computing unit to appropriate hardware for execution according to the scheduling strategy; A monitoring unit, used to use a multimodal semantic fusion model and a time series prediction model to perform dynamic modal change monitoring during task execution, analyze the modal change trend during the task execution in real time, and dynamically adjust the scheduling strategy according to the modal change trend; An output unit, used to merge and verify the results of the subtasks after all the subtasks are completed, and output the final reasoning result; The multimodal semantic fusion model is constructed based on a multi-head attention mechanism and a multi-layer perceptron network, and the time series prediction model is constructed using a long short-term memory network or a gated recurrent unit; The monitoring unit is specifically used for: During the task execution, the feature vectors of different modalities are fused through the multimodal semantic fusion model to generate a multimodal semantic feature vector; Using the multimodal semantic feature vector as the input of the time series prediction model, and determining the modal change trend during the task execution process according to the output result of the time series prediction model, wherein the time series prediction model is used to convert the multimodal semantic feature vector into time series data for prediction; When it is determined according to the modality change trend that the scheduling strategy does not meet the task requirements after the modality change, a task migration operation is performed according to the modality change trend.

8. A device for multi-hardware hybrid large model reasoning, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the method according to any one of claims 1 to 6 is performed.

Images