Autonomous controllable computing power-oriented multi-modal large model training precision alignment and optimization method and system

Abstract

The invention discloses a multi-modal large model training precision alignment and optimization method and system oriented to autonomous controllable computing power. The method comprises the following steps: establishing a dual parallel scheduling framework of a heterogeneous computing environment, and realizing job-level concurrence of forward derivation and weight adjustment; formulating a classification processing scheme based on gradient features, and respectively processing the operation-intensive gradient and the storage-intensive gradient; carrying out model simplification and reconstruction of multi-modal identification, configuring a combined convolution unit in a visual path, and fusing a multi-attention mechanism; creating a hardware sensing precision compression conversion method, and selecting a compression strategy according to the characteristics of a domestic chip; an end-to-end integrated training configuration system is established, and a complete service chain is provided through a standardized interface. According to the method, efficient training and precision maintenance of the multi-modal large model are realized on an autonomous controllable computing power platform through a multi-level optimization strategy, and the training efficiency and the resource utilization rate are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence computing technology, in particular to a multi-modal large model training precision alignment and optimization method and system for autonomous controllable computing power. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-modal large models are increasingly widely used in computer vision, natural language processing and other fields. However, the existing technology in the training process of multi-modal large models faces the following outstanding problems: First, the resource utilization rate of traditional training methods on heterogeneous computing platforms is low. Due to the difference in computing characteristics between GPU and CPU, the conventional parallel scheduling strategy is difficult to effectively coordinate the forward derivation and parameter update tasks, resulting in idle computing resources and low training efficiency. Especially when dealing with large-scale multi-modal data, this resource waste phenomenon is more pronounced.

[0003] Second, the existing gradient processing scheme lacks targeted optimization of gradient features. In the data parallel training scene, the gradients generated by different network layers have significantly different computing and storage characteristics, but existing methods usually adopt a unified processing strategy, which not only increases communication overhead, but also causes tension in video memory resources.

[0004] Third, there is a contradiction between the complexity of multi-modal models and deployment needs. In order to maintain the precision of multi-modal feature alignment, the model often needs a large amount of parameters and computation, which conflicts with the limited computing resources of edge devices. Existing model compression methods often ignore the particularity of multi-modal tasks, resulting in serious precision loss.

[0005] In addition, the ecological compatibility of domestic autonomous controllable chips is insufficient. Different manufacturers' AI chips differ in computing power architecture, instruction set, etc., making it difficult for trained models to seamlessly migrate between different chip platforms, increasing deployment costs and difficulties.

[0006] Finally, the existing training system lacks an end-to-end integrated solution. Each link from data preprocessing to model deployment often requires human intervention, which not only increases the operation complexity, but also affects the overall efficiency of the training process.

[0007] Therefore, there is an urgent need for a comprehensive solution that can fully utilize autonomous controllable computing power resources to achieve efficient training and precision alignment of multi-modal large models. SUMMARY

[0008] The purpose of the present application is to provide a multi-modal large model training precision alignment and optimization method and system for autonomous controllable computing power, which solves the problems of low resource utilization, low training efficiency, and large model compression precision loss in the prior art.

[0009] In a first aspect, the embodiments of the present application provide a multi-modal large model training precision alignment and optimization method for autonomous controllable computing power, which comprises: A dual parallel scheduling framework of a heterogeneous computing environment is established to realize concurrent execution of forward derivation and weight adjustment tasks of the model at the job level and to realize sharing of computing resources between the accelerator and the host processor at the system level; A classification processing scheme based on gradient characteristics is formulated, and ring global aggregation between accelerators and unified processing of the host processor memory are respectively adopted for operation-intensive and storage-intensive gradients; Model simplification and reconstruction of multi-modal recognition are carried out, and combined convolution units are configured in the visual path and a multiple attention mechanism is fused; A hardware-aware precision compression conversion method is created, and a compression strategy is selected according to the operation performance parameters of the target chip and platform compatibility is completed with the help of the device abstraction layer; An end-to-end integrated training configuration system is constructed to integrate parallel scheduling, gradient processing, simplified design and precision conversion functions, and to provide a complete service chain from data injection to terminal verification through a standardized interface.

[0010] In a second aspect, the embodiments of the present application provide a multi-modal large model training precision alignment and optimization system for autonomous controllable computing power, which is applied to the multi-modal large model training precision alignment and optimization method for autonomous controllable computing power as described in the first aspect, and comprises: A dual parallel scheduling module is configured to establish a dual parallel scheduling framework of a heterogeneous computing environment to realize concurrent execution of forward derivation and weight adjustment at the job level and sharing of computing resources between the accelerator and the host processor; A gradient classification processing module is configured to formulate a classification processing scheme based on gradient characteristics, and to respectively adopt ring global aggregation between accelerators and unified processing of the host processor memory for operation-intensive and storage-intensive gradients; A model simplification and reconstruction module is configured to carry out model simplification and reconstruction of multi-modal recognition, and to configure combined convolution units in the visual path and fuse a multiple attention mechanism; A precision compression conversion module is configured to create a hardware-aware precision compression conversion method, and to select a compression strategy according to the operation performance parameters of the target chip and to complete platform compatibility with the help of the device abstraction layer; A training configuration integration module is configured to construct an end-to-end integrated training configuration system to integrate parallel scheduling, gradient processing, simplified design and precision conversion functions, and to provide a complete service chain from data injection to terminal verification through a standardized interface.

[0011] In a third aspect, the embodiments of the present application provide an electronic device, which comprises: A processor; A memory for storing processor-executable instructions; The processor is configured to implement the method for training precision alignment and optimization of a multi-modal large model facing autonomous controllable computing power as in the first aspect when executing the instructions.

[0012] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium storing a program, the program instructing a device to execute the method for training precision alignment and optimization of a multi-modal large model facing autonomous controllable computing power as in the first aspect.

[0013] The present application has the following beneficial effects: 1. Resource utilization efficiency is significantly improved. By establishing a dual parallel scheduling framework of heterogeneous computing environments, the optimal allocation of computing resources between GPUs and CPUs is achieved. Coarse-grained parallel scheduling ensures task-level concurrent execution, and fine-grained scheduling fully utilizes the idle computing power of accelerators, thereby improving the overall computing resource utilization and effectively solving the problem of idle resources on traditional heterogeneous platforms.

[0014] 2. Training efficiency is greatly improved. The classification processing scheme based on gradient characteristics adopts differentiated processing strategies for different types of gradients. Operation-intensive gradients are synchronized efficiently through ring global aggregation, and storage-intensive gradients are processed uniformly through CPU memory to reduce communication overhead, thereby improving training speed and reducing memory occupancy.

[0015] 3. Model precision is effectively maintained. The model simplification and reconstruction of multi-modal perception combine convolution units and multiple attention mechanisms to reduce model complexity while maintaining feature expression capability. The progressive compression strategy and knowledge distillation technology ensure that the performance loss of the model on multi-modal tasks is controlled within 2%, achieving the best balance between precision and efficiency.

[0016] 4. Cross-platform deployment capability is enhanced. The hardware-aware precision compression conversion method decouples the model and the chip platform through the device abstraction layer, supporting seamless migration between different domestic AI chips. The adaptive selection of quantization strategies and dynamic precision adjustment function enable the same model to run efficiently on multiple autonomous and controllable chips, reducing deployment costs.

[0017] 5. High degree of automation in the whole process. The end-to-end integrated training configuration system realizes full-process automation from data injection to terminal verification through modular service orchestration and intelligent workflow management. The unified standardized interface and visual operation interface greatly reduce the use threshold, improve system usability and maintenance efficiency.

[0018] 6. System stability and reliability are improved. The full-link monitoring and diagnosis platform collects performance data of each stage of training in real time, provides automatic alarm and root cause analysis functions. Self-adaptive load balancing and dynamic resource scheduling mechanism ensures stable operation of the system under complex load, and shortens the fault recovery time. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of a multi-modal large model training precision alignment and optimization method for autonomous controllable computing power is provided for an embodiment of the present application.

[0020] Figure 2 A multi-modal large model training precision alignment and optimization system architecture diagram is provided for the autonomous controllable computing power of the present application.

[0021] Figure 3 An electronic device schematic diagram is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0023] It should be noted that at least one in the embodiments of the present application refers to one or more, and multiple refers to two or more. Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application.

[0024] Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0025] Embodiment one

[0026] Figure 1 A flowchart of a multi-modal large model training precision alignment and optimization method for autonomous controllable computing power is provided for an embodiment of the present application. As shown in Figure 1 A multi-modal large model training precision alignment and optimization method for autonomous controllable computing power, comprising: Step S100, a dual parallel scheduling framework of a heterogeneous computing environment is established, concurrent execution of forward derivation and weight adjustment tasks of a model is realized at a job level, and sharing of computing resources between an accelerator and a host processor is realized at a system level. The dual parallel scheduling framework of the heterogeneous computing environment is established to realize efficient cooperation and utilization of computing resources. Specifically, task-level concurrency: by parallelizing the two core tasks of forward derivation (executed by an accelerator such as a GPU) and weight adjustment (executed by a host processor such as a CPU) in the neural network level dimension, the traditional serial execution mode is broken, and the single training iteration time is shortened. Operation resource sharing: dynamically coordinating and sharing the high parallel computing capability of the accelerator and the general logic processing capability of the host processor, solving the problem of coexistence of resource idling and bottleneck in the heterogeneous platform, and improving the overall training efficiency from the system scheduling level.

[0027] Specifically, in the embodiment, the dual parallel scheduling framework of the heterogeneous computing environment includes: Coarse-grained job parallel scheduling, a static task allocation strategy is adopted, forward derivation tasks are allocated to the accelerator for execution, weight adjustment tasks are allocated to the host processor for execution, and a pipeline parallel is formed in the neural network level dimension; task-level static parallelization is realized, and a training pipeline is constructed to hide CPU processing delay. Static task allocation is a predefined and relatively fixed allocation strategy during training. The computationally intensive forward derivation (including backpropagation) is allocated to the accelerator (such as a GPU) for execution, because the GPU is good at processing large-scale parallel computing. At the same time, the weight adjustment (i.e., parameter update, such as using the optimizer algorithm SGD) with logical control complexity but relatively small amount of calculation is allocated to the host processor (such as a CPU) for execution.

[0028] And pipeline concurrency is the key of this step. It is not to start parameter update of all layers after backpropagation of all layers is completed. As shown in the following figure, when the GPU completes backpropagation of the Lth layer, the (L+1)th layer calculation can be started immediately, and at the same time, the CPU can take over the gradient of the Lth layer and start to execute parameter update. This pipeline formed in the neural network level dimension makes the CPU and the GPU busy at the same time, effectively reduces the waiting time between processors, and improves the overall throughput.

[0029] Fine-grained job parallel scheduling, based on runtime monitoring of accelerator idle periods, dynamically divides weights to adjust tasks through an adaptive load balancing method, allowing the accelerator to perform part of the weight adjustment subtasks during waiting for the host processor; on the basis of coarse-grained parallelism, dynamic and more fine-grained load balancing is achieved to further squeeze the idle computing power of the GPU. Through monitoring and identification, the system continuously monitors the state of the GPU in real time. When the GPU completes the calculation of the current layer, but needs to wait for the CPU to complete the parameter update of the previous layer before continuing, an idle period will be generated. Dynamic task division is set through an adaptive load balancing method (for example, a prediction function based on historical waiting time and the current batch size B: ), dynamically divides a complete parameter update task (for example, updating 1 million parameters) into two parts. For example, the GPU is responsible for updating 300,000 parameters (subtasks), and the CPU is responsible for updating the remaining 700,000 parameters. Finally, through resource reuse, the computing power that would otherwise be wasted during waiting is used to perform part of the parameter update subtasks. This not only reduces the total waiting time, but also reduces the amount of gradient data that needs to be transferred from the GPU to the CPU (because part of the update has been completed inside the GPU), thereby reducing the amount of data migration.

[0030] Job dependency relationship coordination component, maintaining the timing constraint relationship between forward derivation and weight adjustment, ensuring the consistency of training semantics. Ensures the correctness of the training logic and final results of the model in the complex parallel execution process. Among them, the timing constraint is maintained, and the deep learning training has strict dependency relationship. For example, the parameter update of the Lth layer must be performed after the gradient of the Lth layer is calculated; while the back propagation of the L+1th layer depends on the updated parameters (in some architectures) or at least the activation values of the Lth layer. This component acts like a traffic police, maintaining the order of these forward derivation and weight adjustment. At the same time, it ensures semantic consistency. Its existence guarantees that although the calculation process is highly parallelized and disordered, the convergence behavior of the final model is consistent with that under serial execution. That is, for the same input data, the weight update result of the model trained by this parallel scheduling is mathematically equivalent (or within a certain error tolerance) to the model trained by executing the training step by step in sequence, thereby ensuring the uniformity of training semantic logic and the stability of model optimization.

[0031] The above three sub-steps constitute a complete parallel scheduling scheme from macro (static pipeline) to micro (dynamic load division), and throughout the consistency guarantee mechanism (dependency coordination). It maximizes resource utilization through multi-level, dynamic and static combined parallel strategy, while ensuring the correctness and stability of the training process.

[0032] Specifically, in the present embodiment, the adaptive load balancing method comprises the following steps: Step 3.1: Monitoring and data collection, real-time perception of system state, providing data basis for dynamic decision. Monitor the idle period generated by the accelerator in the waiting process of the main processor weight adjustment, and record the average waiting time in the training period. The idle period is specifically: the system continuously tracks the time period that the GPU (accelerator) must wait for the CPU (main processor) to complete the parameter update of the previous layer after completing the back propagation of one layer. This time is a direct manifestation of load imbalance. At this time, the average waiting time is recorded. In order to eliminate the fluctuation of single measurement, the system will calculate the average waiting time in a sliding window. For example, record the average waiting time of the last N training iterations and take the average: This average value can more stably reflect the load balancing status of the system.

[0033] Step 3.2: Dynamic prediction and decision, intelligently predict the optimal task division strategy based on historical data and current context. Based on historical waiting time data and current batch size, calculate the task allocation ratio through a prediction model, and dynamically adjust the weight adjustment load between the accelerator and the main processor.

[0034] The input features are: the decision model considers two key inputs: historical waiting time (reflecting the current load pressure) and current batch size B (the larger the batch, the more likely it is to affect ). Prediction model calculation: calculate the proportion of parameter update tasks that the GPU should undertake through a prediction model (for example, a simple linear regression model or a small neural network). For example, where is the weight coefficient set by the model through learning or experience. The larger the value is, the more serious the CPU bottleneck is, and the proportion of tasks allocated to the GPU should be increased. is used to consider the impact of batch size, and the logarithm is used to prevent its impact from being too large. The operation ensures that the proportion of tasks allocated to the GPU will not exceed a safe upper limit (for example, 70%), preventing the GPU from affecting its core back propagation task due to excessive participation in parameter update.

[0035] Step 3.3: Adaptive adjustment mechanism, set safety valve and trigger conditions, achieve robust automatic adjustment. Establish task partitioning boundary adaptive adjustment mechanism, automatically increase accelerator side weight adjustment task proportion when accelerator latency exceeds preset threshold. Set critical threshold: preset an acceptable maximum latency threshold . For example, if the average GPU waiting time exceeds 20 ms, the system efficiency is considered too low. When monitoring > , the control system will trigger adjustment and automatically increase the value of . The adjustment is not unlimited, and the goal is to let the GPU utilization rate reach a limit usage rate (e.g. 95%) instead of 100%, leaving a margin for system fluctuations to prevent overload.

[0036] Step 3.4: Task execution and synchronization, ensure that the tasks dynamically divided can be executed correctly and consistently. Through the task scheduler, fine-grained task allocation and synchronization are realized, ensuring that the weight adjustment tasks executed by the accelerator and the main processor remain consistent when merged. Specifically, through task distribution, the task scheduler (job dispatcher) updates the parameters according to the calculated , and accurately divides the parameter update task (e.g. updating a weight matrix W) into two subtasks. For example, W has 1000 rows, =0.3, then GPU is responsible for updating rows 1-300, and CPU is responsible for updating rows 301-1000. Parallel execution and synchronization: GPU and CPU handle the part of the parameters they are responsible for in parallel. Parameter merging and consistency guarantee: this is a key step. After both are completed, the scheduler needs to safely integrate the separately updated parameter fragments ( and ) back into the complete, updated weight matrix . This process must guarantee: atomicity: the merging operation cannot be interrupted. Order: ensure that the complete is ready before the subsequent forward derivation begins. Only in this way can the parameter uniformity, i.e. the consistency of the model state, be guaranteed.

[0037] The four steps constitute a complete monitoring-decision-adjustment-execution automation control loop. Through real-time feedback and dynamic prediction, it optimizes the coarse-grained static parallel scheduling into an intelligent, fine-grained dynamic load balancing system that responds to the system runtime state, thereby maximizing the utilization efficiency of heterogeneous computing resources while ensuring the correctness of training.

[0038] Step S110, a classification processing scheme based on gradient characteristics is formulated, and inter-accelerator ring global aggregation and main processor memory unified processing are adopted for operation-intensive and storage-intensive gradients respectively. The classification processing scheme based on gradient characteristics is formulated to optimize gradient communication and aggregation, and relieve bandwidth and video memory pressure. According to the characteristics (computational complexity, memory occupation) of the gradient in the calculation process, the gradient is intelligently identified as operation-intensive and storage-intensive, which provides the basis for differentiated processing. Among them, the differentiated processing includes: for the operation-intensive gradient, the high-speed interconnection between the accelerators is used, and the ring global aggregation mode is adopted to realize efficient parallel synchronization and reduce communication delay. For the storage-intensive gradient, it is unloaded to the main processor memory for unified processing, which significantly reduces the burden of accelerator video memory and bus communication pressure. This step solves the key bottleneck in data parallel training from the communication optimization level.

[0039] Specifically, in the present embodiment, the classification processing scheme based on gradient characteristics includes the following steps: Step 4.1: Gradient characteristic analysis and classification, intelligently identifying and distinguishing gradient types to provide the basis for subsequent differentiated processing. A gradient characteristic analysis module is established to monitor the computational complexity and memory occupation characteristics in the gradient calculation process, and to identify operation-intensive gradients and storage-intensive gradients.

[0040] Specifically, the gradient characteristic analysis module monitors two key indicators in the gradient calculation process: computational complexity: the number of floating point operations (FLOPs) required to generate the gradient. For example, the gradient calculation of a large convolutional layer is usually more complex than that of a small fully connected layer. Memory occupation characteristics: the size of the gradient tensor itself and the temporary storage space required during calculation. Based on the preset threshold or classification model, the gradient is automatically labeled as two categories, completing automatic identification. One is operation-intensive gradient: the characteristic is that the computational overhead is much larger than the communication overhead. For example, the gradient of a 128x128x256 convolution kernel has a large amount of calculation, but the amount of gradient data that needs to be synchronized is relatively controllable. The other is storage-intensive gradient: the characteristic is that the parameter amount is huge, but the overhead of calculating the gradient itself is relatively small. For example, the weight gradient of a fully connected layer with an input dimension of 4096 and an output dimension of 8192 has a tensor shape of [4096, 8192], and the data amount is as high as 33M elements, which requires a large communication bandwidth to transfer it.

[0041] Step 4.2: Operation-intensive gradient processing, efficient and low-delay synchronization of computation-intensive gradients. For operation-intensive gradients, inter-accelerator ring global aggregation processing is adopted to realize parallel collection and global synchronization of gradient data through ring communication topology.

[0042] A ring-based global aggregation is employed between accelerators (e.g., between GPUs). The ring communication topology connects multiple GPUs into a logical ring. Each GPU performs two tasks simultaneously: sending data to upstream GPUs and receiving data from downstream GPUs. Workflow: Assuming there are 4 GPUs (… Step 1: Each GPU calculates its own gradient. Step Two: Will Send to Simultaneously receive from of G1 will Send to Simultaneously receive from of And so on. Third step: Each GPU adds the received gradient to its local gradient. Now, have + , have + And so on. After (N-1) steps, each GPU has the sum of all gradients. This method distributes the communication load evenly across all links, avoiding the bandwidth bottleneck on the server side in traditional parameter servers, and is particularly suitable for parallel collection and globally synchronized computation-intensive gradients.

[0043] Step 4.3: Storage-intensive gradient processing, offloading large gradients to alleviate pressure on accelerator memory and interconnect bandwidth. For storage-intensive gradients, unified processing is performed in the main processor's memory, unloading the gradient data to the main processor's memory pool for unified reduction calculation. Processing mode: Unified processing in the main processor (CPU) memory. Workflow: Unloading: All GPUs send the storage-intensive gradients they identify (e.g., the gradient of the 33M-element fully connected layer) to the CPU's memory, forming a memory pool. Unified reduction: The CPU directly performs a summation and averaging operation (i.e., reduction calculation) on these large gradients sent from all GPUs in its memory. Broadcast: The CPU will display the calculated average gradient. Then it broadcasts back to all GPUs. While this increases data transfer between the CPU and GPUs, it completely avoids transferring massive amounts of data between GPUs, thus reducing communication overhead between accelerators. The CPU's powerful memory management capabilities and large memory capacity are well-suited for handling this type of task.

[0044] Step 4.4: Dynamic classification strategy, allowing classification criteria to adapt to changes in the training process, maintaining classification accuracy. Construct a dynamic gradient classification strategy that adaptively adjusts the gradient classification threshold based on changes in network layer features. During the training process, the gradient distribution and importance of different network layers may change. The system will adaptively adjust the threshold (e.g., computational complexity threshold, memory occupancy threshold) used to distinguish between two types of gradients based on the training progress and observed changes in network layer features. This ensures that classification can always adapt to the current training state, optimizing recognition accuracy.

[0045] Step 4.5: Process flow coordination, ensuring seamless collaboration between two different processing modes without introducing conflicts or delays. Coordinate the two types of gradient processing flows through the gradient processing scheduler to ensure effective timing connection between ring-shaped global aggregation and memory unified processing. The gradient processing scheduler acts as the overall commander. It is responsible for coordinating the two parallel processes of ring-shaped global aggregation and memory unified processing. It needs to ensure that the two processes are correctly connected in time, for example, waiting for both types of processing to complete before proceeding to the next step of parameter update. There will be no conflicts due to resource competition (such as PCIe bandwidth). Through this coordination, the stability of the training process is maintained, preventing model convergence problems caused by asynchronous gradient processing.

[0046] The above five steps collectively build a refined gradient processing pipeline. It classifies gradients into two optimal processing channels through intelligent classification (Step 4.1): low-latency channel (Step 4.2, ring aggregation) to handle computational bottlenecks, and high-bandwidth channel (Step 4.3, CPU unified processing) to handle storage / communication bottlenecks. And through dynamic adjustment (Step 4.4) and central scheduling (Step 4.5) to ensure efficient and stable operation of the entire system, thereby improving the efficiency and scalability of large-scale data parallel training as a whole.

[0047] Specifically, in the present embodiment, the gradient feature analysis module realizes multi-dimensional feature extraction and dynamic classification in the following ways: Step 5.1: Constructing a gradient feature vector space, quantizing gradient features into structured data to provide input for intelligent classification. Construct a gradient feature vector space, form a feature vector based on the calculation delay, memory occupancy, and communication bandwidth requirements of the gradient tensor. Specifically, perform multi-dimensional feature extraction, no longer relying on a single or two features for judgment, but real-time collection of three-dimensional key indicators to form a feature vector V: Calculation delay (L): the time (milliseconds) taken to calculate the gradient. Memory occupancy (M): the ratio (percentage) of the amount of video memory occupied by the gradient tensor to the total video memory of the GPU where it is located. Communication bandwidth requirement (B): according to the data volume of the gradient tensor and the synchronization period set by the model, estimate the required communication bandwidth (MB / s). Combine these three indicators into a feature vector V = [L, M, B]. This vector comprehensively represents the calculation, storage, and communication attributes of a gradient tensor.

[0048] Step 5.2: Deep neural network feature classification, using a nonlinear model to achieve high-precision, probability-based gradient type identification. Use a deep neural network feature classifier to output a gradient type probability distribution based on the feature vector. Use a pre-trained deep neural network (DNN) as the classifier. This DNN can be a simple multi-layer perceptron (MLP). Input the feature vector V obtained in step 5.1 into this DNN. The output layer of the DNN (usually using the Softmax activation function) will output a probability distribution . For example: P = [0.85, 0.15] indicates that the gradient has an 85% probability of being computation-intensive and a 15% probability of being storage-intensive. Compared to using fixed thresholds, DNNs can learn complex, nonlinear relationships between features, enabling more accurate identification.

[0049] Step 5.3: Dynamic threshold adjustment mechanism, allowing classification criteria to evolve dynamically with training, improving the adaptability of classification strategies. Establish a dynamic threshold adjustment mechanism to automatically update classification decision boundaries based on training phase and model structure changes. In the early stages of training, gradient magnitudes can be large and unstable; in the later stages of training, gradients become small and stable. The gradient features of different model structures (such as CNN and Transformer) also differ greatly. This mechanism automatically updates the decision boundaries of classification based on the training phase (such as the current epoch number) and model structure changes (such as different types of layers). Early training (lenient strategy): To ensure stability, a lower classification confidence threshold may be set. For example, as long as it is classified as computation-intensive, avoiding overly arbitrary classification when the gradient is unstable. Late training (tight strategy): To achieve optimal performance, increase the threshold. For example, require to be classified as computation-intensive, making classification more accurate.

[0050] Step 5.4: Feedback Optimization Loop, Self-Optimization through Continuous Learning. Introduce a feedback optimization loop to continuously refine the feature extraction weights and classifier parameters. Compare the actual processing efficiency (e.g., actual time consumption) with the expected performance indicators (e.g., theoretical optimal time consumption) after gradient classification. Use the performance difference as a feedback signal to continuously adjust the parameters of the two key components: Feature extraction weights: Adjust the weights of different dimensions or preprocessing methods when calculating the feature vector V. Classifier parameters: Fine-tune the weights within the DNN classifier to make better classification decisions. Form a closed loop to improve the accuracy of gradient classification and thus continuously improve the efficiency of the entire training system.

[0051] Step 5.5: Abnormal Gradient Processing Channel, Robustness Guarantee, Handle difficult cases that the classification system cannot reliably identify. Configure an abnormal gradient processing channel to start a re-computation, review, or manual intervention process for feature abnormal gradients. Identify abnormalities: When the feature vector V deviates significantly from the normal distribution, or when the DNN output has two low probability values (e.g., P = [0.51, 0.49]), it is difficult to make a high-confidence classification. Set up special processing procedures, including: Gradient Recalculation: Trigger the recalculation of this gradient to eliminate transient calculation errors. Feature Review: Start a more complex and time-consuming backup feature analysis process for secondary judgment. Manual intervention interface: In extreme cases, record such gradients and alarm for engineers to analyze later, thereby optimizing the system. Ensure that the system can safely and stably run even when encountering unexpected situations, ensuring the robustness of the classification system.

[0052] These five steps collectively upgrade a simple if-else classifier to an intelligent adaptive system with perception (Step 5.1), thinking (Step 5.2), growth (Steps 5.3 and 5.4), and fault tolerance (Step 5.5) capabilities. By introducing machine learning models and feedback control, it significantly improves the intelligence level of gradient classification and its reliability in complex real-world training environments.

[0053] Step S120, model simplification and reconstruction of multi-modal recognition is carried out, a combination convolution unit is configured in the visual path, and a multi-attention mechanism is fused. On the premise of ensuring accuracy, the model complexity and calculation amount are reduced, and the foundation is laid for end-side deployment. Among them, the structure lightweight is: in the visual path responsible for image processing, a combination convolution unit is used to replace the standard convolution, which fundamentally reduces the parameter amount and floating point operation number. The feature preservation and enhancement is: the multi-attention mechanism is introduced, so that the model can still focus on the key information across modalities after simplification, the feature extraction and alignment ability is strengthened, and the accuracy loss caused by lightweight is compensated. This step innovates from the model architecture level, and realizes the balance between accuracy and efficiency.

[0054] Specifically, in the present embodiment, the model simplification and reconstruction of multi-modal recognition includes the following steps: Step 6.1: Constructing a heterogeneous convolution architecture for the visual path, using a more efficient calculation unit to replace the standard convolution, which fundamentally reduces the calculation burden. The heterogeneous convolution architecture of the visual path is constructed, and the standard convolution, the depth separable convolution and the channel shuffle operation are fused to form a combination convolution unit.

[0055] Specifically, the architecture core is: a combination convolution unit is created, which is not a single type of convolution, but a deep fusion of three operations: standard convolution: retaining strong feature expression ability. Depth separable convolution: greatly reducing the parameter amount and calculation amount. The ratio of its calculation amount to that of traditional convolution is about , where N is the output channel number and K is the convolution kernel size. Channel shuffle operation: solving the problem of insufficient information flow between channels in depth separable convolution, promoting information interaction between feature channels. Through this mixed design, the calculation complexity is significantly reduced while maintaining the feature expression ability.

[0056] Step 6.2: Designing a cross-modal attention fusion mechanism to enhance the model's ability to focus on key cross-modal information on the lightweight skeleton. Design a cross-modal attention fusion mechanism, integrate a dual attention module of channel attention and spatial attention at the output end of the combination convolution unit. The core of the mechanism is: at the output end of the combination convolution unit, a dual attention module is integrated, which processes the attention of two dimensions in parallel: channel attention: learning the importance weight of each feature channel. For example, using the Squeeze-and-Excitation structure in SENet, a weight vector is generated, and the feature map U is weighted: This realizes global context modeling and enhances the recognition of discriminative features. Spatial attention: learning the importance weight of each spatial position of the feature map. A weight matrix is generated, and the feature map is weighted: This improves the location awareness ability, making the model pay more attention to the area where the target is located. By simultaneously recalibrating features from the channel and spatial dimensions, the model can more accurately focus on the visual area related to the text modality, strengthening cross-modal feature alignment.

[0057] Step 6.3: Establish a multi-scale feature preservation network to dynamically optimize the fusion of features at different levels, solving the problem of multi-scale target detection and feature alignment. A multi-scale feature preservation network is established, with adaptive feature selection gates deployed at different levels to dynamically adjust the feature fusion ratio according to the input modality characteristics. Specifically, the network structure is as follows: adaptive feature selection gates are deployed at different levels (such as shallow, middle, and deep) in the visual path. Dynamic adjustment: this gating mechanism dynamically adjusts the fusion ratio of features transmitted from the shallow layer (rich in details and location information) and the deep layer (rich in semantics and class information) according to the input modality characteristics (such as image texture complexity and correlation with text description). For example, for small targets that require precise positioning, the gate may increase the weight of shallow detail features (γ biased towards shallow layer). For complex scene understanding, it may increase the weight of deep semantic features (γ biased towards deep layer). This adaptive multi-scale fusion ensures the accuracy of multi-modal feature alignment, allowing the model to maintain good performance in various complex scenarios.

[0058] Step 6.4: Implement a progressive model compression strategy to reduce the model while maintaining its performance through a phased, easy-to-difficult compression process. Implement a progressive model compression strategy to maintain model performance through combined convolution unit deployment, attention distillation, and knowledge distillation. Specifically, the three-stage process: Stage One (Structural Simplification): Deploy combined convolution units in the visual path to complete the initial simplification of the model skeleton. Stage Two (Knowledge Transfer): Through attention distillation, the complex attention map (i.e., attention pattern) in a large and accurate teacher model (Teacher Model) is used as a supervisory signal to force the simplified student model (Student Model) to imitate learning, thereby transferring the powerful feature focusing ability of the teacher model to the simplified model. Stage Three (Performance Alignment): Further adopt knowledge distillation to make the student model not only learn the final output (logits) of the teacher model, but also learn the intermediate features, thereby maintaining the model's performance on multi-modal tasks. This progressive strategy can more effectively preserve the knowledge of the original model than one-time compression, reducing precision loss.

[0059] Step 6.5: Configure a dynamic inference optimization mechanism to enable the trained lightweight model to adapt to different deployment environments. Configure the dynamic inference optimization mechanism to automatically adjust the number of combined convolution unit branches and the calculation precision of the attention module according to the resource status of the deployment environment. Specifically, the core of the mechanism is that the model is no longer static during inference, but can adaptively adjust according to the computing resource status of the deployment environment (such as chip computing power, available memory, power consumption budget). The adjustment content includes: the number of combined convolution unit branches: in resource-constrained situations, some branches in the combined convolution unit are closed or simplified to further reduce the computational load. The calculation precision of the attention module: for example, the calculation of the attention weight is reduced from 32-bit floating-point number to 16-bit, or even 8-bit integer. The performance optimization adaptation on different self-controllable chip platforms is achieved, providing great flexibility for efficient deployment on various edge devices.

[0060] These five steps constitute a complete technical chain from the innovation of bottom-level computing units (6.1), to the enhancement of internal feature processing mechanisms (6.2, 6.3), to the optimization of macro training strategies (6.4), and finally to the flexibility guarantee of deployment (6.5). It systematically solves the contradiction between model lightweight and multi-modal performance maintenance, ensuring that the simplified model is still powerful and efficient on self-controllable chips.

[0061] Specifically, in this embodiment, the heterogeneous convolution architecture for building the visual pathway adopts a phased feature reorganization strategy, including the following steps: Step 7.1: Design a reconfigurable convolution kernel group to realize dynamic structuring of the convolution kernel, enabling it to adaptively extract multi-scale features. Design a reconfigurable convolution kernel group to decompose a standard KxK convolution kernel (such as 3x3) into multiple sub-kernels (for example, a set of 1x1 kernel, 3x3 kernel, and 5x5 kernel). Each sub-kernel is responsible for capturing different scale feature patterns (such as details, local structure, and global context). Dynamic weight allocation: according to the characteristics of the input feature map, a lightweight gating network is used to dynamically calculate the fusion weights of each sub-kernel . The final convolution result , where This means that for image regions with complex texture, the model may automatically assign higher weights to the 1x1 sub-kernel for capturing details; for regions that require a larger receptive field, the 5x5 sub-kernel may dominate. This realizes adaptive adjustment of the sub-kernel combination method according to the input feature characteristics, replacing the fixed and unchanged convolution kernel, and enhances the feature expression ability of the model.

[0062] Step 7.2: Establish a channel interaction enhancement module to break the channel isolation of deep separable convolution and promote cross-channel information flow while ensuring efficiency. Establish a channel interaction enhancement module to introduce a cross-channel information exchange mechanism based on deep separable convolution. Feature fusion is achieved through the alternating use of grouped convolution and channel shuffling. On the basis of deep separable convolution (DWConv), which is efficient but has no interaction between channels, a cross-channel information exchange mechanism is introduced. The strategy of alternating use of grouped convolution and channel shuffling is adopted. For example, feature map -> grouped convolution (preliminary fusion of group information) -> channel shuffling (disrupt and reorganize the information of each group) -> grouped convolution (continue to fuse in the new group) ->... This alternating strategy is like an efficient information mixer that achieves sufficient fusion of feature information while controlling the growth of computational complexity (grouped convolution has small computational complexity), making up for the inherent shortcomings of DWConv.

[0063] Step 7.3: Implement a multi-level feature distillation process to refine features at the output of the multi-branch output of the heterogeneous convolution architecture and retain the most important information. Implement a multi-level feature distillation process by setting feature selection gates at the output of each branch of the combined convolution unit. Based on channel attention weights, the importance of feature maps is sorted. At the output of each convolution branch of the combined convolution unit, a feature selection gate is set. This gate sorts the importance of each channel of the feature map based on channel attention weights (such as weights generated by the SE module). Then, through a threshold mechanism or Top-K selection, key features are retained and redundant information is suppressed. Assuming that a branch outputs 256-channel features, the gate may only allow the top 128 channels with the highest importance weights to pass, while the features of the remaining channels are significantly weakened or set to zero. This achieves multi-level feature distillation, actively filtering out noise and redundancy in the early stages of feature propagation, allowing subsequent network layers to focus on more valuable information, improving computational efficiency and feature quality.

[0064] Step 7.4: Build an adaptive computing path to dynamically activate different computing sub-networks according to the complexity of the input instance, achieving on-demand computing. Build an adaptive computing path to dynamically activate or sleep part of the convolution branches according to the complexity of the input data. According to the complexity of the input data (which can be quickly evaluated by the feature amplitude, entropy of the initial layer, or a lightweight discriminator), the system automatically decides whether to activate or sleep part of the convolution branches in the combined convolution unit. There are two modes: one is simple scenarios (such as a single object in a solid color background): a lightweight computing mode is adopted, which may only activate the basic 1x1 convolution and DWConv branches. The other is complex scenarios (such as dense crowds, complex backgrounds): full computing capacity is enabled, all branches are activated, including the more computationally intensive reconfigurable convolution kernel groups. This achieves a balance between computational efficiency and model performance, significantly improving the average inference speed overall, especially suitable for edge devices with fluctuating computing resources.

[0065] Step 7.5: Configure the cross-modal feature alignment component to actively guide and correct the extraction process of visual features using semantic information of the text modality. Configure the cross-modal feature alignment component and introduce a text-guided attention mechanism at the key nodes of the visual path to guide the extraction and fusion of visual features using semantic information of the language modality. By introducing text guidance, a text-guided attention mechanism is introduced at the key nodes of the visual path (such as the end of Backbone or the Neck module). The working mechanism is as follows: the text modality (such as the descriptive sentence of the image) is converted into a semantic vector T through an NLP encoder. Use T as Query, visual feature map V as Key and Value, and perform Cross-Attention calculation: This process allows the semantics of the text to filter the visual features: the most relevant descriptions are selected and the visual features are weighted according to the answers. By using the semantic information of the language modality to guide the extraction and fusion of visual features, the consistency of the multi-modal representation space is actively ensured at the feature level, greatly improving the accuracy of tasks such as image-text matching.

[0066] These five steps upgrade the above heterogeneous convolutional architecture from a relatively static structure to a dynamic reconfigurable (7.1), information-rich interaction (7.2), feature actively refined (7.3), on-demand computing allocation (7.4), and cross-modal semantic guided (7.5) intelligent, adaptive feature extraction system. It optimizes the visual feature processing flow from the micro convolution kernel to the macro computing path, providing core technical support for efficient and accurate operation of multi-modal large models in resource-constrained environments.

[0067] Step S130, create a hardware-aware precision compression conversion method, select a compression strategy according to the target chip operation performance parameters and complete platform compatibility with the help of the device abstraction layer. Realize efficient and lossless migration of the trained model to the specific chip. Adaptive compression does not mean using a single quantization strategy, but intelligently selecting the most suitable quantization scheme (symmetric / asymmetric) according to the hardware characteristics (computing power architecture, memory bandwidth, etc.) of the target domestic chip. Platform compatibility is achieved by decoupling the model from the underlying hardware through the device abstraction layer, encapsulating hardware differences, so that the same trained model can be seamlessly deployed on different domestic AI chips without modifying the code. This step solves the last hurdle of model landing application, i.e. cross-platform adaptation.

[0068] Specifically, in the present embodiment, the hardware-aware precision compression conversion method comprises the following steps: Step 8.1: Construct a chip characteristic-aware quantization strategy selection mechanism to automatically select the optimal quantization scheme based on the specific characteristics of the target hardware, achieving a tailor-made approach. Construct a chip characteristic-aware quantization strategy selection mechanism to automatically select symmetric quantization or asymmetric quantization schemes by analyzing the computing power architecture, memory bandwidth, and instruction set characteristics of the target domestic smart chip. Through hardware analysis, analyze the key characteristics of the target domestic smart chip, including: computing power architecture: whether it has a high-efficiency integer calculation unit (such as INT8 DSP), support for asymmetric quantization, etc. Memory bandwidth: whether memory bandwidth is a bottleneck or abundant. Instruction set characteristics: whether it supports specific quantization operation instructions. Through a decision model, based on the above analysis, establish a decision model (such as a rule-based engine or a simple classifier). For example: if the chip's INT8 calculation unit is powerful and the memory bandwidth is limited, prefer symmetric quantization. Because symmetric quantization (zero point O=0) is simpler to calculate, it can maximize the use of INT8 computing power and reduce data transfer. Formula: Q=round(V / S). If the chip supports asymmetric quantization and the data distribution is far from the zero point (such as the output after ReLU activation), select asymmetric quantization for possibly higher precision. Formula: Q=round(V / S+O). Realize automatic selection of symmetric quantization or asymmetric quantization scheme to ensure that the quantized model achieves the best balance between precision and speed on the target hardware.

[0069] Step 8.2: Design a hierarchical precision allocation scheme to implement differentiated quantization for different parts of the model, balancing between overall compression rate and key accuracy. Design a hierarchical precision allocation scheme to implement differentiated precision configuration according to the sensitivity differences of different layers in the network to quantization error. Sensitivity analysis is used to evaluate the sensitivity differences of different layers in the network to quantization error. Generally, input / output layers, and layers with small-size convolution kernels are more sensitive to quantization. Differentiated configuration includes: Feature extraction layers (such as most convolution layers): These layers focus on feature transformation and are somewhat robust to slight errors, so they can use aggressive 8-bit integer quantization to significantly reduce computation and storage. Classification / prediction layers (such as the last fully connected layer or detection head): The output of these layers directly affects the final result, so they require high precision and therefore retain 16-bit floating-point precision to minimize precision loss. This coarse and fine grain combination scheme achieves an optimal balance between model size and inference accuracy, achieving better final performance than global uniform quantization.

[0070] Step 8.3: Develop a unified interface specification for the device abstraction layer to decouple the quantized model from the underlying hardware and achieve one-time quantization and multi-deployment. Develop a unified interface specification for the device abstraction layer to define standardized model conversion, memory management, and computation scheduling interfaces. Define standardized interfaces: Develop a device abstraction layer that defines standard operation interfaces independent of hardware, such as: : Load the quantized model. Allocate memory. : Perform inference computation. Chip vendors need to implement the details of these interfaces for their specific hardware according to this specification for platform compatibility. In this way, the same quantized model can be deployed and run on different domestic chip platforms without modifying the code by calling these standard interfaces.

[0071] Step 8.4: Implement dynamic precision adjustment function to dynamically adjust the calculation precision according to the actual environment during inference runtime, and realize intelligent energy efficiency management. Implement dynamic precision adjustment function to automatically adjust the quantization bit number of activation value during inference phase according to real-time load and power consumption constraints of the device. Including state monitoring, real-time monitoring of device real-time load (such as CPU usage) and power consumption constraints (such as remaining power, cooling situation). And dynamic adjustment, high load / ample power: use higher precision (such as activation value also uses INT8), pursue the highest accuracy. Automatically switch to lower precision (such as reduce activation value from INT8 to INT4), although it may introduce a small precision loss, but can significantly reduce power consumption and calculation delay. Maximize the energy efficiency ratio under the premise of ensuring inference accuracy, especially suitable for power-sensitive edge devices.

[0072] Step 8.5: Establish a model fine-tuning mechanism for quantization awareness, let the model preview the low-precision environment before quantization, and thus reduce the acclimatization when formally deployed. Establish a model fine-tuning mechanism for quantization awareness, introduce quantization noise simulation and gradient redirection technology during compression conversion process. Specifically, quantization noise simulation is: in the last stage of model training (fine-tuning), introduce quantization noise simulation in forward propagation. That is, during calculation, simulate the process of quantization and dequantization: , (this is pseudo-quantized data). Use participate in subsequent calculations, so that the model weights adapt to this numerical precision loss. Gradient redirection technology is: during backpropagation, since the gradient of the round function is almost zero everywhere, techniques such as Straight-Through Estimator (STE) are needed to pass the gradient directly to the floating-point weights to ensure that the model can continue to learn. The formula is: (when the gradient is returned, ignore the round operation). Make the model adapt to the low-precision computing environment in advance before quantization, so that the weights find an optimal solution that performs well in floating-point training and has a small loss after quantization, thereby significantly reducing the precision loss after deployment.

[0073] The five steps constitute a complete, closed-loop model compression and deployment pipeline from strategy selection (8.1) -> structure optimization (8.2) -> interface abstraction (8.3) -> dynamic running (8.4) -> forward-looking training (8.5). It not only focuses on how to compress small, but also focuses on how to compress well (high precision) and use it (easy deployment, self-adaptation), ensuring that the optimized model can run efficiently and reliably on a variety of self-controlled chips.

[0074] Step S140, build an end-to-end integrated training configuration system, integrate parallel scheduling, gradient processing, simplified design and precision conversion functions, and provide complete service chain from data injection to terminal verification through standardized interface. The technical innovations of the first four steps are integrated into an automated and easy-to-use system engineering. The scattered parallel scheduling, gradient processing, model simplification, precision conversion and other technical modules are integrated into a unified platform. Through standardized interface, it provides complete service chain from data input, model training, optimization and compression to final testing and verification on terminal device, greatly reducing the use threshold and technical complexity of users. This step is a system-level innovation, ensuring that all technical components can work together to provide one-stop solution for users.

[0075] Specifically, in the present embodiment, the construction of the end-to-end integrated training configuration system includes the following steps: Build a modular service orchestration engine, encapsulate parallel scheduling, gradient processing, simplified design and precision conversion functions into independent micro-service units; realize the service and flexible assembly of technical components. Parallel scheduling, gradient processing, simplified design, precision conversion and other core functions are encapsulated into independent microservices. Through service grid, unified management and dynamic collaboration are realized, realizing plug-and-play and elastic scaling of functions.

[0076] Design a unified data bus architecture, establish a standardized data exchange format and transmission protocol; establish a standardized data flow pipeline. Define a unified data exchange format and transmission protocol to ensure seamless flow and real-time synchronization of multi-modal training data between distributed computing nodes, solving the problem of inconsistent data formats and low transmission efficiency.

[0077] Implement an intelligent workflow orchestration system, define the execution dependency relationship of training tasks based on a directed acyclic graph model; provide visual whole-process training task management. Based on a directed acyclic graph model, define the execution dependency relationship of each step in the training task. Through the visual interface, the user can configure the complete training process by dragging and dropping, and automatically generate the optimal execution plan, greatly reducing the user's use threshold.

[0078] An adaptive resource scheduler is developed to dynamically allocate heterogeneous computing resources according to the computing characteristics and resource requirements of training tasks; intelligent allocation and optimization of computing resources are realized. According to the computing characteristics and real-time resource requirements of training tasks, GPU, CPU and other heterogeneous computing resources are dynamically allocated. The resource utilization is continuously monitored during the training process, and automatic load balancing adjustment is performed to ensure efficient use of resources.

[0079] A full-link monitoring and diagnosis platform is established to collect performance indicators and accuracy data at each stage of training through point embedding. System-level observability and operation and maintenance support are provided. Through point embedding, performance indicators and accuracy data at each stage of training are collected to provide real-time visual monitoring, automatic alarm for abnormalities and root cause analysis functions. When the system performance decreases or training abnormalities occur, the problem source can be quickly located.

[0080] The five steps constitute a complete and closed-loop technical system from bottom-layer computing scheduling (step 100) -> mid-term training communication optimization (step 110) -> model structure optimization (step 120) -> late-stage deployment adaptation (step 130) -> final system integration (step 140). They work together to finally achieve efficient training of multi-modal large models on an autonomous and controllable computing power platform and ensure the overall goal of precision alignment and optimization.

[0081] Embodiment Two As shown in Figure 2 The present application provides a multi-modal large model training precision alignment and optimization system architecture diagram for autonomous and controllable computing power, which is applied to the multi-modal large model training precision alignment and optimization system for autonomous and controllable computing power as described in Embodiment One, and includes a dual parallel scheduling module 200, a gradient classification processing module 210, a model simplification and reconstruction module 220, a precision compression and conversion module 230, and a training configuration integration module 240.

[0082] The dual parallel scheduling module 200 is configured to establish a dual parallel scheduling framework of a heterogeneous computing environment, realize job-level concurrency of forward derivation and weight adjustment, and share operation resources between accelerators and main processors.

[0083] The gradient classification processing module 210 is configured to develop a classification processing scheme based on gradient characteristics, and separately use inter-accelerator ring global aggregation and main processor memory unified processing for operation-intensive and storage-intensive gradients.

[0084] The model simplification and reconstruction module 220 is configured to carry out model simplification and reconstruction for multi-modal recognition, configure combined convolution units in the visual path, and fuse multiple attention mechanisms.

[0085] The precision compression conversion module 230 is configured to create a hardware-aware precision compression conversion method, select a compression strategy according to a target chip operation performance parameter, and complete platform compatibility with the aid of a device abstraction layer.

[0086] The training configuration integration module 240 is configured to build an end-to-end integrated training configuration system, integrate parallel scheduling, gradient processing, simplified design, and precision conversion functions, and provide a complete service chain from data injection to terminal verification through a standardized interface.

[0087] Figure 3 An electronic device is provided in an embodiment of the present application. As shown in Figure 3 The electronic device at least includes the following parts: a processor 301 and a memory 300, a communication interface 303, and a bus 302.

[0088] In an embodiment of the present application, the memory 300 is used to store processor 301 executable instructions, and the processor 301 is configured to implement the method of the first aspect when executing the instructions.

[0089] In an embodiment of the present application, a computer readable storage medium includes instructions, and the instructions instruct the device to execute the method of the first aspect. For example, the instructions instruct the device to execute the method shown in the flow steps in Figure 1

[0090] The program working in the electronic device related to an embodiment of the present application can be a program (a program for making a computer function) for controlling a central processing unit (CPU) and the like to realize the functions of the above-mentioned embodiments related to one scheme of the present application. Then, the information processed by these systems is temporarily stored in a random access memory (RAM) when it is processed, and then stored in various ROMs such as a read-only memory (Flash ROM), a hard disk drive (HDD), and the like, and read out, corrected, and written by a CPU as needed.

[0091] It should be noted that a part of the electronic device of the above-mentioned embodiments can also be realized by a computer. In this case, a program for realizing the control function can be recorded in a computer readable recording medium, and realized by reading the program recorded in the recording medium into a computer and executing it.

[0092] It should be noted that the computer mentioned here refers to a computer built into the electronic device, and a computer including an OS, a peripheral device, and the like hardware. In addition, the computer readable recording medium refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and the like storage system such as a hard disk built into a computer.​

[0093] Moreover, the computer-readable recording medium can include a medium that dynamically stores a program for a short time, such as a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a medium that stores a program for a fixed time, such as a volatile memory inside a computer as a server or a client in this case. Furthermore, the above program can be a program for realizing a part of the above functions, and can also be a program that can realize the above functions by being combined with a program already recorded in a computer.

[0094] Furthermore, the electronic device in the above embodiments can also be realized as an aggregate (system group) constituted by a plurality of systems. Each system constituting the system group can have a part or all of each function or each functional block of the electronic device of the above embodiments. As the system group, all of each function or each functional block of the electronic device can be possessed.

[0095] Those skilled in the art will recognize that the above embodiments are merely illustrative of the application and should not be taken as limiting the scope of the application, as defined by the appended claims. Where technical features mentioned in any portion of the specification or claims are followed by notation such as means for (plus) or step for (plus), this is intended to have the same open-ended scope as the corresponding limit feature of the technical features set forth in the same claim portion. Accordingly, the terms means for and step for are intended to encompass the identical open-ended terms means for and step for.

Claims

1. A method for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power, characterized in that, Includes the following steps: A dual parallel scheduling framework is established for heterogeneous computing environments. At the job level, the concurrent execution of the model's forward derivation and weight adjustment tasks is realized, and at the system level, computing resources are shared between the accelerator and the main processor. A classification processing scheme based on gradient features was developed, and computationally intensive and storage-intensive gradients were processed by inter-accelerator circular global aggregation and unified processing in main processor memory, respectively. The model for multimodal recognition is simplified and reconstructed by configuring combined convolutional units in the visual pathway and integrating multiple attention mechanisms. A hardware-aware precision compression conversion method is created, which selects a compression strategy based on the target chip's computing performance parameters and achieves platform compatibility with the help of a device abstraction layer; Build an end-to-end integrated training configuration system that integrates parallel scheduling, gradient processing, simplified design and precision conversion functions, and provides a complete service chain from data injection to terminal verification through standardized interfaces.

2. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 1, characterized in that, The dual parallel scheduling framework for establishing a heterogeneous computing environment includes: Coarse-grained parallel scheduling employs a static task allocation strategy, assigning forward inference tasks to accelerators for execution and weight adjustment tasks to the main processor for execution, thus forming pipelined parallelism at the neural network layer level. Fine-grained parallel scheduling of jobs is based on the idle time of the accelerator monitored during runtime. The weight adjustment tasks are dynamically divided through an adaptive load balancing method, so that the accelerator can execute some weight adjustment sub-tasks while waiting for the main processor. The job dependency coordination component maintains the timing constraints between forward derivation and weight adjustment.

3. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 2, characterized in that, The adaptive load balancing method includes the following steps: Monitor the idle periods generated by the accelerator while waiting for the main processor weights to be adjusted, and record the average waiting time during the training cycle; Based on historical waiting time data and current batch size, the task allocation ratio is calculated through a predictive model, and the load is dynamically adjusted by weighting the accelerator and the main processor. Establish a task partitioning boundary adaptive adjustment mechanism. When the accelerator waiting time exceeds a preset threshold, automatically increase the weight on the accelerator side to adjust the task ratio. Fine-grained task allocation and synchronization are performed through a task scheduler to ensure that the weight adjustment tasks executed by the accelerator and the main processor maintain consistent parameters during merging.

4. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 1, characterized in that, The formulation of a gradient feature-based classification scheme includes the following steps: Establish a gradient feature analysis module to monitor the computational complexity and memory usage characteristics during gradient calculation, and identify computationally intensive gradients and storage-intensive gradients; For computationally intensive gradients, a circular global aggregation process is used between accelerators, and gradient data is collected in parallel and synchronized globally through a circular communication topology. For storage-intensive gradients, the gradient data is unloaded to the main processor memory pool for unified reduction calculation. Construct a dynamic gradient classification strategy that adaptively adjusts the gradient classification threshold based on changes in network layer features; The gradient processing scheduler coordinates the two types of gradient processing processes to ensure effective timing coordination between the circular global aggregation and the unified memory processing.

5. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 4, characterized in that, The gradient feature analysis module achieves multi-dimensional feature extraction and dynamic classification in the following ways: Construct a gradient feature vector space, and form feature vectors based on the computational latency, memory usage, and communication bandwidth requirements of the gradient tensor; A deep neural network feature classifier is used to output a gradient type probability distribution based on the feature vector. Establish a dynamic threshold adjustment mechanism to automatically update the classification decision boundary based on changes in the training phase and model structure. A feedback optimization loop is introduced to adjust the feature extraction weights and classifier parameters by comparing the difference between the actual processing efficiency and the expected performance indicators. Configure an abnormal gradient processing channel to initiate recalculation, verification, or manual intervention processes for gradient data with abnormal features.

6. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 1, characterized in that, The model simplification and reconstruction for multimodal recognition includes the following steps: A heterogeneous convolutional architecture for visual pathways is constructed, which integrates standard convolution, depthwise separable convolution, and channel shuffling operations to form combined convolutional units; Design a cross-modal attention fusion mechanism by integrating a dual attention module of channel attention and spatial attention at the output of the combined convolutional unit; A multi-scale feature preservation network is established, and adaptive feature selection gating is deployed at different levels to dynamically adjust the feature fusion ratio according to the characteristics of the input modality. A progressive model compression strategy is implemented, which maintains model performance by sequentially deploying combined convolutional units, attention distillation, and knowledge distillation. Configure a dynamic inference optimization mechanism to automatically adjust the number of branches in the combined convolutional unit and the computational accuracy of the attention module based on the deployment environment and resource conditions.

7. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 6, characterized in that, The heterogeneous convolutional architecture for constructing visual pathways employs a staged feature reorganization strategy, including the following steps: Design a reconfigurable convolutional kernel group, decompose the standard convolutional kernel into multiple sub-convolutional kernels, and adaptively adjust the combination of sub-convolutional kernels according to the characteristics of the input features; A channel interaction enhancement module is established, which introduces a cross-channel information exchange mechanism on the basis of depthwise separable convolution, and performs feature fusion by alternating the use of grouped convolution and channel shuffling; A multi-level feature distillation process is implemented, and feature filtering gates are set at the output of each branch of the combined convolutional unit. The feature maps are ranked by importance based on the channel attention weights. Construct an adaptive computation path that dynamically activates or suspends some convolutional branches based on the complexity of the input data. Configure a cross-modal feature alignment component, introduce a text-guided attention mechanism at key nodes of the visual pathway, and use language modality semantic information to guide visual feature extraction and fusion.

8. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 1, characterized in that, The method for creating hardware-aware precision compression conversion includes the following steps: Construct a chip characteristic-aware quantization strategy selection mechanism, and automatically select symmetric or asymmetric quantization schemes by analyzing the computing power architecture, memory bandwidth and instruction set characteristics of the target domestic smart chip. Design a layered precision allocation scheme, and implement differentiated precision configuration based on the differences in the sensitivity of network layers to quantization error; Develop a unified interface specification for the device abstraction layer, and define standardized interfaces for model conversion, memory management, and computation scheduling; The dynamic precision adjustment function is invoked to automatically adjust the quantization bit depth of the activation value during the inference phase based on the real-time load and power consumption constraints of the device. A model fine-tuning mechanism based on quantization perception is established, and quantization noise simulation and gradient redirection techniques are introduced during the compression and conversion process.

9. The method for aligning and optimizing the training accuracy of multimodal large models oriented towards autonomous and controllable computing power according to claim 1, characterized in that, The construction of the end-to-end integrated training configuration system includes the following steps: Build a modular service orchestration engine that encapsulates parallel scheduling, gradient processing, simplified design and precision conversion functions into independent microservice units; Design a unified data bus architecture and establish standardized data exchange formats and transmission protocols; The intelligent workflow orchestration system is invoked, and the execution dependencies of training tasks are defined based on a directed acyclic graph model. Develop an adaptive resource scheduler to dynamically allocate heterogeneous computing resources based on the computational characteristics and resource requirements of the training task; Establish a full-link monitoring and diagnostic platform, and collect performance indicators and accuracy data for each stage of training through data collection and monitoring.

10. A system for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power, applied to the method for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power as described in any one of claims 1 to 9, characterized in that, The system includes: The dual parallel scheduling module is configured to establish a dual parallel scheduling framework for heterogeneous computing environments. At the job level, it enables concurrent execution of forward inference of the model and weight adjustment tasks, and at the system level, it enables sharing of computing resources between the accelerator and the main processor. The gradient classification processing module is configured to formulate a classification processing scheme based on gradient features, and to use accelerator-based circular global aggregation and main processor memory-based unified processing for computationally intensive and storage-intensive gradients, respectively. The model simplification and reconstruction module is configured to perform model simplification and reconstruction for multimodal recognition, and combines convolutional units in the visual pathway and integrates multiple attention mechanisms. The precision compression conversion module is configured to create a hardware-aware precision compression conversion method, which selects a compression strategy based on the target chip's computing performance parameters and achieves platform compatibility with the help of the device abstraction layer. The training configuration integration module is configured to build an end-to-end integrated training configuration system, integrating parallel scheduling, gradient processing, simplified design and precision conversion functions, and providing a complete service chain from data injection to terminal verification through standardized interfaces.

Images