Multi-modal AI model training method based on distributed network architecture
Through distributed network architecture and advanced technical means, problems such as data synchronization, consistency and resource allocation in multimodal AI model training are solved, which significantly improves training efficiency and performance, and achieves efficient multimodal data fusion and fault tolerance. deal with.
Patent Information
- Application Number
- CN202411190744.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The prior art has shortcomings in data synchronization, data consistency, computing resource allocation, fault-tolerant processing and parallel computing when processing multimodal data, which affects the training efficiency and performance of multimodal AI models.
Adopting a distributed network architecture, data synchronization is achieved through the distributed file system HDFS and gRPC protocols, the improved Paxos consistency algorithm and the Kubernetes resource management framework for data consistency and resource allocation, and combining the Prometheus monitoring system and the TensorFlow Distributed framework for fault tolerance and parallelism training and use attention mechanisms to perform multimodal data fusion.
It significantly improves the training efficiency and performance of multimodal AI models, ensures data consistency and optimized utilization of computing resources, and enhances the reliability and stability of the system.
Smart Images

Figure CN118981503A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and distributed computing technology, and in particular to a multimodal AI model training method based on a distributed network architecture. Background Art
[0002] With the development of artificial intelligence technology, the processing and analysis of multimodal data has gradually become a research hotspot. Multimodal AI models need to simultaneously process heterogeneous data from different data sources, such as images, text, and audio, which places higher requirements on model training. However, existing technologies have many defects in processing multimodal data and model training, which are mainly manifested in the following aspects: 1. Data synchronization problem: Existing distributed systems focus more on the processing of single-modal data and lack effective solutions for the synchronization of multimodal data. Since multimodal data comes from different sources, it is necessary to keep data synchronized in a distributed environment to ensure the consistency and effectiveness of model training. However, in a multi-node environment, achieving real-time synchronization of multimodal data faces great challenges, and data delays and asynchrony often occur, affecting the model training effect.
[0003] 2. Data consistency issues: Multimodal data is prone to inconsistency during transmission and processing, especially in a distributed environment, where data consistency between different nodes is difficult to guarantee. Existing consistency algorithms have performance bottlenecks and complexity issues when processing multimodal data, leading to data inconsistency, which in turn affects the training effect and performance of the model.
[0004] 3. Computing resource allocation problem: Most existing resource management strategies are static allocation and cannot flexibly respond to changes in computing load. In a distributed environment, how to efficiently allocate computing resources to fully utilize the computing power of each node and accelerate the model training process is an urgent problem to be solved. Existing technologies have shortcomings in dynamic resource scheduling and load balancing, making it difficult to achieve optimal utilization of computing resources.
[0005] 4. Fault-tolerance processing issues: In a distributed environment, node failures and network delays are inevitable, but the existing fault-tolerance mechanisms are often slow to respond and take a long time to recover when dealing with failures in large-scale distributed environments. The lack of an effective fault detection and recovery mechanism makes it difficult to ensure system stability and the continuity of the training process.
[0006] 5. Parallel computing and model training issues: Although distributed training frameworks exist, they are still limited in processing multimodal data. Existing parallel computing technologies and model training methods are difficult to achieve efficient parallel processing when faced with large-scale, multi-source heterogeneous data, and the training speed and model performance are limited.
[0007] 6. Multimodal data processing and fusion issues: The processing and fusion of multimodal data has always been a difficult point in AI model training. Existing technologies have limitations in data fusion methods, making it difficult to achieve efficient and accurate comprehensive processing of multimodal data, which affects the comprehensive performance and application effect of the model.
[0008] Therefore, how to provide a multimodal AI model training method based on a distributed network architecture is an urgent problem that technicians in this field need to solve. Summary of the invention
[0009] One purpose of the present invention is to propose a multimodal AI model training method based on a distributed network architecture. The present invention solves many defects in the prior art through innovative distributed data synchronization protocols, consistency assurance mechanisms, dynamic resource allocation strategies, fault tolerance and recovery mechanisms, and advanced parallel computing and multimodal data processing and fusion technologies, significantly improves the efficiency, performance and reliability of multimodal AI model training, and provides strong technical support for AI model training of large-scale, multi-source heterogeneous data.
[0010] A multimodal AI model training method based on a distributed network architecture according to an embodiment of the present invention includes the following steps: S1. In a multi-node environment, multimodal data is stored through the distributed file system HDFS, and the gRPC protocol is used to transmit multimodal data between nodes to achieve data synchronization; S2. Preliminary preprocessing of the multimodal data, specifically including denoising, normalization and enhancement of image data, word segmentation, stop word removal and word vectorization of text data, and denoising, framing and feature extraction of audio data; S3, using the improved Paxos consensus algorithm to ensure data consistency between multiple nodes, using a distributed consensus mechanism to ensure data consistency in the event of network partitions or node failures, and using the distributed database Apache Cassandra for data storage; S4. Deploy the Kubernetes resource management framework to monitor the computing resource usage of each node in real time, dynamically adjust the computing task allocation according to the node load, and balance the task load of each node through Nginx load balancing technology; S5. In a distributed environment, the Prometheus distributed monitoring system is used to monitor the node status in real time and automatically detect node failures. When a failure is detected, task redistribution and data backup are triggered to ensure rapid system recovery. S6, using the TensorFlow Distributed distributed training framework, combined with data parallelism and model parallelism technology, to distribute large-scale data and model parameters to multiple computing nodes for parallel training, thus accelerating the model training process; S7, further processing and feature extraction of the multimodal data; S8. Use the attention mechanism to fuse data features of different modalities, generate comprehensive feature representation, and improve the training effect and overall performance of multimodal AI models.
[0011] Further, the S1 specifically includes: S11. In a multi-node environment, the multimodal data is distributed and stored in a distributed file system HDFS, where the multimodal data includes image data, text data, and audio data, and each node stores and accesses the data through the distributed file system HDFS; S12, storing and managing the image data through the distributed file system HDFS, including block storage and redundant storage of the image data, to ensure high availability and reliability of the data in the event of a node failure; S13, for text data, distributed storage and index management are performed through the distributed file system HDFS to ensure data reading efficiency in a large-scale text data environment, and to improve the retrieval speed of text data by using inverted index technology; S14. For audio data, the distributed file system HDFS is used for distributed storage, and the efficient storage and transmission of audio data in a distributed node environment is ensured through the sharding and multi-copy mechanism of audio data; S15. During data transmission between multiple nodes, the gRPC protocol is used to achieve efficient data transmission, including: Define gRPC service and message formats to ensure uniformity and standardization in data transmission between multiple nodes; Utilize the streaming function of gRPC to transmit multimodal data in real time, including streaming of image data, text data, and audio data; During the transmission process, data compression technology is used to reduce the amount of data transmitted and improve data transmission efficiency, including JPEG compression of image data, gzip compression of text data, and FLAC compression of audio data; Establish efficient data transmission channels between nodes and use gRPC's multiplexing and load balancing functions to ensure the stability and efficiency of data transmission; To ensure the security of data transmission, the TLS protocol is used to encrypt the gRPC data transmission channel to prevent data from being eavesdropped and tampered with during transmission; During the data transmission process, the transmission status is monitored in real time, the transmission log is recorded, and the log analysis tool is used to detect and handle the anomalies in the transmission process; After the data transmission is completed, the data is checked and confirmed at each node to ensure the integrity and consistency of the transmitted data. The transmitted data is checked using the check code, and the success of the data transmission is determined by the check result.
[0012] Furthermore, the S3 improved Paxos consensus algorithm specifically includes: S31. Run a Paxos instance in a multi-node environment, including three roles: proposer, acceptor, and learner. Optimize the message delivery mechanism by batch processing messages and message compression technology to reduce communication overhead. The message compression technology formula is: Among them, C msg Indicates the compressed message size, S i Represents the size of the i-th message, and N represents the number of messages. This formula reduces the size of each message by compressing the message to reduce the burden of network communication. The compressed message size is the average value calculated by logarithmically scaling the sizes of all messages.
[0013] The message delivery mechanism is optimized through batch processing of messages and message compression technology, which reduces communication overhead, improves the efficiency of message delivery, and reduces the consumption of network bandwidth.
[0014] S32, the proposer receives the request from the client and generates a proposal. The proposal contains a unique proposal number n and a proposal value v, and combines multiple proposals into one message to improve the message transmission efficiency, reduce the number of processing times for a single proposal, and reduce the network load. The proposer sends Prepare messages to multiple recipients, where the batch processing formula is: Among them, B msg Indicates the batch processed messages, P j Denotes the jth proposal, ∈ denotes the coefficient, and m denotes the number of proposals; proposal batching reduces the number of messages by merging multiple proposals into one message. ∈ in the formula is a small positive number used to adjust the logarithmic scale of the proposal size.
[0015] S33. After the acceptor receives the Prepare message, if the number n is greater than the maximum number it has responded to, it accepts the Prepare message and promises not to accept proposals with numbers less than n. At the same time, it replies to the proposer with a Promise message, which contains the number of the maximum numbered proposal accepted by the acceptor and the corresponding proposal value. S34. After receiving the Promise messages from the majority of recipients, the proposer immediately generates a proposal numbered n and with a proposal value of v, and sends an Accept request to the recipient. The message processing speed is improved through multi-threaded parallel processing. The formula for multi-threaded parallel processing is: Among them, T process represents the total time of parallel processing, T i represents the processing time of the i-th thread, k represents the number of threads, and α represents the constant coefficient; The message processing speed is improved through multi-threaded parallel processing. The α in the formula is used to adjust the square root part of the thread processing time, further optimizing the total processing time.
[0016] S35. After the acceptor receives the Accept request, if the number n is greater than or equal to the maximum number it has responded to, it accepts the proposal and replies with an Accepted message to the proposer. The Accepted message contains the number and value of the proposal. A dynamic priority mechanism is used to ensure that important proposals are prioritized to reach consensus. The formula of the dynamic priority mechanism is: Among them, P priority represents the proposal priority, L represents the delay of the proposal, T represents the importance of the proposal, R represents the resource consumption of the proposal, and β is a small positive number to avoid the denominator being zero; The dynamic priority mechanism calculates the priority of proposals by comprehensively considering the delay, importance, and resource consumption of proposals to ensure that important proposals are processed first. The β in the formula avoids the situation where the denominator is zero.
[0017] S36. After the proposer receives the Accepted message from the majority of acceptors, the proposal is considered to have reached consensus. The proposer broadcasts the proposal value to all learners and reduces the broadcast delay through hierarchical broadcast and intelligent broadcast strategies. The formula for hierarchical broadcast is: Among them, H broadcast Indicates the total time of hierarchical broadcast, B l represents the broadcast time of layer l, and h represents the number of layers; Hierarchical broadcasting reduces the total broadcasting time through a hierarchical broadcasting strategy. The broadcasting time of each layer in the formula is calculated in logarithmic form, which further optimizes the broadcasting efficiency.
[0018] S37. After receiving the notification from the proposer, the learner records the proposed value and applies it to ensure data consistency in the distributed system. An efficient consistency check technology is used to quickly check data consistency before each write operation. The formula for efficient consistency check is: Among them, E check Indicates the total time of consistency check, H(D i ) represents the hash value calculation time of the i-th data block, n represents the number of data blocks, and γ represents a constant coefficient; The consistency check optimizes the check time by calculating the hash value and adjusting the coefficient to ensure the consistency of the data before the write operation. The γ in the formula is used to optimize the square root part of the hash value calculation time.
[0019] S38. In the event of network partition or node failure, the acceptor and proposer will renegotiate the proposal according to the improved Paxos consensus algorithm steps, and introduce faulty node isolation and dynamic priority mechanisms to ensure that the system can quickly reach consensus and maintain consistency after failure recovery; S39. Use the distributed database Apache Cassandra for data storage, combine the incremental update strategy and the improved Paxos consistency algorithm to achieve strong consistency storage. Before each write operation, reach a consensus among multiple nodes through an optimized consensus mechanism to ensure the consistency of the write operation on all copies. The formula of the incremental update strategy is: Among them, Δ update represents the amount of data updated incrementally, δ i represents the data size of the ith update, and m represents the number of updates; The incremental update strategy reduces the data transmission and storage overhead by only transmitting and storing the changed data. The formula takes into account the square of the amount of changed data, further optimizing the update strategy.
[0020] Furthermore, the S4 specifically includes: S41. In a multi-node environment, install and configure the Kubernetes resource management framework and deploy the Kubernetes node agent on each computing node to ensure that each node can be managed and scheduled by the Kubernetes master node; S42. Use the Kubernetes resource monitoring module to collect and monitor the CPU, memory, and network bandwidth resource usage of each node in real time, and aggregate the monitoring data to the Kubernetes master node for use by the resource scheduling algorithm; S43. Design and implement a dynamic priority scheduling algorithm. The algorithm predicts the load change trend of each node based on real-time monitoring data and historical resource usage, and preferentially allocates resources to nodes with lighter loads to improve resource utilization. i (t) is calculated by the following formula: Among them, P i (t) represents the priority of node i at time t, which is used to determine the priority of resource allocation. i (t) represents the CPU usage of node i at time t, MEM i (t) represents the memory usage of node i at time t, α, β, γ represent weight coefficients, which are used to balance the impact of CPU usage, memory usage and historical load change trend on priority, H i (t) represents the historical load change trend of node i at time t, indicating the resource usage of the node in the past period of time, which is calculated by the exponentially weighted moving average of historical data: Among them, λ represents the smoothing coefficient, and its value range is 0<λ≤1; S44. Configure Nginx load balancer to evenly distribute user requests and data traffic to different computing nodes by setting reverse proxy and load balancing rules, ensuring that the task load of each node is evenly distributed and avoiding overloading of a single node; S45. Design and implement a load balancing optimization algorithm based on machine learning, using the historical load data and current status of the node to dynamically adjust the load balancing rules of Nginx; use a linear regression model to predict the load condition L of node i after time t i (t+Δt): L i (t+Δt)=θ0+θ1·CPU i (t)+θ2·MEM i (t)+θ3·Net i (t); Among them, L i (t+Δt) represents the predicted load of node i at time t+Δt, θ0, θ1, θ2, θ3 represent the parameters of the linear regression model, CPU i (t) represents the CPU usage of node i at time t, MEM i (t) represents the memory usage of node i at time t, Net i (t) represents the network bandwidth usage of node i at time t; the load balancing rule is dynamically adjusted according to the predicted load situation, and the node j with the lightest load is selected to process the new task: Among them, j represents the selected node with the lightest load; S46. When a resource bottleneck or failure is detected on a node, use Kubernetes's automatic task rescheduling function to reallocate the affected tasks to other nodes with sufficient resources. S47. Introduce a scheduling decision recording mechanism based on blockchain technology, record each task scheduling and resource allocation decision on the blockchain, and calculate the hash value H of the blockchain record as follows: H=SHA256(T i ||P i (t)||L i (t+Δt)); Where H represents the hash value of the blockchain record, SHA256 represents the hash function, Ti represents the task ID, Pi(t) represents the priority of node i at time t, and L i (t+Δt) represents the predicted load of node i at time t+Δt, and || represents the string concatenation operator, which is used to concatenate multiple values into a string for hash calculation.
[0021] Furthermore, the S7 specifically includes: using a convolutional neural network to extract features from image data; using a long short-term memory network to extract features from text data; using a Mel spectrum to convert audio data, and using a convolutional neural network to extract features.
[0022] The beneficial effects of the present invention are: 1. Efficient data synchronization: The present invention realizes real-time synchronization of multimodal data between different nodes through the distributed file system HDFS and the efficient data transmission protocol gRPC, significantly reduces the problems of data delay and asynchrony, and ensures data consistency and effectiveness during model training.
[0023] 2. Strong consistency data guarantee: Using the improved Paxos consistency algorithm, a solid data consistency guarantee mechanism is established between multiple nodes. The distributed consensus mechanism ensures data consistency in the event of network partitions or node failures, and uses the Apache Cassandra distributed database for data storage, which completely solves the consistency problem in the process of multimodal data processing and transmission, and avoids model training errors caused by data inconsistency.
[0024] 3. Dynamic resource allocation: By deploying the Kubernetes resource management framework, the computing resource usage of each node is monitored in real time, and the allocation of computing tasks is dynamically adjusted according to the load situation. Combined with Nginx load balancing technology, it ensures that the computing resources of each node are optimally utilized, significantly improving the efficiency and performance of model training.
[0025] 4. Reliable fault tolerance and recovery mechanism: In a distributed environment, the present invention uses the Prometheus distributed monitoring system to monitor the node status in real time, automatically detect node failures and trigger task redistribution and data backup, ensuring that the system can recover quickly when a failure occurs, ensuring the stability and continuity of the training process, and greatly improving the reliability of the system.
[0026] 5. Efficient parallel training: Using the TensorFlow Distributed distributed training framework, combined with data parallelism and model parallelism, large-scale data and model parameters are distributed to multiple computing nodes for parallel training, which effectively accelerates the model training process and improves the training speed and model performance.
[0027] 6. Accurate multimodal data processing and fusion: Through preliminary preprocessing and feature extraction of image, text and audio data, the data features of different modalities are fused in combination with the attention mechanism to generate a comprehensive feature representation. The fusion technology of the present invention improves the efficiency and accuracy of comprehensive processing of multimodal data, and significantly improves the training effect and comprehensive performance of the AI model. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 A flowchart of a multimodal AI model training method based on a distributed network architecture proposed by the present invention; Figure 2 A schematic diagram of a multi-node data consistency assurance mechanism for a multimodal AI model training method based on a distributed network architecture proposed in the present invention; Figure 3 A schematic diagram of dynamic allocation of computing resources and load balancing for a multimodal AI model training method based on a distributed network architecture proposed in the present invention. DETAILED DESCRIPTION
[0029] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0030] refer to Figure 1-3 , a multimodal AI model training method based on a distributed network architecture, comprising the following steps: S1. In a multi-node environment, multimodal data is stored through the distributed file system HDFS, and the gRPC protocol is used to transmit multimodal data between nodes to achieve data synchronization; S2. Preliminary preprocessing of the multimodal data, specifically including denoising, normalization and enhancement of image data, word segmentation, stop word removal and word vectorization of text data, and denoising, framing and feature extraction of audio data; S3, using the improved Paxos consensus algorithm to ensure data consistency between multiple nodes, using a distributed consensus mechanism to ensure data consistency in the event of network partitions or node failures, and using the distributed database Apache Cassandra for data storage; S4. Deploy the Kubernetes resource management framework to monitor the computing resource usage of each node in real time, dynamically adjust the computing task allocation according to the node load, and balance the task load of each node through Nginx load balancing technology; S5. In a distributed environment, the Prometheus distributed monitoring system is used to monitor the node status in real time and automatically detect node failures. When a failure is detected, task redistribution and data backup are triggered to ensure rapid system recovery. S6, using the TensorFlow Distributed distributed training framework, combined with data parallelism and model parallelism technology, to distribute large-scale data and model parameters to multiple computing nodes for parallel training, thus accelerating the model training process; S7, further processing and feature extraction of the multimodal data; S8. Use the attention mechanism to fuse data features of different modalities, generate comprehensive feature representation, and improve the training effect and overall performance of multimodal AI models.
[0031] In this implementation, S1 specifically includes: S11. In a multi-node environment, the multimodal data is distributed and stored in a distributed file system HDFS, where the multimodal data includes image data, text data, and audio data, and each node stores and accesses the data through the distributed file system HDFS; S12, storing and managing the image data through the distributed file system HDFS, including block storage and redundant storage of the image data, to ensure high availability and reliability of the data in the event of a node failure; S13, for text data, distributed storage and index management are performed through the distributed file system HDFS to ensure data reading efficiency in a large-scale text data environment, and to improve the retrieval speed of text data by using inverted index technology; S14. For audio data, the distributed file system HDFS is used for distributed storage, and the efficient storage and transmission of audio data in a distributed node environment is ensured through the sharding and multi-copy mechanism of audio data; S15. During data transmission between multiple nodes, the gRPC protocol is used to achieve efficient data transmission, including: Define gRPC service and message formats to ensure uniformity and standardization in data transmission between multiple nodes; Utilize the streaming function of gRPC to transmit multimodal data in real time, including streaming of image data, text data, and audio data; During the transmission process, data compression technology is used to reduce the amount of data transmitted and improve data transmission efficiency, including JPEG compression of image data, gzip compression of text data, and FLAC compression of audio data; Establish efficient data transmission channels between nodes and use gRPC's multiplexing and load balancing functions to ensure the stability and efficiency of data transmission; To ensure the security of data transmission, the TLS protocol is used to encrypt the gRPC data transmission channel to prevent data from being eavesdropped and tampered with during transmission; During the data transmission process, the transmission status is monitored in real time, the transmission log is recorded, and the log analysis tool is used to detect and handle the anomalies in the transmission process; After the data transmission is completed, the data is checked and confirmed at each node to ensure the integrity and consistency of the transmitted data. The transmitted data is checked using the check code, and the success of the data transmission is determined by the check result.
[0032] In this implementation, the S3 improved Paxos consensus algorithm specifically includes: S31. Run a Paxos instance in a multi-node environment, including three roles: proposer, acceptor, and learner. Optimize the message delivery mechanism by batch processing messages and message compression technology to reduce communication overhead. The message compression technology formula is: Among them, C msg Indicates the compressed message size, S i Represents the size of the i-th message, and N represents the number of messages. This formula reduces the size of each message by compressing the message to reduce the burden of network communication. The compressed message size is the average value calculated by logarithmically scaling the sizes of all messages.
[0033] S32, the proposer receives the request from the client and generates a proposal. The proposal contains a unique proposal number n and a proposal value v, and combines multiple proposals into one message to improve the efficiency of message transmission. The proposer sends Prepare messages to multiple recipients, where the batch processing formula is: Among them, Bmsg Indicates the batch processed messages, P j Denotes the jth proposal, ∈ denotes the coefficient, and m denotes the number of proposals; proposal batching reduces the number of messages by merging multiple proposals into one message. ∈ in the formula is a small positive number used to adjust the logarithmic scale of the proposal size.
[0034] S33. After the acceptor receives the Prepare message, if the number n is greater than the maximum number it has responded to, it accepts the Prepare message and promises not to accept proposals with numbers less than n. At the same time, it replies to the proposer with a Promise message, which contains the number of the maximum numbered proposal accepted by the acceptor and the corresponding proposal value. S34. After receiving the Promise messages from the majority of recipients, the proposer immediately generates a proposal numbered n and with a proposal value of v, and sends an Accept request to the recipient. The message processing speed is improved through multi-threaded parallel processing. The formula for multi-threaded parallel processing is: Among them, T process represents the total time of parallel processing, T i represents the processing time of the i-th thread, k represents the number of threads, and α represents the constant coefficient; The message processing speed is improved through multi-threaded parallel processing. The α in the formula is used to adjust the square root part of the thread processing time, further optimizing the total processing time.
[0035] S35. After the acceptor receives the Accept request, if the number n is greater than or equal to the maximum number it has responded to, it accepts the proposal and replies with an Accepted message to the proposer. The Accepted message contains the number and value of the proposal. A dynamic priority mechanism is used to ensure that important proposals are prioritized to reach consensus. The formula of the dynamic priority mechanism is: Among them, P priority represents the proposal priority, L represents the delay of the proposal, T represents the importance of the proposal, R represents the resource consumption of the proposal, and β is a small positive number to avoid the denominator being zero; The dynamic priority mechanism calculates the priority of proposals by comprehensively considering the delay, importance, and resource consumption of proposals to ensure that important proposals are processed first. The β in the formula avoids the situation where the denominator is zero.
[0036] S36. After the proposer receives the Accepted message from the majority of acceptors, the proposal is considered to have reached consensus. The proposer broadcasts the proposal value to all learners and reduces the broadcast delay through hierarchical broadcast and intelligent broadcast strategies. The formula for hierarchical broadcast is: Among them, H broadcast Indicates the total time of hierarchical broadcast, B l represents the broadcast time of layer l, and h represents the number of layers; Hierarchical broadcasting reduces the total broadcasting time through a hierarchical broadcasting strategy. The broadcasting time of each layer in the formula is calculated in logarithmic form, which further optimizes the broadcasting efficiency.
[0037] S37. After receiving the notification from the proposer, the learner records the proposed value and applies it to ensure data consistency in the distributed system. An efficient consistency check technology is used to quickly check data consistency before each write operation. The formula for efficient consistency check is: Among them, E check Indicates the total time of consistency check, H(D i ) represents the hash value calculation time of the i-th data block, n represents the number of data blocks, and γ represents a constant coefficient; The consistency check optimizes the check time by calculating the hash value and adjusting the coefficient to ensure the consistency of the data before the write operation. The γ in the formula is used to optimize the square root part of the hash value calculation time.
[0038] S38. In the event of network partition or node failure, the acceptor and proposer will renegotiate the proposal according to the improved Paxos consensus algorithm steps, and introduce faulty node isolation and dynamic priority mechanisms to ensure that the system can quickly reach consensus and maintain consistency after failure recovery; S39. Use the distributed database Apache Cassandra for data storage, combine the incremental update strategy and the improved Paxos consistency algorithm to achieve strong consistency storage. Before each write operation, reach a consensus among multiple nodes through an optimized consensus mechanism to ensure the consistency of the write operation on all copies. The formula of the incremental update strategy is: Among them, Δ update represents the amount of data updated incrementally, δ i represents the data size of the ith update, and m represents the number of updates; The incremental update strategy reduces the data transmission and storage overhead by only transmitting and storing the changed data. The formula takes into account the square of the amount of changed data, further optimizing the update strategy.
[0039] In this implementation manner, the S4 specifically includes: S41. In a multi-node environment, install and configure the Kubernetes resource management framework and deploy the Kubernetes node agent on each computing node to ensure that each node can be managed and scheduled by the Kubernetes master node; S42. Use the Kubernetes resource monitoring module to collect and monitor the CPU, memory, and network bandwidth resource usage of each node in real time, and aggregate the monitoring data to the Kubernetes master node for use by the resource scheduling algorithm; S43. Design and implement a dynamic priority scheduling algorithm. The algorithm predicts the load change trend of each node based on real-time monitoring data and historical resource usage, and preferentially allocates resources to nodes with lighter loads to improve resource utilization. i (t) is calculated by the following formula: Among them, P i (t) represents the priority of node i at time t, which is used to determine the priority of resource allocation. i (t) represents the CPU usage of node i at time t, MEM i (t) represents the memory usage of node i at time t, α, β, γ represent weight coefficients, which are used to balance the impact of CPU usage, memory usage and historical load change trend on priority, H i (t) represents the historical load change trend of node i at time t, indicating the resource usage of the node in the past period of time, which is calculated by the exponentially weighted moving average of historical data: Among them, λ represents the smoothing coefficient, and its value range is 0<λ≤1; S44. Configure Nginx load balancer to evenly distribute user requests and data traffic to different computing nodes by setting reverse proxy and load balancing rules, ensuring that the task load of each node is evenly distributed and avoiding overloading of a single node; S45. Design and implement a load balancing optimization algorithm based on machine learning, using the historical load data and current status of the node to dynamically adjust the load balancing rules of Nginx; use a linear regression model to predict the load condition L of node i after time t i (t+Δt): L i (t+Δt)=θ0+θ1·CPU i (t)+θ2·MEM i (t)+θ3·Net i (t); Among them, L i(t+Δt) represents the predicted load of node i at time t+Δt, θ0, θ1, θ2, θ3 represent the parameters of the linear regression model, CPU i (t) represents the CPU usage of node i at time t, MEM i (t) represents the memory usage of node i at time t, Net i (t) represents the network bandwidth usage of node i at time t; the load balancing rule is dynamically adjusted according to the predicted load situation, and the node j with the lightest load is selected to process the new task: Among them, j represents the selected node with the lightest load; S46. When a resource bottleneck or failure is detected on a node, use Kubernetes's automatic task rescheduling function to reallocate the affected tasks to other nodes with sufficient resources. S47. Introduce a scheduling decision recording mechanism based on blockchain technology, record each task scheduling and resource allocation decision on the blockchain, and calculate the hash value H of the blockchain record as follows: H=SHA256(T i ||P i (t)||L i (t+Δt)); Where H represents the hash value of the blockchain record, SHA256 represents the hash function, and T i Indicates the task ID, P i (t) represents the priority of node i at time t, L i (t+Δt) represents the predicted load of node i at time t+Δt, and || represents the string concatenation operator, which is used to concatenate multiple values into a string for hash calculation.
[0040] In this implementation, S7 specifically includes: using a convolutional neural network to extract features from image data; using a long short-term memory network to extract features from text data; using a Mel spectrum to convert audio data, and using a convolutional neural network to extract features.
[0041] Embodiment 1: In order to verify the feasibility of the present invention in implementation, the present invention is applied to an Internet company. The company needs to build a multimodal AI model to improve the accuracy of its recommendation system and user experience. The model needs to process a large amount of heterogeneous data from different data sources, including images browsed by users, submitted text comments, and audio data of voice input. Due to the huge amount of data and the scattered sources, the traditional centralized data processing method cannot meet the real-time processing and training requirements, so it is decided to adopt a multimodal AI model training method based on a distributed network architecture.
[0042] In order to achieve efficient multimodal AI model training, the system first deployed a distributed file system HDFS in a multi-node environment to store multimodal data from different data sources. The data includes millions of product images browsed by users, hundreds of thousands of text reviews, and tens of thousands of hours of user voice input. The gRPC protocol is used to transmit these multimodal data between nodes to achieve real-time data synchronization.
[0043] After data synchronization is completed, the system performs preliminary preprocessing on the data. Image data is denoised by Gaussian filtering and normalized by Min-Max normalization. Data diversity is increased by data enhancement techniques such as random cropping and rotation. Text data is segmented using the NLTK library and stop words are removed. The Word2Vec model is then used to convert text into word vectors. Audio data is first subjected to wavelet transformation to remove noise, then framed, with each frame length of 25ms, and finally Mel-frequency cepstral coefficients (MFCCs) are extracted as features.
[0044] In order to ensure data consistency among multiple nodes, the system uses an improved Paxos consistency algorithm to ensure data consistency in the event of network partitions or node failures through a distributed consensus mechanism, and uses the Apache Cassandra distributed database for data storage to ensure data consistency and persistence.
[0045] In terms of resource management, the system deploys the Kubernetes resource management framework to monitor the computing resource usage of each node in real time and dynamically adjust the computing task allocation according to the node load. Through Nginx load balancing technology, it ensures that the computing resources of each node are optimally utilized to avoid overloading or idleness of some nodes.
[0046] In a distributed environment, the system uses the Prometheus distributed monitoring system to monitor node status in real time, automatically detect node failures and trigger task redistribution and data backup. When a node fails, the system can quickly redistribute tasks on the node to other normally operating nodes and restore the data of the failed node from the backup data to ensure the stability of the system and the continuity of the training process.
[0047] Using the TensorFlow Distributed distributed training framework, the system combines data parallelism and model parallelism to distribute large-scale data and model parameters to multiple computing nodes for parallel training, accelerating the model training process. During the training process, the system further processes and extracts features from multimodal data. Image data is extracted through convolutional neural networks (CNNs), text data is extracted through long short-term memory networks (LSTMs), and audio data is extracted through Mel-spectrogram conversion and convolutional neural networks.
[0048] Finally, the system uses the attention mechanism to fuse the data features of different modalities, generate a comprehensive feature representation, and input it into the AI model for training and optimization. In this way, the system achieves efficient multimodal AI model training, greatly improving the training effect and overall performance of the model.
[0049] Table 1 Comparison of comprehensive performance data of traditional method and method of the present invention Table 1 fully demonstrates that the various performance indicators of the present invention in practical applications are significantly improved compared with traditional methods. These data fully demonstrate the technical advantages and beneficial effects of the present invention in data synchronization, data consistency assurance, computing resource utilization, fault detection and recovery, model training speed, and recommendation system performance.
[0050] In summary, the present invention significantly improves the efficiency, performance and reliability of multimodal AI model training through efficient data synchronization and consistency assurance mechanism, dynamic resource allocation strategy, reliable fault tolerance and recovery mechanism, and advanced parallel computing and multimodal data processing and fusion technology, solves many defects in the existing technology, and provides strong technical support for AI model training of large-scale, multi-source heterogeneous data.
[0051] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A multimodal AI model training method based on a distributed network architecture, characterized in that: The steps include: S1. In a multi-node environment, multimodal data is stored through the distributed file system HDFS, and the gRPC protocol is used to transmit multimodal data between nodes to achieve data synchronization; S2. Preliminary preprocessing of the multimodal data, specifically including denoising, normalization and enhancement of image data, word segmentation, stop word removal and word vectorization of text data, and denoising, framing and feature extraction of audio data; S3, using the improved Paxos consensus algorithm to ensure data consistency between multiple nodes, using a distributed consensus mechanism to ensure data consistency in the event of network partitions or node failures, and using the distributed database Apache Cassandra for data storage; S4. Deploy the Kubernetes resource management framework to monitor the computing resource usage of each node in real time, dynamically adjust the computing task allocation according to the node load, and balance the task load of each node through Nginx load balancing technology; S5. In a distributed environment, use the Prometheus distributed monitoring system to monitor node status in real time and automatically detect node failures. When a failure is detected, it triggers task redistribution and data backup. S6, using the TensorFlow Distributed distributed training framework, combined with data parallelism and model parallelism technology, to distribute large-scale data and model parameters to multiple computing nodes for parallel training, thus accelerating the model training process; S7, further processing and feature extraction of the multimodal data; S8. Use the attention mechanism to fuse data features of different modalities, generate comprehensive feature representation, and improve the training effect and overall performance of multimodal AI models.
2. According to a multimodal AI model training method based on a distributed network architecture according to claim 1, it is characterized in that: The S1 specifically includes: S11. In a multi-node environment, the multimodal data is distributed and stored in a distributed file system HDFS, where the multimodal data includes image data, text data, and audio data, and each node stores and accesses the data through the distributed file system HDFS; S12, storing and managing the image data through the distributed file system HDFS, including block storage and redundant storage of the image data, to ensure high availability and reliability of the data in the event of a node failure; S13, for text data, distributed storage and index management are performed through the distributed file system HDFS to ensure data reading efficiency in a large-scale text data environment, and to improve the retrieval speed of text data by using inverted index technology; S14. For audio data, the distributed file system HDFS is used for distributed storage, and the efficient storage and transmission of audio data in a distributed node environment is ensured through the sharding and multi-copy mechanism of audio data; S15. During data transmission between multiple nodes, the gRPC protocol is used to achieve efficient data transmission, including: Define gRPC service and message formats to ensure uniformity and standardization in the data transmission process between multiple nodes; use gRPC's streaming function to transmit multimodal data in real time, including streaming transmission of image data, text data, and audio data; During the transmission process, data compression technology is used to reduce the amount of data transmitted and improve data transmission efficiency, including JPEG compression of image data, gzip compression of text data, and FLAC compression of audio data; Establish efficient data transmission channels between nodes and use gRPC's multiplexing and load balancing functions to ensure the stability and efficiency of data transmission; To ensure the security of data transmission, the TLS protocol is used to encrypt the gRPC data transmission channel to prevent data from being eavesdropped and tampered with during transmission; During the data transmission process, the transmission status is monitored in real time, the transmission log is recorded, and the log analysis tool is used to detect and handle the anomalies in the transmission process; After the data transmission is completed, the data is checked and confirmed at each node to ensure the integrity and consistency of the transmitted data. The transmitted data is checked using the check code, and the success of the data transmission is determined by the check result.
3. According to a multimodal AI model training method based on a distributed network architecture according to claim 1, it is characterized in that: The S3 improved Paxos consensus algorithm specifically includes: S31. Run a Paxos instance in a multi-node environment, including three roles: proposer, acceptor, and learner. Optimize the message delivery mechanism by batch processing messages and message compression technology to reduce communication overhead. The message compression technology formula is: Among them, C msg Indicates the compressed message size, S i represents the size of the i-th message, and N represents the number of messages; S32. The proposer receives the request from the client and generates a proposal. The proposal contains a unique proposal number n and a proposal value v. Multiple proposals are combined in one message and sent. The proposer sends a Prepare message to multiple recipients. The batch processing formula is: Among them, B msg Indicates the batch processed messages, P j represents the jth proposal, ∈ represents the coefficient, and m represents the number of proposals; S33. After the acceptor receives the Prepare message, if the number n is greater than the maximum number it has responded to, it accepts the Prepare message and promises not to accept proposals with numbers less than n. At the same time, it replies to the proposer with a Promise message, which contains the number of the maximum numbered proposal accepted by the acceptor and the corresponding proposal value. S34. After receiving the Promise messages from the majority of recipients, the proposer immediately generates a proposal numbered n and with a proposal value of v, and sends an Accept request to the recipient. The message processing speed is improved through multi-threaded parallel processing. The formula for multi-threaded parallel processing is: Among them, T process represents the total time of parallel processing, T i represents the processing time of the i-th thread, k represents the number of threads, and α represents the constant coefficient; S35. After the acceptor receives the Accept request, if the number n is greater than or equal to the maximum number it has responded to, it accepts the proposal and replies with an Accepted message to the proposer. The Accepted message contains the number and value of the proposal and adopts a dynamic priority mechanism. The formula of the dynamic priority mechanism is: Among them, P priority represents the proposal priority, L represents the delay of the proposal, T represents the importance of the proposal, R represents the resource consumption of the proposal, and β is a small positive number to avoid the denominator being zero; S36. After the proposer receives the Accepted message from the majority of acceptors, the proposal is considered to have reached consensus. The proposer broadcasts the proposal value to all learners and reduces the broadcast delay through hierarchical broadcast and intelligent broadcast strategies. The formula for hierarchical broadcast is: Among them, H broadcast Indicates the total time of hierarchical broadcast, B l represents the broadcast time of layer l, and h represents the number of layers; S37. After receiving the notification from the proposer, the learner records the proposed value and applies it. It uses efficient consistency check technology to quickly check data consistency before each write operation. The formula for efficient consistency check is: Among them, E check Indicates the total time of consistency check, H(D i ) represents the hash value calculation time of the i-th data block, n represents the number of data blocks, and γ represents a constant coefficient; S38. In the event of network partition or node failure, the acceptor and proposer will renegotiate the proposal according to the improved Paxos consensus algorithm steps, and introduce faulty node isolation and dynamic priority mechanisms; S39, using the distributed database Apache Cassandra for data storage, combined with the incremental update strategy and the improved Paxos consistency algorithm to achieve strong consistency storage. Before each write operation, consensus is reached among multiple nodes through an optimized consensus mechanism. The formula for the incremental update strategy is: Among them, Δ update represents the amount of data updated incrementally, δ i represents the data size of the ith update, and m represents the number of updates.
4. According to claim 1, a multimodal AI model training method based on a distributed network architecture is characterized in that: The S4 specifically includes: S41. In a multi-node environment, install and configure the Kubernetes resource management framework and deploy the Kubernetes node agent on each compute node. S42. Use the Kubernetes resource monitoring module to collect and monitor the CPU, memory, and network bandwidth resource usage of each node in real time, and aggregate the monitoring data to the Kubernetes master control node; S43. Design and implement a dynamic priority scheduling algorithm. The algorithm predicts the load change trend of each node based on real-time monitoring data and historical resource usage, and preferentially allocates resources to nodes with lighter loads to improve resource utilization. i (t) is calculated by the following formula: Among them, P i (t) represents the priority of node i at time t, CPU i (t) represents the CPU usage of node i at time t, MEM i (t) represents the memory usage of node i at time t, α, β, γ represent weight coefficients, H i (t) represents the historical load change trend of node i at time t, which is calculated by the exponentially weighted moving average of historical data: Among them, λ represents the smoothing coefficient, and its value range is 0<λ≤1; S44. Configure the Nginx load balancer to evenly distribute user requests and data traffic to different computing nodes by setting reverse proxy and load balancing rules; S45. Design and implement a load balancing optimization algorithm based on machine learning, using the historical load data and current status of the node to dynamically adjust the load balancing rules of Nginx; use a linear regression model to predict the load condition L of node i after time t i (t+Δt): L i (t+Δt)=θ0+θ1·CPU i (t)+θ2·MEM i (t)+θ3·Net i (t); Among them, L i (t+Δt) represents the predicted load of node i at time t+Δt, θ0, θ1, θ2, θ3 represent the parameters of the linear regression model, CPU i (t) represents the CPU usage of node i at time t, MEM i (t) represents the memory usage of node i at time t, Net i (t) represents the network bandwidth usage of node i at time t; the load balancing rule is dynamically adjusted according to the predicted load situation, and the node j with the lightest load is selected to process the new task: Among them, j represents the selected node with the lightest load; S46. When a resource bottleneck or failure is detected on a node, use Kubernetes's automatic task rescheduling function to reallocate the affected tasks to other nodes with sufficient resources. S47. Introduce a scheduling decision recording mechanism based on blockchain technology, record each task scheduling and resource allocation decision on the blockchain, and calculate the hash value H of the blockchain record as follows: H=SHA256(T i ||P i (t)||L i (t+Δt)); Where H represents the hash value of the blockchain record, SHA256 represents the hash function, and T i Indicates the task ID, P i (t) represents the priority of node i at time t, L i (t+Δt) represents the predicted load of node i at time t+Δt, and || represents the string concatenation operator, which is used to concatenate multiple values into a string for hash calculation.
5. According to a multimodal AI model training method based on a distributed network architecture according to claim 1, it is characterized in that: The S7 specifically includes: using a convolutional neural network to extract features from image data; using a long short-term memory network to extract features from text data; using a Mel spectrum to convert audio data, and using a convolutional neural network to extract features.
Citation Information
Patent Citations
Distributed database automatic operation and maintenance method and system based on artificial intelligence
CN112579391A
Distributed training and reasoning method, system and device based on artificial intelligence, and readable storage medium
CN114035937A
Model training method and system, data processing method and system and electronic equipment
CN116070113A
Model automatic parallel method and device based on load balancing and storage medium
CN116400963A
Intelligent computing network scheduling method for computing and communication fusion of large model task
CN117667360A
